Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
IanR
Helper III
Helper III

Does refresh bring back all the data then filter or filter first then bring back the data


I am bringing data back from CRM Online to be analysed in Power BI Desktop. The refresh takes over an hour and the numbers beside the circling dots suggest that over the ten or so tables I am bringing back about 5 GB of data. Massive! I have two main questions:

 

1) Are the numbers beside the circling dots really correct – am I actually bringing back that much data or is it just sifting through that much data on the server?
2) If I severely hack back the amount of data that I include in my model – by reducing the number of fields and adding filters to reduce the number of records – will this be applied at the server end when I requery, thereby reducing the amount of data being transferred, or will all the data still come back and be filtered in Power BI Desktop/Query?

 

I’d love to publish the reports I have created and set up automatic refreshing but I am worried that this will fall over if it is asked to work with so much data.

Just in case the answer varies according to the type of connection I am using oData to connect to Microsoft Dynamics CRM Online.

 

1 ACCEPTED SOLUTION
Greg_Deckler
Super User
Super User

You would have to look at your data model to see how big it ends up (you can judge this by just looking at the pbix file size. Odds are you are shifting through that much data but it is all getting compressed into the data model so it is not as big when stored.

 

If you edit your query to only include the columns you are using, it should reduce the amount of data being pulled back over the wire and also reduce your data model size.

 

CRM has a lot of "extraneous" fields, I find it is very advantagous to only bring back the dozen or so fields per table that are actually relevant.


@ me in replies or I'll lose your thread!!!
Instead of a Kudo, please vote for this idea
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
Mastering Power BI 2nd Edition

DAX is easy, CALCULATE makes DAX hard...

View solution in original post

6 REPLIES 6
Greg_Deckler
Super User
Super User

You would have to look at your data model to see how big it ends up (you can judge this by just looking at the pbix file size. Odds are you are shifting through that much data but it is all getting compressed into the data model so it is not as big when stored.

 

If you edit your query to only include the columns you are using, it should reduce the amount of data being pulled back over the wire and also reduce your data model size.

 

CRM has a lot of "extraneous" fields, I find it is very advantagous to only bring back the dozen or so fields per table that are actually relevant.


@ me in replies or I'll lose your thread!!!
Instead of a Kudo, please vote for this idea
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
Mastering Power BI 2nd Edition

DAX is easy, CALCULATE makes DAX hard...

Sounds encouraging - I will experiment this evening.

 

Is there a report somewhere that shows how much data has been transferred and how long it has taken? If I'm not watching the screen when the circling dots pop-up disapprears I can't be sure of either. As these downloads take over an hour that's a lot of times I have to keep popping back to see whether it's finished or how much data has transferred so far.

Hi @IanR,

 

>>Is there a report somewhere that shows how much data has been transferred and how long it has taken?

Current, power bi not support this feature, perhaps you can post this requirement to ideas.

 

Regards,

Xiaoxin Sheng

Community Support Team _ Xiaoxin
If this post helps, please consider accept as solution to help other members find it more quickly.

>>Is there a report somewhere that shows how much data has been transferred and how long it has taken?

Current, power bi not support this feature, perhaps you can post this requirement to ideas.

 

How do I do that?

Hi @IanR,

 

Post an idea at below link:

power bi ideas.

 

Regards,
Xiaoxin Sheng

Community Support Team _ Xiaoxin
If this post helps, please consider accept as solution to help other members find it more quickly.

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.