cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
IanR
Helper III
Helper III

Does refresh bring back all the data then filter or filter first then bring back the data


I am bringing data back from CRM Online to be analysed in Power BI Desktop. The refresh takes over an hour and the numbers beside the circling dots suggest that over the ten or so tables I am bringing back about 5 GB of data. Massive! I have two main questions:

 

1) Are the numbers beside the circling dots really correct – am I actually bringing back that much data or is it just sifting through that much data on the server?
2) If I severely hack back the amount of data that I include in my model – by reducing the number of fields and adding filters to reduce the number of records – will this be applied at the server end when I requery, thereby reducing the amount of data being transferred, or will all the data still come back and be filtered in Power BI Desktop/Query?

 

I’d love to publish the reports I have created and set up automatic refreshing but I am worried that this will fall over if it is asked to work with so much data.

Just in case the answer varies according to the type of connection I am using oData to connect to Microsoft Dynamics CRM Online.

 

1 ACCEPTED SOLUTION
Greg_Deckler
Super User
Super User

You would have to look at your data model to see how big it ends up (you can judge this by just looking at the pbix file size. Odds are you are shifting through that much data but it is all getting compressed into the data model so it is not as big when stored.

 

If you edit your query to only include the columns you are using, it should reduce the amount of data being pulled back over the wire and also reduce your data model size.

 

CRM has a lot of "extraneous" fields, I find it is very advantagous to only bring back the dozen or so fields per table that are actually relevant.


@ me in replies or I'll lose your thread!!!
Instead of a Kudo, please vote for this idea
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
Mastering Power BI 2nd Edition

DAX is easy, CALCULATE makes DAX hard...

View solution in original post

6 REPLIES 6
Greg_Deckler
Super User
Super User

You would have to look at your data model to see how big it ends up (you can judge this by just looking at the pbix file size. Odds are you are shifting through that much data but it is all getting compressed into the data model so it is not as big when stored.

 

If you edit your query to only include the columns you are using, it should reduce the amount of data being pulled back over the wire and also reduce your data model size.

 

CRM has a lot of "extraneous" fields, I find it is very advantagous to only bring back the dozen or so fields per table that are actually relevant.


@ me in replies or I'll lose your thread!!!
Instead of a Kudo, please vote for this idea
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
Mastering Power BI 2nd Edition

DAX is easy, CALCULATE makes DAX hard...

Sounds encouraging - I will experiment this evening.

 

Is there a report somewhere that shows how much data has been transferred and how long it has taken? If I'm not watching the screen when the circling dots pop-up disapprears I can't be sure of either. As these downloads take over an hour that's a lot of times I have to keep popping back to see whether it's finished or how much data has transferred so far.

Hi @IanR,

 

>>Is there a report somewhere that shows how much data has been transferred and how long it has taken?

Current, power bi not support this feature, perhaps you can post this requirement to ideas.

 

Regards,

Xiaoxin Sheng

Community Support Team _ Xiaoxin
If this post helps, please consider accept as solution to help other members find it more quickly.

>>Is there a report somewhere that shows how much data has been transferred and how long it has taken?

Current, power bi not support this feature, perhaps you can post this requirement to ideas.

 

How do I do that?

Hi @IanR,

 

Post an idea at below link:

power bi ideas.

 

Regards,
Xiaoxin Sheng

Community Support Team _ Xiaoxin
If this post helps, please consider accept as solution to help other members find it more quickly.

Helpful resources

Announcements
Carousel_PBI_Wave1

2023 Release Wave 1 Plans

Power BI release plans for 2023 release wave 1 describes all new features releasing from April 2023 through September 2023.

Power BI Summit Carousel 2

Global Power BI Training

Make sure you register today for the Power BI Summit 2023. Don't miss all of the great sessions and speakers!

Thank you 2022 Review

2022 Monthly Feature Releases

We had a great 2022 with a ton of feature releases to help you drive a data culture.