Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Grow your Fabric skills and prepare for the DP-600 certification exam by completing the latest Microsoft Fabric challenge.

Reply
Simple_tuition1
Frequent Visitor

Best Practice for only loading recent data in Power BI Desktop

Hi,

 

I'm currently creating an relational Database with data from several sources. I would like to create a dashboard report every quarter with the newest data in Power BI Desktop. The data I get is also quarterly. Since I just build the database, only data from the last quarter is in the database, so I can directly load from the DB and create the dashboards with the data in PBI Desktop. But what if the database gets bigger and multiple quarter data ist loaded into the db. I can imagine after some time it will take longer and longer to load the data into the model, since there will be more rows.

 

What would be the best practice to prevent this kind of scenario from happening?

Further what is the best approach to only filter the data from the latest quarter?

Is it possible to only load data into power query or the model from a certain date like having a column with year and quarter as a key. Loading only rows with this key into power query?

Or do I have to create a filter on the dashboard page with a certain date key? But this wouldn't solve the issue that the loaded data size is getting bigger and bigger with time.

1 ACCEPTED SOLUTION
MarkLaf
Solution Sage
Solution Sage

If your data source is a relational database, there is a good chance that it supports query folding (https://learn.microsoft.com/en-us/power-query/power-query-folding). This means that query steps you define in Power Query will, instead of importing all data and then applying transformations, will instead convert them to your DB's native query language and pass it along so that your DB performs the transformation (e.g. filtering on dates) and then Power Query only loads the (filtered) output rather than all data.

So, you can probably simply make a copy of your pbix, change the filter step (which should be one of your first transform steps), and let your DB only pass the data you need for the period you filtered on. You could even use a parameter to control the filter and not even deal with opening pbix's; just make a copy in the service, change parameter value, refresh.

 

To address your questions specifically:

 

What would be the best practice to prevent this kind of scenario from happening?

Probably you can leverage query folding as mentioned above. The page I linked has a pretty good explainer on what transformations fold, how to check if folding is occurring, etc.

 

Further what is the best approach to only filter the data from the latest quarter?

If you are already going the route of manually creating a new report every quarter, you can probably just manually specify the start and end date of the period you want in a filter on the table.

"best approach" can depend on the situation, but if it were me I would think about just making one report with a Direct Query connection to your database. Way less admin overhead to maintain. https://learn.microsoft.com/en-us/power-bi/connect-data/desktop-use-directquery 

 

Is it possible to only load data into power query or the model from a certain date like having a column with year and quarter as a key. Loading only rows with this key into power query?

Yes, this is exactly the intent of query folding.

 

Or do I have to create a filter on the dashboard page with a certain date key? But this wouldn't solve the issue that the loaded data size is getting bigger and bigger with time.

☝️

View solution in original post

2 REPLIES 2
MarkLaf
Solution Sage
Solution Sage

If your data source is a relational database, there is a good chance that it supports query folding (https://learn.microsoft.com/en-us/power-query/power-query-folding). This means that query steps you define in Power Query will, instead of importing all data and then applying transformations, will instead convert them to your DB's native query language and pass it along so that your DB performs the transformation (e.g. filtering on dates) and then Power Query only loads the (filtered) output rather than all data.

So, you can probably simply make a copy of your pbix, change the filter step (which should be one of your first transform steps), and let your DB only pass the data you need for the period you filtered on. You could even use a parameter to control the filter and not even deal with opening pbix's; just make a copy in the service, change parameter value, refresh.

 

To address your questions specifically:

 

What would be the best practice to prevent this kind of scenario from happening?

Probably you can leverage query folding as mentioned above. The page I linked has a pretty good explainer on what transformations fold, how to check if folding is occurring, etc.

 

Further what is the best approach to only filter the data from the latest quarter?

If you are already going the route of manually creating a new report every quarter, you can probably just manually specify the start and end date of the period you want in a filter on the table.

"best approach" can depend on the situation, but if it were me I would think about just making one report with a Direct Query connection to your database. Way less admin overhead to maintain. https://learn.microsoft.com/en-us/power-bi/connect-data/desktop-use-directquery 

 

Is it possible to only load data into power query or the model from a certain date like having a column with year and quarter as a key. Loading only rows with this key into power query?

Yes, this is exactly the intent of query folding.

 

Or do I have to create a filter on the dashboard page with a certain date key? But this wouldn't solve the issue that the loaded data size is getting bigger and bigger with time.

☝️

Idrissshatila
Super User
Super User

Hello @Simple_tuition1 ,

For your case you should look at incremental refresh, refer to this link https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview

 

If I answered your question, please mark my post as solution so it would appeare to others, Appreciate your Kudos 👍

 

Follow me on Linkedin



Did I answer your question? Mark my post as a solution! Appreciate your Kudos
Follow me on LinkedIn linkedIn
Vote for my Community Mobile App Idea

Proud to be a Super User!




Helpful resources

Announcements
Europe Fabric Conference

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.