cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Lyhis
Helper IV
Helper IV

Working with big datasets.

Hi

 

I need to create a new report and use incremental refresh in the future. For that I can not use custom SQL while pulling data from data warehouse because then incremental refresh will not work. Problem is I need data from tables that have data for 10+ years, I only need last 6 months worth of data. Can I filter the data in these tables without custom SQL before pulling the data? I can not overload this data warehouse pulling 10+ years of data and after that using filter. My only idea once again is creating a tabular model to another server but that's extra step for me for creating just 1 report.

4 REPLIES 4
judgejebber
Regular Visitor

Just create a sql view and select it like you would a table. Then incremental works just fine and can query fold succesfully.

If I create a view with a WHERE condition going back to almost a year doesn't that mean it still has to go through that same query that I am trying to avoid? This query takes too much time and resources on the data warehouse.

lbendlin
Super User
Super User

please provide more context for "I can not use custom SQL"

 

When you filter in Power Query it will attempt to push that filter upstream via "Query folding"  aka "native query".

 

please clarify " because then incremental refresh will not work " - that is not necessarily related.

With "Custom SQL" I meant when geting data. Get data-> Sql server. Under advanced options "SQL statement (optional, requires database)" I've learned that if I have anything written there then incremental refresh wouldn't work, so far this is the way I have been geting the data I need without downloading everything. Now I need another way since I have to get incremental refresh working.

Helpful resources

Announcements
UG GA Amplification 768x460.png

Launching new user group features

Learn how to create your own user groups today!

Power BI October Update 2021.jpg

Power BI Release

Click here to read more about the October 2021 Release!

Community Connections 768x460.jpg

Community & How To Videos

Check out the new Power Platform Community Connections gallery!

Teds Dev Camp Oct. 2021 768x460.jpg

Power BI Dev Camp - October 28th, 2021

Mark your calendars and join us for our next Power BI Dev Camp!