cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Highlighted
ottspider Frequent Visitor
Frequent Visitor

Limit data load in Desktop without Parameters

Hi, I have 2 years worth of data (45 million rows of data - c1.4 billion fields) to add to app.powerbi.com.  I have setup the Incremental Refresh Perameters and it appear to work but I can not get the inital limited data set into PowerBI, upon which the refresh will add records.

 

The issue I have is that there is too much data to work with in the Desktop and I am unable to limit the data set by setting paramters for with a posted date range (text ie: '2018-09-28' rather that date/time format) and limit X does not work. 

 

SAMPLE QUERY:

"select *
from TABLE X
where sync_date >= "&Start_Date&" (NOT ALLOWED WITH INCREMENTAL REFRESH)
and sync_date < "&End_Date&";" (NOT ALLOWED WITH INCREMENTAL REFRESH)

 

Parameter can not be changed in app.powerbi when using Incremental Refresh but there is too much data for the desktop version to handle if I increase the date range to 2 years or even 1+ months. 

 

Question - how do I get an inital small dataset into the App if I can not limit the date range (perameters) / volume of data?

   

Thanks in advance Smiley Happy

3 REPLIES 3
ottspider Frequent Visitor
Frequent Visitor

Incremental Refresh - how to load initial 45 million rows of data that is to be refreshed?

Hi, I have 2 years worth of data (45 million rows of data - c1.4 billion fields) to add to app.powerbi.com.  I have setup the Incremental Refresh Perameters correctly and it appear to work but I can not get the inital data to update into PowerBI!

 

The issue I have is the in the desktop there is too much data to work with and I am unable to limit data set by setting paramters for with a posted date range (text ie: '2018-09-28' rather that date/time format) and limit X does not work. 

 

SAMPLE QUERY:

"select *
from TABLE X
where sync_date >= "&Start_Date&" (NOT ALLOWED WITH INCREMENTAL REFRESH)
and sync_date < "&End_Date&";" (NOT ALLOWED WITH INCREMENTAL REFRESH)

 

Parameter can not be changed in app.powerbi, only in the desktop, but there is too much data for the desktop version to handle if I increase the date range to 2 years so how do I get an inital dataset into the App if I can not limit the date range / volume of data?

 

I much be missing something as everytime I update queries I have to limit the dataset so that it does not take hours to 1) view the table and 2) Apply query updates. 

 

Any thoughts welcome and I've tried everything I can think of!

 

Thanks in advance Smiley Happy

 

Community Support Team
Community Support Team

Re: Limit data load in Desktop without Parameters

Hi @ottspider,

 

You can use the edit the data in advance to filter the data that you want. 

 

2222.PNG

Regards,

Frank

Community Support Team _ Frank
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
ottspider Frequent Visitor
Frequent Visitor

Re: Limit data load in Desktop without Parameters

Thanks for the reply - I am using the Advanced Editor to generate the query table and the date range filter does work to limit the view of data.  However, the date range does not appear to get ignored by the Incremental Refresh when the query is published. 

 

Query.JPG

 

2 Questions -

 

1. Is this correct? -

  • as there is no date range in the above query the inital query will return 45,000,000 rows of data (2 years) - ie. the date range filter is applied once the query has generated the complete data set?
  • the date range may then filter out 1 day / 1 hour of data. 

2. the RangeStart and RangeEnd filter (required for Incremental Refresh) does not appear to be ignored when the report is published and refreshed via On Demand or Schedule.  As such I am unable to get the historic data into the Premium app despite the following Incermental Refresh settings- 

 

Incremental Refresh.JPG

 

In summary - I want all 45 million row in the Premium app but not in the Desktop as it take hours to populate the table / view.  How can I have 1 day / 1 hour of data in the desktop and 2 months / 2 years in the Premium App?

 

Thanks again.

Helpful resources

Announcements
Community Highlights

Community Highlights

Find out what's new in the Power BI Community!

Summit North America

Power Platform Summit North America

Register by September 5 to save $200

Virtual Launch Event

Microsoft Business Applications Virtual Launch Event

Watch the event on demand for an in-depth look at the new innovations across Dynamics 365 and the Microsoft Power Platform.

MBAS Gallery

Watch Sessions On Demand!

Continue your learning in our online communities.

Users Online
Currently online: 97 members 1,515 guests
Please welcome our newest community members: