Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
ottspider
Frequent Visitor

Limit data load in Desktop without Parameters

Hi, I have 2 years worth of data (45 million rows of data - c1.4 billion fields) to add to app.powerbi.com.  I have setup the Incremental Refresh Perameters and it appear to work but I can not get the inital limited data set into PowerBI, upon which the refresh will add records.

 

The issue I have is that there is too much data to work with in the Desktop and I am unable to limit the data set by setting paramters for with a posted date range (text ie: '2018-09-28' rather that date/time format) and limit X does not work. 

 

SAMPLE QUERY:

"select *
from TABLE X
where sync_date >= "&Start_Date&" (NOT ALLOWED WITH INCREMENTAL REFRESH)
and sync_date < "&End_Date&";" (NOT ALLOWED WITH INCREMENTAL REFRESH)

 

Parameter can not be changed in app.powerbi when using Incremental Refresh but there is too much data for the desktop version to handle if I increase the date range to 2 years or even 1+ months. 

 

Question - how do I get an inital small dataset into the App if I can not limit the date range (perameters) / volume of data?

   

Thanks in advance Smiley Happy

3 REPLIES 3
v-frfei-msft
Community Support
Community Support

Hi @ottspider,

 

You can use the edit the data in advance to filter the data that you want. 

 

2222.PNG

Regards,

Frank

Community Support Team _ Frank
If this post helps, then please consider Accept it as the solution to help the others find it more quickly.

Thanks for the reply - I am using the Advanced Editor to generate the query table and the date range filter does work to limit the view of data.  However, the date range does not appear to get ignored by the Incremental Refresh when the query is published. 

 

Query.JPG

 

2 Questions -

 

1. Is this correct? -

  • as there is no date range in the above query the inital query will return 45,000,000 rows of data (2 years) - ie. the date range filter is applied once the query has generated the complete data set?
  • the date range may then filter out 1 day / 1 hour of data. 

2. the RangeStart and RangeEnd filter (required for Incremental Refresh) does not appear to be ignored when the report is published and refreshed via On Demand or Schedule.  As such I am unable to get the historic data into the Premium app despite the following Incermental Refresh settings- 

 

Incremental Refresh.JPG

 

In summary - I want all 45 million row in the Premium app but not in the Desktop as it take hours to populate the table / view.  How can I have 1 day / 1 hour of data in the desktop and 2 months / 2 years in the Premium App?

 

Thanks again.

ottspider
Frequent Visitor

Hi, I have 2 years worth of data (45 million rows of data - c1.4 billion fields) to add to app.powerbi.com.  I have setup the Incremental Refresh Perameters correctly and it appear to work but I can not get the inital data to update into PowerBI!

 

The issue I have is the in the desktop there is too much data to work with and I am unable to limit data set by setting paramters for with a posted date range (text ie: '2018-09-28' rather that date/time format) and limit X does not work. 

 

SAMPLE QUERY:

"select *
from TABLE X
where sync_date >= "&Start_Date&" (NOT ALLOWED WITH INCREMENTAL REFRESH)
and sync_date < "&End_Date&";" (NOT ALLOWED WITH INCREMENTAL REFRESH)

 

Parameter can not be changed in app.powerbi, only in the desktop, but there is too much data for the desktop version to handle if I increase the date range to 2 years so how do I get an inital dataset into the App if I can not limit the date range / volume of data?

 

I much be missing something as everytime I update queries I have to limit the dataset so that it does not take hours to 1) view the table and 2) Apply query updates. 

 

Any thoughts welcome and I've tried everything I can think of!

 

Thanks in advance 🙂

 

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.