Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
wpf_
Post Prodigy
Post Prodigy

incremental refresh not working - still refreshing whole data set - am I missing something?

I have set up incremental refresh on my table to store only 5 years of data and refresh only the last 3 days of data if it detects data change.  Prior to this step I created a RangeStart and RangeEnd parameters in power query and used it to apply the filter.  I had to move this 'filtered row's' step up because there was a 'Changed Type' step that when i tried to view native query was greyed out, so i left this step last.  All steps before that I can view native query, except for the very first step 'Source'.  

 

When I published to bi service it took the normal time that I expected.  Then when it was refreshing on its scheduled time, it took the same time to refresh.  So I do not think incremental refresh took place.  Is there a step I missed for this to work like doing something in bi service?  

 

Also back in power query in applied steps, the first step 'Source' has view native query greyed out.  Wondering if this is stoping it.  If so how can I fixed this?  Here is the list of my applied steps in order and what it shows in the function bar:

 

Source (view native query greyed out, but can't move this step down):            = Oracle.Database("db-name", [HierarchicalNagivation=true])

Navigation (I can view native query):                = Nameholdingtable{[Name="table-name"]}[Data]

Filtered Rows (I can view native query):            = Table.SelectRows(table-name, each [date-field] >= RangeStart and [date-field] < RangeEnd)

Changed Type (view native query greyed out): = Table.TransformColumnTypes(#"Filtered Rows", {{"Hour-field", Int64.Type}})

6 REPLIES 6
lbendlin
Super User
Super User

What happens when you try to access your dataset via DAX Studio ?

lbendlin
Super User
Super User

You can view the partitions with your favorite XMLA client.  Not sure if XMLA endpoint access is limited to Premium.

 

Be aware that Power BI has its own scheme for partitions based on calendar dates ( months, quarters, years) that may not exactly match your indicated desired settings. But the final result is similar.

@lbendlin I do think premium is required for XMLA enpoint, which I do not have.  Are there any other tools available that I can use to see the partitions without premium?

 

I do know now that the dataset is limited to the 30 days that I have set in incremental refresh settings, so that part is working for sure.  But what I don't know is if it is actually incrementally refreshing.  The reason is because even though I only have 30 days worth of data, the refresh still took 15 minutes to do, even though I only asked to refresh 2 days of it whenever new data is available.  In the desktop a manual refresh took less then a minute and thats loading all 30 days even.  I'm not sure why a scheduled refresh in service takes that long.   How else can i find out if it's incrementally refreshing without premium ? Thanks

lbendlin
Super User
Super User

it is normal that the Source step Native Query is greyed out. In fact, you don't really need Query Folding for incremental refresh. It is preferred but not required.

 

The only surefire way to check if incremental refresh is working is to look at the partitions and their last processed dates.

lbendlin_0-1604441767107.png

 

 

@lbendlin Where do you view the partitions?  Do you need premium capacity for that?

 

When I set up incremental refresh for the table, I set the option "store rows where column "OrderDate" is in the last:" to 30 days.  And I selected the option to detect data changes.  However when I publish to bi service the dates are from the beginning of the year 1/1/2020 to the current date.  Why is it not only for 30 days as I have specified?

Hey mate, did you find the answer to this?? Have the exact same problem

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.