Dataflow Scheduled Refresh Doesn't Load Recent Data

[arusky]

Hi,

We created some dataflows to feed the data into many datasets.

We felt that it would be more efficient instead of refreshing directly from the server multiple times for the same source (Snowflake).

However, I noticed that the dataflow doesn't load the recent data.

For example, as of today, 10/18/2022, the data should be up to 10/17/2022.

When I open the report, the data is only up to 10/15/2022.

So since 10/16/2022 refresh, yesterday and today's refresh did not load the complete data.

The dataflow scheduled refresh result says all were successful.

To make sure, I scheduled the dataflow refresh to be multiple times.

When I manually refresh, it seems to load everything.

By the way, it is not an issue from Snowflake or in between Snowflake and PBI server because another report with direct refresh from Snowflake brought the complete data.

Does this happen to anyone?

[NorfolkDumpling · Accepted solution]

We had a similar issue with incremental refresh recently.

The table had a policy to store rows from the past 1 year and refresh the past 1 month of data. We also had the Detect data changed box ticked and using an Amended_DateTime field to detect data changes.

We were using a Created_DateTime field to filter by and it was thought this would be sufficient to detect new records on its own. For a newly created record, the software would set the Amended_DateTime to null logically  because the record hadn't yet been changed. 

What we found in test was, the initial load of some sample data was fine as was the first subsequent refresh but further refreshes would not recognise new rows until the record was changed and Amended_DateTime was set to a non null value. In our case, this was happening several days later on the live system. 

The solution was to change the source program so that when a new record is created then both the Created_DateTime and Amended_DateTime fields are set to the same value. 

[v-luwang-msft]

Hi @arusky ，

Pls check if you have set up a refresh schedule for the report's dataset. 

Similar error rfer:

https://community.powerbi.com/t5/Service/Report-using-dataflow-as-data-source-not-getting-updated-da... 

Best Regards

Lucien

[arusky]

Thank you, @v-luwang-msft for your response.

However, my case is not in this. The scheduled refresh of the dataset has been running fine as well.

Since the dataflow refresh takes less than 30 min, I scheduled all the dataset to be refreshed at least after 30 min from the dataflow refresh.