Problem with refreshing the data sources

[Anonymous]

Hello Guys, 

I have a problem with refreshing the data source.

I am importing files from Sharepoint , individual file for each day , and I Combine them by powerquery.

But I made a filter before appending ( file starts with some characters ) 

But when I add another file to the same folder , and refresh power bi model , it doesn't add the new file 

It works only when i enter to the power query editor and click > refresh preview

But i want to automate it to work with the schedualed refresh on the online service

any help please ?

the first pic is after clicking refresh on the bi file itself and didn't refreshed all files in the folder

the second pic is after clicking refresh preview in the power query editor

[BA_Pete · Accepted solution]

Hi @Anonymous ,

This is how Power Query is supposed to work.

When you refresh data, even though the preview in Power Query doesn't update, the new data will actually be loaded into your model for use. I guess this is to speed up the data refresh process so the system doesn't waste time refreshing the query previews at the same time.

If you go into Options & settings, you can adjust whether Power Query performs background preview refreshes while you're working. This may help alleviate your concerns as then your previews will update as you're working on other queries.

Pete

[v-shex-msft · Accepted solution]

HI @Anonymous,

According to your description, I think it should more related to 'lazy Evaluation' that appears on refresh/processing with m query formulas. You can take a look at the below blog to know more about this:

Lazy Evaluation & Query Folding in Power BI / Power Query 

In addition, you can also refer to BA_Pete 's suggestion to check the option to allows 'data previews to download in the backend' if they fixed the issue.

Regards,

Xiaoxin Sheng

[Anonymous · Accepted solution]

Thank You guys for helping me out, But unfortunately all the sources you mentioned didn't help

However, I found a solution after a long search.

instead of getting the data using SharePoint Folder , you should use a blank query and type 

= SharePoint.Contents(" # Sharepoint Folder ")

this should work and refresh automatically after any change in the SharePoint folder without the need of manual click Refresh

[v-shex-msft · Accepted solution]

HI @Anonymous,

According to your description, I think it should more related to 'lazy Evaluation' that appears on refresh/processing with m query formulas. You can take a look at the below blog to know more about this:

Lazy Evaluation & Query Folding in Power BI / Power Query 

In addition, you can also refer to BA_Pete 's suggestion to check the option to allows 'data previews to download in the backend' if they fixed the issue.

Regards,

Xiaoxin Sheng

[BA_Pete · Accepted solution]

Hi @Anonymous ,

This is how Power Query is supposed to work.

When you refresh data, even though the preview in Power Query doesn't update, the new data will actually be loaded into your model for use. I guess this is to speed up the data refresh process so the system doesn't waste time refreshing the query previews at the same time.

If you go into Options & settings, you can adjust whether Power Query performs background preview refreshes while you're working. This may help alleviate your concerns as then your previews will update as you're working on other queries.

Pete

[amitchandak]

@Anonymous , refer if these can help

https://powerbi.microsoft.com/en-us/blog/combining-excel-files-hosted-on-a-sharepoint-folder/

https://medium.com/@wkrzywiec/getting-data-from-the-latest-file-in-a-folder-using-power-query-51dfa4bff711

https://blog.crossjoin.co.uk/2020/04/13/keep-the-existing-data-in-your-power-bi-dataset-and-add-new-data-to-it-using-incremental-refresh/

https://www.thebiccountant.com/2017/01/11/incremental-load-in-powerbi-using-dax-union/