cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
seancody
Regular Visitor

Update Power BI historical data daily

I collect data daily via URL download to csv (standard tabular format). I use Excel Power Query to transform and summarise this data (groupby row count by column header name) so that it is crunched from rows in excess of 100,000 down to hundreds (Table 1). I then create a second Excel Power Query which adds the contents of Table 1 to an historic version of the same data (Table 2). Finally, I use another Power Query, this time in Power BI, to draw the contents of Table 2 to Power BI where I visualise and share the summarised, historic data with stakeholders. Clearly this is a very manual exercise, but I have yet to find an alternative daily update methodology which simplifies the entire process (in particular, the historic update with daily data). Ideally, I need a process which operates automatically, daily, directly from source to Power BI to stakeholder (ie without needing to do the transformation in Excel)

Any advice or alternative courses of action much appreciated.

Thanks.
PS Recently I have connected the external URL above to MS Dataverse (so that every new line item in the source data is automatically collected real time). MS Dataverse can be connected directly to Power BI (but not to Excel as far as I can see).

1 ACCEPTED SOLUTION
lbendlin
Super User
Super User

Power BI has no memory. You need to do that aggregation and merging in the upstream system.

View solution in original post

3 REPLIES 3
v-stephen-msft
Community Support
Community Support

Hi @seancody ,


Could you tell me if your problem has been solved?
If it is, kindly Accept it as the solution. More people will benefit from it.
Or you are still confused about it, please provide me with more details about your problem.


Best Regards,
Stephen Tao

 

v-stephen-msft
Community Support
Community Support

Hi @seancody ,

 

It sounds like you need to configure scheduled refresh, please kindly refer to Configure scheduled refresh - Power BI | Microsoft Docs.

In addition, you can clean and transform data directly in Power Query of Power BI. There is no limitation on the rows of data you can import into Power BI currently.

 

Best Regards,

Stephen Tao

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

lbendlin
Super User
Super User

Power BI has no memory. You need to do that aggregation and merging in the upstream system.

Helpful resources

Announcements
Microsoft Build 768x460.png

Microsoft Build is May 24-26. Have you registered yet?

Come together to explore latest innovations in code and application development—and gain insights from experts from around the world.

May 23 2022 epsiode 5 without aka link.jpg

The Power BI Community Show

Welcome to the Power BI Community Show! Jeroen ter Heerdt talks about the importance of Data Modeling.

Power BI Dev Camp Session 22 with aka link 768x460.jpg

Check it out!

Mark your calendars and join us on Thursday, May 26 at 11a PDT for a great session with Ted Pattison!

charticulator_carousel_with_text (1).png

Charticulator Design Challenge

Put your data visualization and design skills to the test! This exciting challenge is happening now through May 31st!