cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Highlighted
Frequent Visitor

Create historic data table with Web incoming data

Hello all. I did a lot of research here and so far i couldn't find any suitable solution. I create a table in Power Query Editor that contains sensor data directly from Web. This data is a CSV-dat file and usually updated with new lines every 2 hours. The "current" is the month May in this excercise - see photo attached. In June this file will contain only "June" data and the "May" data will then be available under a different file name. The files are small (<1000 rows) so data handling does not seem to be a issue.

 

Now i want to create Visual Graphs for this hourly-updated data and publish the report. Is there any way - in every refresh - to collect the newly incoming data into a table ?  I want to to automate the process and not having to manually load/join tables every month.

 

"Incremental refresh" here would not work as the incoming data contain only the current month.

Other commands such as "Union" will also not work.

 

I hope the above was clear - any input and guidance appreciated! Regards

 

 1.jpg

2 ACCEPTED SOLUTIONS

Accepted Solutions
Highlighted
Frequent Visitor

Re: Create historic data table with Web incoming data

Dear Greg and all,

i finally managed to do it using a Dataflow and a Dataset. The i schedule refresh intervals for both and the report is always up to date.

More information on how to do it you can find under this page - which was really heplful for me :

https://exceed.hr/hr/blog/incremental-refresh-using-powerbi-pro-and-dataflows 

 

So far i don't see any drawbacks with this approach apart from the fact tha the data update is a bit slow.

 

Antonis

View solution in original post

Highlighted
Frequent Visitor

Re: Create historic data table with Web incoming data

Dear Greg,

 

i finally managed to do it using a combination of dataflows and datasets. The update interval has to be set with a difference of 30 minutes in order for this to work correctly.

Some instructions can be found under this link https://exceed.hr/hr/blog/incremental-refresh-using-powerbi-pro-and-dataflows 

View solution in original post

4 REPLIES 4
Highlighted
Super User IV
Super User IV

Re: Create historic data table with Web incoming data

So generally no. What I would recommend would be to setup a process via Power Automate or other means by which files are copied to a single folder and given a unique name. Then use a Folder query in Power BI.


---------------------------------------

Putting square pegs in round holes since 1972.

I have a NEW book! 
DAX Cookbook from Packt
Over 120 DAX Recipes!




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!




Highlighted
Frequent Visitor

Re: Create historic data table with Web incoming data

I see...it's getting really complicated then..Thanks for the answer.

Highlighted
Frequent Visitor

Re: Create historic data table with Web incoming data

Dear Greg and all,

i finally managed to do it using a Dataflow and a Dataset. The i schedule refresh intervals for both and the report is always up to date.

More information on how to do it you can find under this page - which was really heplful for me :

https://exceed.hr/hr/blog/incremental-refresh-using-powerbi-pro-and-dataflows 

 

So far i don't see any drawbacks with this approach apart from the fact tha the data update is a bit slow.

 

Antonis

View solution in original post

Highlighted
Frequent Visitor

Re: Create historic data table with Web incoming data

Dear Greg,

 

i finally managed to do it using a combination of dataflows and datasets. The update interval has to be set with a difference of 30 minutes in order for this to work correctly.

Some instructions can be found under this link https://exceed.hr/hr/blog/incremental-refresh-using-powerbi-pro-and-dataflows 

View solution in original post

Helpful resources

Announcements
June 2020 Community Highlights

June 2020 Community Highlights

Featured community members, changes to the Community, and more! Read up on recent Power BI community news.

Using the Community

Using the Community

Need help with the Power BI Community? Our 'Using the Community' support articles are a great place to start.

Community Summit North America

Community Summit North America

Innovate, Collaborate, Grow. The top training and networking event across the globe for Microsoft Business Applications

Power Platform 2020 release wave 2 plan

Power Platform 2020 release wave 2 plan

Features releasing from October 2020 through March 2021

Top Solution Authors
Top Kudoed Authors