cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Greg_Davurse
Helper I
Helper I

Data Refresh Storage and Traffic Concerns

I have a couple questions concerning the data refresh when connecting to an on-premise SQL server/DB. 

 

Once the report has been made and the Refresh Schedule has been set, how is the data refreshed and stored? How is the Power BI Service updating the data and where is it stored? For Example: if I have 10,000,000 records, does the PBI Service refresh all 10,000,000 records and then some if additional records have been added? If so, this brings me to my next main question...Will this/can this cause latency issues on the server do to a large amount of data traffic since Power BI is pulling a large amount of data from the Server/DB, and if so, is there any documention on best practices or does anyone have any advice when setting up these refreshes?

 

I apologize in advance if I've made any assumptions in my question. Please correct if I have misinterpretted anything. 

1 ACCEPTED SOLUTION
Greg_Deckler
Super User
Super User

If this is an on-premise database, then Power BI Service uses an on-premises gateway to perform the data load. Currently, Power BI does not have incremental load so all 10,000,000 records plus additional would be refreshed. If that original data never changes and latency is a concern what I would do would be to create a view in your database and 2 or 3 queries. Essentially, create one query that grabs all of your original data. Then, create a view that only shows new data. Create another query to grab that data from that view. Append the two queries. This way, you will only refresh a subset of your data and not all 10,000,000 records every time. 

 

You could also use Direct Query, which pulls the data real time instead of importing.


@ me in replies or I'll lose your thread!!!
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
Mastering Power BI 2nd Edition

View solution in original post

2 REPLIES 2
Greg_Deckler
Super User
Super User

If this is an on-premise database, then Power BI Service uses an on-premises gateway to perform the data load. Currently, Power BI does not have incremental load so all 10,000,000 records plus additional would be refreshed. If that original data never changes and latency is a concern what I would do would be to create a view in your database and 2 or 3 queries. Essentially, create one query that grabs all of your original data. Then, create a view that only shows new data. Create another query to grab that data from that view. Append the two queries. This way, you will only refresh a subset of your data and not all 10,000,000 records every time. 

 

You could also use Direct Query, which pulls the data real time instead of importing.


@ me in replies or I'll lose your thread!!!
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
Mastering Power BI 2nd Edition

Thanks! Your explanation helps and answers my question. 

Helpful resources

Announcements
August 2022 update 768x460.jpg

Check it Out!

Click here to learn more about the August 2022 updates!

August 1 episode 9_no_dates 768x460.jpg

The Power BI Community Show

Watch the playback when Priya Sathy and Charles Webb discuss Datamarts! Kelly also shares Power BI Community updates.

Power Platform Conf 2022 768x460.jpg

Join us for Microsoft Power Platform Conference

The first Microsoft-sponsored Power Platform Conference is coming in September. 100+ speakers, 150+ sessions, and what's new and next for Power Platform.

Top Solution Authors