cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Greg_Davurse
Helper I
Helper I

Data Refresh Storage and Traffic Concerns

I have a couple questions concerning the data refresh when connecting to an on-premise SQL server/DB. 

 

Once the report has been made and the Refresh Schedule has been set, how is the data refreshed and stored? How is the Power BI Service updating the data and where is it stored? For Example: if I have 10,000,000 records, does the PBI Service refresh all 10,000,000 records and then some if additional records have been added? If so, this brings me to my next main question...Will this/can this cause latency issues on the server do to a large amount of data traffic since Power BI is pulling a large amount of data from the Server/DB, and if so, is there any documention on best practices or does anyone have any advice when setting up these refreshes?

 

I apologize in advance if I've made any assumptions in my question. Please correct if I have misinterpretted anything. 

1 ACCEPTED SOLUTION
Greg_Deckler
Super User
Super User

If this is an on-premise database, then Power BI Service uses an on-premises gateway to perform the data load. Currently, Power BI does not have incremental load so all 10,000,000 records plus additional would be refreshed. If that original data never changes and latency is a concern what I would do would be to create a view in your database and 2 or 3 queries. Essentially, create one query that grabs all of your original data. Then, create a view that only shows new data. Create another query to grab that data from that view. Append the two queries. This way, you will only refresh a subset of your data and not all 10,000,000 records every time. 

 

You could also use Direct Query, which pulls the data real time instead of importing.


@ me in replies or I'll lose your thread!!!
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
Learn Power BI 2nd Edition

View solution in original post

2 REPLIES 2
Greg_Deckler
Super User
Super User

If this is an on-premise database, then Power BI Service uses an on-premises gateway to perform the data load. Currently, Power BI does not have incremental load so all 10,000,000 records plus additional would be refreshed. If that original data never changes and latency is a concern what I would do would be to create a view in your database and 2 or 3 queries. Essentially, create one query that grabs all of your original data. Then, create a view that only shows new data. Create another query to grab that data from that view. Append the two queries. This way, you will only refresh a subset of your data and not all 10,000,000 records every time. 

 

You could also use Direct Query, which pulls the data real time instead of importing.


@ me in replies or I'll lose your thread!!!
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
Learn Power BI 2nd Edition

Thanks! Your explanation helps and answers my question. 

Helpful resources

Announcements
Power Platform Conf 2022 768x460.jpg

Join us for Microsoft Power Platform Conference

The first Microsoft-sponsored Power Platform Conference is coming in September. 100+ speakers, 150+ sessions, and what's new and next for Power Platform.

Power BI Dev Camp Session 23 768x460.jpg

Check it Out!

Mark your calendars and join us on Thursday, June 30 at 11a PDT for a great session with Ted Pattison!

June 20 episode 7 with aka link 768x460.jpg

The Power BI Community Show

Join us on June 20 at 11 am PDT when Kim Manis shares the latest on Azure Synapse analytics, the Microsoft Intelligent Data Platform, and notable Power BI Updates from Microsoft Build 2022.

Top Solution Authors