Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
I require 3 years worth of data (600,000 rows) from a IBM db2 tableview. Each day an additional 100-200 lines are added to the database. In order to speed up data load processing times I split the data betwen two queries...the first is a query of an excel file of exported years 2020 through 2021 data, connection only, "Refresh this connection on Refresh All" box unchecked. The second is a query of the tableview which provides the dynamic 2022 data. I then merge the 2 data sets and export to data model.
I expected this to be faster than a query of all 3 years worth of data from the tableview but am experiencing very similar data load times between the two methods. Am I approaching this the right way?
Thanks
Solved! Go to Solution.
Unfortunately, whenever you refresh, the append step is triggering refresh of your static data too. If not already, I would look into incremental refresh on your original IBM db2 query (no Excel).
Pat
To learn more about Power BI, follow me on Twitter or subscribe on YouTube.
Hi, @troyhimes
May I ask if your problem has been solved? Is the above post helpful to you?
If it does, could you please mark the post which help as Answered? It will help the others in the community find the solution easily if they face the same problem with you. Thank you.😀
Best Regards,
Community Support Team _ Zeon Zheng
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Unfortunately, whenever you refresh, the append step is triggering refresh of your static data too. If not already, I would look into incremental refresh on your original IBM db2 query (no Excel).
Pat
To learn more about Power BI, follow me on Twitter or subscribe on YouTube.
Hi Pat, thanks for the info regarding the append process. What do you mean by 'incremental refresh'?
Instead of appending the two queries is there a downside to 'linking' the tables in the data model through a common date table?
Incremental refresh is a special kind of refresh you can set up in the Service.
See these resources:
https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview
https://radacad.com/all-you-need-to-know-about-the-incremental-refresh-in-power-bi-load-changes-only
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the April 2024 Power BI update to learn about new features.
User | Count |
---|---|
100 | |
51 | |
19 | |
12 | |
11 |