Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
Hi,
I have data imported from bigquery using the SIMBA ODBC driver. My organisation has power BI premium capcity. There are billions of rows in the views as the data is in long format.
Data structure preview: There are around 50,000 SKUs tracked across 40 regions on a daily basis. This is around 2million rows with dates as column headers. Since this structure is not supported for visualisation, data is unpivoted for visualisation purpose. This makes the number of rows go upto 300millions for 5months data. I'm now concerned about the increasing 2million rows daily and the dashboard is expected to be in place for at least an year.
Here comes the major issue. I wish to refresh the data on a daily basis and below are the few roadblocks:
Could someone please help me in scheduling a successful daily refresh for this huge dataset? Any suggestions would be greatly appreciated
Regards,
Swetha
Solved! Go to Solution.
Hi @swethabonthu ,
You may following those tips to reduce the size of dataset or optimize the model of dataset based on this document, some tips may not reduce the time of refresh.
Or you can increase the timeout value in connector function.
Best Regards,
Jay
Community Support Team _ Jay Wang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
You can use PBI Spy (www.pbispy.com) to quickly identify columns that are not being used.
Reach out to me if you need any assistance doing this.
OMG! I wish I would have known about this before!!
I've been given a report with 5 tables, each one with about 350 columns, I was going crazy trying to figure out what data was really used.
THANK YOU!
tables with 350 columns? I wonder you use large datasets storage format on premium capacities? is it ok with model performance ? I mean P2 OR P3 premium capacity can keep good performance for large datasets with 350+columns talbes.
Glad to be of assistance. This is the first real public release of PBI Spy. If anything does not work or is unclear please don't hesitate to contact me. I'm trying to make it as good as it can be, but it's a journey.
Hi @swethabonthu ,
You may following those tips to reduce the size of dataset or optimize the model of dataset based on this document, some tips may not reduce the time of refresh.
Or you can increase the timeout value in connector function.
Best Regards,
Jay
Community Support Team _ Jay Wang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Please explain the reasoning for not using incremental refresh in a bit more detail. Do your data rows come with a "last modified date" tag?
Data imported from bigquery uses complex machine learning techniques and the historic data is not fixed. Also, I don't have "last modified date" tag
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the April 2024 Power BI update to learn about new features.