Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
Hi everybody,
I've got Pro license. I've got a folder collecting CSVs from today to the past 12 month, 1 month is approx. 1,5GB. I could use directly folder source, but I'm using it through a dataflow. The refresh is very slow, in the next months I will may be over the 10GBs limit and I'm occupying Microsoft's and my bandwidth in a way that is nonsense, because this data could be subjected to incremental refresh day by day (production machines log). Premium capacity has no reason to be considered in relation to the smallness of the project and this only one necessity.
Does anyone have a best practice to extract and transfer data in csv in a compressed way? Do I need to make a program to inject these CSVs in a database that acts as a more compressed datasource?
Any advice?
Thanks very much in advance
Solved! Go to Solution.
@AGo You could spin up an Azure Analysis Services instance and build the model in there for cheaper. Then you could do a live connection from Power BI to that model without incurring the larger overhead. This would allow you to scale as well at a lower price point.
I wonder if you could ZIP them and read them into a dataflow from a OneDrive/SharePoint folder:
https://community.powerbi.com/t5/Desktop/PowerQuery-read-from-folder-with-many-zip-file/td-p/57334
https://github.com/hohlick/Power-Query-Excel-Formats/blob/master/UnZip.pq
Not sure of performance on files that size especially with a custom M function.
10 GB limit refers to in-memory uncompressed...and not including any memory overhead needed.
I've stuggled getting .pbix's greater than 3 GB to publish to Premium.
I was thinking of published data model size limitation...I see now you probably meant workspace storage limitation.
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the April 2024 Power BI update to learn about new features.