Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more.
Get startedGrow your Fabric skills and prepare for the DP-600 certification exam by completing the latest Microsoft Fabric challenge.
Hi all,
I'm currently working with data that comes in form of one CSV file per month, containing over 15mn rows each. I'm required to have at data going back to at least January 2018, so that's about 400mn rows now. I have compressed each month's file into parquet, kept them in my local disk, and then I import each one to PBI, where I append them all to make visuals.
The load process to PBI is currently very slow and I'm looking into incremental refresh for next month's data. Maybe you already have some suggestions regarding the steps I currently go through, but that's not the main question here.
As I said, I'm keeping the files on my disk and I don't think that I'll be able to do this forever. The other option I see is uploading the tables to BigQuery (my company uses Google Cloud Platform already) and importing them from there (not Live Query). I'm wondering if the import and refresh from BigQuery will be much slower then importing from the disk. If so, would you have suggestions regarding the file storage and the imports to PBI?
Thanks!!
@rafael-b did you looked at synapse and blob storage. This will be super fast in terms of performance and data loading. Query Parquet files using serverless SQL pool - Azure Synapse Analytics | Microsoft Docs
✨ Follow us on LinkedIn
Check my latest blog post Comparing Selected Client With Other Top N Clients | PeryTUS I would ❤ Kudos if my solution helped. 👉 If you can spend time posting the question, you can also make efforts to give Kudos to whoever helped to solve your problem. It is a token of appreciation!
⚡ Visit us at https://perytus.com, your one-stop-shop for Power BI-related projects/training/consultancy.⚡
Subscribe to the @PowerBIHowTo YT channel for an upcoming video on List and Record functions in Power Query!!
Learn Power BI and Fabric - subscribe to our YT channel - Click here: @PowerBIHowTo
If my solution proved useful, I'd be delighted to receive Kudos. When you put effort into asking a question, it's equally thoughtful to acknowledge and give Kudos to the individual who helped you solve the problem. It's a small gesture that shows appreciation and encouragement! ❤
Did I answer your question? Mark my post as a solution. Proud to be a Super User! Appreciate your Kudos 🙂
Feel free to email me with any of your BI needs.
Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.
Check out the June 2024 Power BI update to learn about new features.
User | Count |
---|---|
93 | |
87 | |
80 | |
69 | |
68 |
User | Count |
---|---|
226 | |
129 | |
119 | |
83 | |
77 |