Import from local vs import from BigQuery

[rafael-b]

Hi all,

I'm currently working with data that comes in form of one CSV file per month, containing over 15mn rows each. I'm required to have at data going back to at least January 2018, so that's about 400mn rows now. I have compressed each month's file into parquet, kept them in my local disk, and then I import each one to PBI, where I append them all to make visuals.

The load process to PBI is currently very slow and I'm looking into incremental refresh for next month's data. Maybe you already have some suggestions regarding the steps I currently go through, but that's not the main question here.

As I said, I'm keeping the files on my disk and I don't think that I'll be able to do this forever. The other option I see is uploading the tables to BigQuery (my company uses Google Cloud Platform already) and importing them from there (not Live Query). I'm wondering if the import and refresh from BigQuery will be much slower then importing from the disk. If so, would you have suggestions regarding the file storage and the imports to PBI?

Thanks!!

[parry2k]

@rafael-b did you looked at synapse and blob storage.  This will be super fast in terms of performance and data loading. Query Parquet files using serverless SQL pool - Azure Synapse Analytics | Microsoft Docs

✨ Follow us on LinkedIn

Check my latest blog post Comparing Selected Client With Other Top N Clients | PeryTUS  I would ❤ Kudos if my solution helped. 👉 If you can spend time posting the question, you can also make efforts to give Kudos to whoever helped to solve your problem. It is a token of appreciation!

⚡ Visit us at https://perytus.com, your one-stop-shop for Power BI-related projects/training/consultancy.⚡