Hi community!
I'm working in a project where my source files are all excel/csv files and I'm having trouble to build the model due to memory capacity.
Project Objetive: create a model for end users to analyze data. They prefer using Power BI Desktop rather than Power BI Service.
Source Files
- CSV Files: 50 files, around 4GB and 44 Million rows. This is updated every week with one file of 1 Million rows and 120 Mb aprox. This is hosted in a sync one drive in each user PC.
- Excel File: 150 thousand rows, 10Mb file. This is hosted in a sync one drive in each user PC.
- 2 more Excel Files with 3-5 thousand rows and 3Mb aprox. This is being access through SharePoint URL, users have access.
Problem
I've managed to build the model and make it work in a PC with 32 GB of RAM, initial load is quite heavy, but once done the update of each new CSV file is quick. However when trying to deploy this in each user PC I'm getting out of memory and can't load the data to the model.
Help Needed:
- Advice on the architecture. Maybe this is the worst strategy to build the model and should directly consider uploading the CSV files to another format/place (maybe a local DB).
- Conclude to users that Power BI Desktop is not an option and should consume the report from Power BI Service.
- Other ideas that I'm not considering.
Help with this would be much appreciated :).
Thanks in advance!
