Power BI premium Large Dataset performance

[gmckinnon]

Hi, we have a large dataset (around 8gb compressed) hosted in a P1 capacity still currently in premium Gen1. Currently we load a subset of the data into pbi desktop and use incremental refresh to load the historical data and new data. 

The dataset will be growing and we want to work out how scaleable this solution is, esspecailly when we move to premium gen 2 and memory is less of an issue and we just need to worry about CPU.

The biggest issue we see right now is when the dataset needs to be loaded into memory which currently takes about 10-15 seconds and seeing spikes in the CPU useage of around 80% during this time.

If the dataset was to grow, how would it perform? Would we need to revert to filtering the data or using direct query (which was terribly slow). If the dataset grew to say 25gb (the max for a p1) could it overload the CPU when moving in and out of memory in the service?

[TomMartens · Accepted solution]

Hey @gmckinnon ,

this is a difficult question, but there is at least one very simple answer.

Keep in mind that you can't leverage the full 25GB of the available dataset size, as there is extra memory needed during processing.

Behavior will change with Premium Gen 2 (What is Microsoft Power BI Premium? - Power BI | Microsoft Docs), i recommend to turn on the preview feature, and gather your experiences, before you migrate to PE2.

Regards,

Tom