Refresh only works once, then runs out of memory

[Mulder2542]

When working with a relatively big dataset (50M+ records), I've run into the situation where the initial refresh (after publishing) works fine every time, but any subsequential refresh, either scheduled or on demand, runs out of memory. It's not a complex dataset and a successful refresh takes about 2 hours.

What can be the reason for this? Is the initial refresh transforming/processing the data (cache? indexing?) somehow, so it takes up more memory in the following updates? 

[ibarrau]

Hi. It sounds like you are reaching the limits. Maybe the first time with the publish is not considering the limitations of refreshing at Power Bi Service. You have a limit of 2 hours to refresh and a maximum of 1gb model size. Then a single table size can't be of 10gb before starting the transformations either. You might have one of those limitations. If it's a memory one it might be that you have a big size model. I would suggest watching some videos about performance tunning or downsizing dataset. There are some keys like building a good data model and other practices.

I hope that helps,

[Mulder2542]

Thanks for your reply (and sorry for my late reply). After talking with some colleagues, I've been told that the issue is that the initial refresh uses less memory because Power BI apparently keeps the old data fully cached during the refresh, so it basically will use twice the amount of memory when a dataset is already in the workspace. Maybe somebody here can confirm this?

[ibarrau]

Make sense. I don't think I have read that in the doc ever, but that's how Azure Analysis Services usually works by default. It saves a backup the second type in case it fails you can still check the previous refresh data. So the first time it's a somehow smaller the amount of memory.

Regards,