Limitation of data size in the Service

[MarkCBB]

Hi there,

I have been asked to create a Power BI solution that will contain a rather large dataset.

I understand that the file size limit is 1GB in Pro and 10GB for Premium. (and that a Workspace has a limit of 10GB) However, I am very certain that even after being compressed in the Desktop I will go over the 10GB limit.  

Is there any way around this limitation, I have happy to use Azure or any solution that allows. I am currently on Pro, but also happy to get Premium. 

I don't think that DQ will work as the performance on a large set will also down the user experience.

Has anyone got any advice or has anyone been able to create a PBI solution of this size?

Regards,

Mark Blackburn 

[AmirNetz]

The compression of Power BI is generally very good, especially with massive amounts of data.

Did you try to load the data to the desktop. We have seen many cases where hundreds of millions of rows can be stored with under 10GB. Sometimes even billions.

If there is a problem, you may want to see if a problematic column can be removed to make it fit. Columns that are many distinct values are harder to compress.

[MarkCBB]

Hello @AmirNetz and @v-yulgu-msft,

Thank you for getting back to me.

I found this blog from Reza Rad http://radacad.com/step-beyond-the-10gb-limitation-of-power-bi 

It looks like this approach is a bit technical for me, but I can always try to learn.

With regards to the tables:

Product table has 20 columns, and the ID column has 19 000 distinct values. (Text)

Place/Store Table: has 18 columns and the ID column has 10 000 distinct values. (Text)

The fact table contains 10 columns and an estimated 50 billion rows.

thankfully there is no Fast ID column, only the above-mentioned ID columns. Also a Date, Units, Value, Stock, Orders, Stock Cost, Status (int), and Range (int)

I don't think that changing the Date to a DateKey is going to make much of a difference, but I am going to try that this week.

[AmirNetz · Accepted solution]

The cadinality is very reasonable. But even with great compression 50B rows is probably not going to fit in 10GB.

So at this point I think you will have two possible avenues:

1. Use Azure Analysis Services to manage the data. There you can go to 100GB-200GB with incremental updates (you will need to master the incremental update technique and scripting for that).

2. Wait a couple of months while we finish the incremental update feature of Power BI. With it you will be able to go beyond the 10GB limitation of today.

A project with 50B rows seems like very serious and meaty. Can you share more details?

[MarkCBB]

@AmirNetz thank you for the information, I have an Azure account that I have been playing around with, could you point me in the right direction to start learning more about what you mean by "(you will need to master the incremental update technique and scripting for that)."

The project is only going to start in 2-3 months time thankfully. 

WTR the project I will send you a message.

[v-yulgu-msft]

Hi @MarkCBB,

Users can upload files to Power BI service up to 1 GB in size. Power BI Premium supports uploads of Power BI Desktop (.pbix) files that are up to 10 GB in size. To use a large dataset, publish it to a workspace that is assigned to Premium capacity. But, as I know, it is not possible to upload a file which is larger than 10 GB. Maybe you could submit a feature request at ideas page.

Regards,

Yuliana Gu