Data capacity

[ozancavus]

Dear Power BI Community,

As a BI consultant I am experimenting with the functionalities of Power BI and even though I have no experience with Microsoft environment I can say that I am very impressed, congratulations for doing such a good job! After two days of digging I have questions in mind.

1. Say that an organization has 1 TB of sales data, how does Power BI deals with such cases where data is much larger 10 GB limit?

2. If it can handle data chunks that are larger than 10 GB, does it use Azure?

3. If it can handle data chunks that are larger than 10 GB, how does it connects to local data sources such as an SQL database with data of 500 GBs?

Again, bravo for the Power BI and keep up the good work!

Best,

Ozan

[Seth_C_Bauer · Accepted solution]

@ozancavus

1. Say that an organization has 1 TB of sales data, how does Power BI deals with such cases where data is much larger 10 GB limit?

The current path would be to have that data in a cube/model, Power BI can connect directly to that cube/model and not import the data.

2. If it can handle data chunks that are larger than 10 GB, does it use Azure? Each workspace/group is limited to 10GB of which all the info already sits in Azure.

3. If it can handle data chunks that are larger than 10 GB, how does it connects to local data sources such as an SQL database with data of 500 GBs? Direct Query to databases/cubes/models only queries the information, it doesn't import it.

Direct Query methods send queries to your data source (Database, cube/model) - the size of those datasources is not a factor because you are not importing the data into Power BI.

[ashishrj · Accepted solution]

@ozancavus The answer to all the queries is using Direct Query for supported data sources wherein data is not loaded into your Power BI Desktop model. It would be even great if you try importing data in SSAS model (tabular/multidimensional) and then create reports out of it ! But remember that direct query requires data sources connection to be always ON. 

If SQL Server in Azure then the Azure VM / Azure Datawarehouse should not be paused / stopped. 

[ashishrj · Accepted solution]

@ozancavus The answer to all the queries is using Direct Query for supported data sources wherein data is not loaded into your Power BI Desktop model. It would be even great if you try importing data in SSAS model (tabular/multidimensional) and then create reports out of it ! But remember that direct query requires data sources connection to be always ON. 

If SQL Server in Azure then the Azure VM / Azure Datawarehouse should not be paused / stopped. 

[ozancavus]

Dear @Seth_C_Bauer @ashishrj,

Thank you for your replies! I understand that in order to show data that are larger than 10GB Power BI uses DirectQuery to connect to the DB and retreieves the result of the query and presents it wihile wihout consuming your 10G data limit.

My next questions are:

1. Where does Power BI Online stores the result of the query (say it returns 5 GB of data)? Does it stores it temporarily in some sort of cache memory that doesn't eat your data capacity?

2. What is the difference of having your (say 8GB) data being uploaded to the Power BI online or accessing it via DirectQuery to the DB? It came to my mind that it might take longer time to return the result of the query rather than showing it directly from the uploaded data, are there any other impacts?

Best

Ozan

[Seth_C_Bauer · Accepted solution]

@ozancavus

Take a look at the security white paper here it will answer many of the questions you may have surrounding how data is consumed/used.

If you still have additional questions after reading that (download the white paper from that link) paper, we can pick it up from there.

[ozancavus]

Dear @Seth_C_Bauer,

Thank you for the document.

Best

Ozan

[Seth_C_Bauer · Accepted solution]

@ozancavus

1. Say that an organization has 1 TB of sales data, how does Power BI deals with such cases where data is much larger 10 GB limit?

The current path would be to have that data in a cube/model, Power BI can connect directly to that cube/model and not import the data.

2. If it can handle data chunks that are larger than 10 GB, does it use Azure? Each workspace/group is limited to 10GB of which all the info already sits in Azure.

3. If it can handle data chunks that are larger than 10 GB, how does it connects to local data sources such as an SQL database with data of 500 GBs? Direct Query to databases/cubes/models only queries the information, it doesn't import it.

Direct Query methods send queries to your data source (Database, cube/model) - the size of those datasources is not a factor because you are not importing the data into Power BI.

[aesculapius]

how much it's cost to have this kind of capacity?

[Seth_C_Bauer]

@aesculapius Somethin I realize hasn't been asked. Where are you pulling your data from, and how? The behaviour you describe will occur when you are streaming data into the Power BI Service and enabling the "historic data analysis". This method allows you to build reports, but there is a threshold for the amount of data it retains. 

This thread contains the specific thresholds.

https://community.powerbi.com/t5/Service/History-Data-Analysis-option-in-Streaming-dataset/td-p/1569...

[aesculapius]

thanks for the information. btw how much would it cost for using that service preciesely. cause i calculated this would cost more than thousand dollar per month. thanks

[elongo]

@Seth_C_Bauer, @ashishrj thanks for your inputs on this topic. I am experiencing something in Power BI which I can't understand:

- I created a dataset on the 30.12.2016 (50  days ago, approx.)

- With that dataset I created a report and pinned to a dashboard.

- I have only consumed 8MB/1GB (0.08%) of space in my Power BI account.

- But my data and reports are being cut day by day, showing me only the last 12 days of data! So every day my data is shorter, and it seems lost. 

WHAT IS GOING ON?

HOW CAN I HAVE ACCESS TO MY DATA SINCE THE BEGINNING OF MY DATASET?

(I AM ABOUT TO LOOSE PRECIOUS DATA IN FEW DAYS IF THIS CONTINUES AND IN THAT CASE POWER BI WONT'T BE USEUFUL FOR MY BUSINESS)

Thanks for any clue you can provide me.

elongo

[Seth_C_Bauer]

@elongo When is the last time you refreshed the dataset? Do you have any page level or report level filters that are constraining the data set, and as each day goes by you are just no longer seeing it in the report?

I would create another report off the dataset and see if the data is actually being removed, or if it is some report setting.

[MarcelBeug]

@elongo

My suggestion would be to raise a new topic yourself (in the Desktop forum) and not directly raise an issue if you don't know what is going on.

Please provide more information, as the issue may just be your queries and/or report definitions as already suggested by @Seth_C_Bauer

And please mind your CAPS LOCK as people may be less willing to react.