cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
panda2004 Frequent Visitor
Frequent Visitor

Re: Is there any maximum size limit and data storage limit when using Power BI Report Server?

Thanks for the explanation!

 

My case is as follows:

  • I have ElasticSearch cluster with weekly indices. 
  • Each index has about 200GB of data.
  • Power BI obviously can't query ElasticSearch directly.
    But, I managed to query data from Power BI Desktop in different ways:
    1. DAX (Custom Qury), with Web.Contents.
    2. ASP.NET Service (Web).
    3. CData ElasitcSearch ODBC.
      We also have CData Power BI Connector, which is a custom conntector. Unfortentlly, DirectQuery with Custom Connectors still supported only with Power BI Desktop (nieghter in Power BI Server / Power BI Server, as far as I can see). Link here.
  • I'm trying to query 4 indices, which means basically 800GB. Of course that my initial query will try to filter the documents needed (not bringing all the documents) and the fields needed (not bringing all the fields in every matching document), so not all 800GB will be imported. It is very slow to query many documents, as each request only get 10,000 documents in single request (paging/scrolling is applied to get all matching documents). However, caching it though Power BI data-set which refreshes every day is fine - if it works (haven't tried it yet). I'm affraid this data set would be too big, so maybe i'll store a seperate data set for each weekly index and merge them later.
  • How complex is to import the data to some SQL Server / Service Analaysis? Remember it's JSON documents and some of them are very complex.
{
	"ProductId":"88432",
	"Comments":[
		{
			"Username":"stpnet",
			"Comment":"Cool Product!",
			"Stars":5
		},
		{
			"Username":"rblatchford",
			"Comment":"Excellent!",
			"Stars":4
		}
	],
	"Tags":[
		{
			"Key":"Category",
			"Value":"Electricity"
		},
		{
			"Key":"IsNew",
			"Value":true
		},
		{
			"Key":"ShippingType",
			"Value":"Worldwide"
		}	
	]
}

 

panda2004 Frequent Visitor
Frequent Visitor

Re: Is there any maximum size limit and data storage limit when using Power BI Report Server?

Thanks for the explanation!

 

My case is as follows:

  • I have ElasticSearch cluster with weekly indices. 
  • Each index has about 200GB of data.
  • Power BI obviously can't query ElasticSearch directly.
    But, I managed to query data from Power BI Desktop in different ways:
    1. DAX (Custom Qury), with Web.Contents.
    2. ASP.NET Service (Web).
    3. CData ElasitcSearch ODBC.
      We also have CData Power BI Connector, which is a custom conntector. Unfortentlly, DirectQuery with Custom Connectors still supported only with Power BI Desktop (nieghter in Power BI Server / Power BI Server, as far as I can see). Link here.
  • I'm trying to query 4 indices, which means basically 800GB. Of course that my initial query will try to filter the documents needed (not bringing all the documents) and the fields needed (not bringing all the fields in every matching document), so not all 800GB will be imported. It is very slow to query many documents, as each request only get 10,000 documents in single request (paging/scrolling is applied to get all matching documents). However, caching it though Power BI data-set which refreshes every day is fine - if it works (haven't tried it yet). I'm affraid this data set would be too big, so maybe i'll store a seperate data set for each weekly index and merge them later.
  • How complex is to import the data to some SQL Server / Service Analaysis? Remember it's JSON documents and some of them are very complex.
{
	"ProductId":"88432",
	"Comments":[
		{
			"Username":"stpnet",
			"Comment":"Cool Product!",
			"Stars":5
		},
		{
			"Username":"rblatchford",
			"Comment":"Excellent!",
			"Stars":4
		}
	],
	"Tags":[
		{
			"Key":"Category",
			"Value":"Electricity"
		},
		{
			"Key":"IsNew",
			"Value":true
		},
		{
			"Key":"ShippingType",
			"Value":"Worldwide"
		}	
	]
}

 

panda2004 Frequent Visitor
Frequent Visitor

Re: Is there any maximum size limit and data storage limit when using Power BI Report Server?

Thanks for the explanation!

 

My case is as follows:

  • I have ElasticSearch cluster with weekly indices. 
  • Each index has about 200GB of data.
  • Power BI obviously can't query ElasticSearch directly.
    But, I managed to query data from Power BI Desktop in different ways:
    1. DAX (Custom Qury), with Web.Contents (ElasticSearch REST API).
    2. CData ElasitcSearch ODBC.
      We also have CData Power BI Connector, which is a custom conntector. Unfortentlly, DirectQuery with Custom Connectors still supported only with Power BI Desktop (nieghter in Power BI Server / Power BI Server, as far as I can see). Link here.
    3. ASP.NET Service (Web).
  • I'm trying to query 4 indices, which means basically 800GB. Of course that my initial query will try to filter the documents needed (not bringing all the documents) and the fields needed (not bringing all the fields in every matching document), so not all 800GB will be imported. It is very slow to query many documents, as each request only get 10,000 documents in single request (paging/scrolling is applied to get all matching documents). However, caching it though Power BI data-set which refreshes every day is fine - if it works (haven't tried it yet). I'm affraid this data set would be too big, so maybe i'll store a seperate data set for each weekly index and merge them later.
  • How complex is to import the data from ElasticSearch to some SQL Server / Analysis Service? Remember it's JSON documents and some of them are very complex. Is it possible that SQL Server / Analysis Service make a DirectQuery for me (using the ElasticSearch ODBC)?
Highlighted
panda2004 Frequent Visitor
Frequent Visitor

Re: Is there any maximum size limit and data storage limit when using Power BI Report Server?

Hey @rblatchford,

 

"your PBIX is 5GB then that's not very big, so is likely to be fine. We have much larger datasets and they work well".

That's good to hear!

You mean that the data-set stored and refreshed in Report Server, right?