push data into Power BI Dataset from asp.net web api

[nasirali]

Hi,

I am getting my data for power bi report from asp.net web api as json. As i am fetching 50-60k records, it takes time to get all data and if there is network traffic, request timeout also happens sometimes.

So my question is:

Is there a way to populate dataset as a stream means instead of getting all the data at once. can i fetch some data pushing it to dataset and so on? It is not a live streaming scenario. 

Any pointers will be very appreciated. 

Thanks.

[v-shex-msft]

Hi @nasirali,

What type of power bi you used ? Desktop or service? If you mean desktop, current it not support get streaming data.

If you mean power bi service, you can simply setting an application to push your data to streaming dateset api.

Real-time streaming in Power BI

Push Data into Power BI Streaming Datasets with C#

But current power bi not support to use steaming dateseting on normal visuals.

Regards,

Xiaoxin Sheng

[nasirali]

Hi @v-shex-msft

Thank you for reply and links. I am using power bi desktop for development of reports then publish to service for embedding in application. I have a report which is getting data from webapi as json and data is being updated with refresh schedule. Everything is working fine.

Only problem is, web api has a request timeout. And if data is large enough it will timeout, So i was thinking if there is a way to push data as i read it. It will be efficient and i will not have this problem. Something like in the second the link you provided. Will it work with normal dataset too or just streaming dataset? 

Regards,

nasir ali

[v-shex-msft · Accepted solution]

Hi @nasirali,

If you push data to streaming api, it should be streaming dataset.(streaming dataset not cache data, it can only use with specific tiles)

If you mean publish data to rest api and create dataset who stored in power bi service. It should be a normal dataset.

I'm not so sure for timeout issue which you mentioned, but I think you can write a loop function to split these data to suitable size/interval and push them.

For power bi rest api limitation, please refer to below link:

Power BI REST API limitations

To POST Rows

• 75 max columns
• 75 max tables
• 10,000 max rows per single POST rows request
• 1,000,000 rows added per hour per dataset
• 5 max pending POST rows requests per dataset
• 120 POST rows requests per minute per dataset
• If table has 250,000 or more rows, 120 POST rows requests per hour per dataset
• 200,000 max rows stored per table in FIFO dataset
• 5,000,000 max rows stored per table in ‘none retention policy’ dataset
• 4,000 characters per value for string column in POST rows operation

Regards,

Xiaoxin Sheng

[kkalyanthaya]

For none retention policy dataset , what is the behaviour if table reaches the max rows of 5,000,000 as specified below.

I believe it is not FIFO and there is no delete individual rows APIs available. If we need to push new data to it, how do we do it ? Is flush all the rows and start again the only option ?

Thanks,

Kiran