Incremental data refresh for large dataset

[PBILover]

Hi ,

trying to implement a incremental dataset refresh on the large dataset  (around 1 million row) on a premium workspace ,but at the first time that it is initial load (loading historical data) getting a timeout error.

Is there any way to handle this situation?

Thanks

[lbendlin]

1 million rows is not a large dataset. How many columns? What's the data source?

[PBILover]

thanks for your reply.

Its almost 69 columns in  which some are custom columns.

I tried to load only half million rows but still it is giving timeout error.

 it gives Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

[lbendlin]

that is usually a transient condition (meaning you may get different results when you try again). what's the data source?

[PBILover]

the data source is a transactional data for communications and source is from cosmos .

did i answer your question correctly? or do u need more info on what's the data source?

[lbendlin]

I don't think the Cosmos driver is exposing the QueryTimeout settings. Try pulling in a narrower dataset (fewer columns) .

[PBILover]

ok, I am trying to do that now.

But this dataset was working fine in a production environ,ent before and suddeny satrted giving diff kinds of error.

Is there any other way  how to divide large dataset into a smaller one and combine the data of all in one for the reporting purpose?

[lbendlin]

In general you want to decide where on the Normalization-Denormalization scale you want to place your data model.  Normalization results in smaller size but increased complexity, Denormalization results in less complexity but can also lead to very wide tables like yours.