Importing 320 million

[alakiran]

Hi,

I am trying to import a large data set of 320 million rows into powerbi desktop and it is failing with below error. Any suggestions/ help?

Apply query changes

Failed to save modifications to the server. Error returned. There's not enough memory to complete this operation. please try again later when there may be more memory available.

[zoloturu]

@alakiran,

320 mln rows is quite big amount. I have some suggestions but firstly please clarify below:

1) How often do you need to run import from Azure SQL? What is the frequency?

2) I guess you not plan to use this file in Power BI Service. Correct?

3) How much is your RAM?

[Greg_Deckler]

What is the data source? Sounds like you need more memory or should switch to Direct Query.

[alakiran]

Azure SQL Server is the data source, Direct Query has a limitation to retrieve only 1 million records.

[eniX]

you need more RAM apparently. Be prepared to the fact that your .pbix-file may exceed the 1 GB size limitation (PBI Premium needed?)

[alakiran]

Hi,

I am trying to import 320 million records to PowerBI desktop and after running for 5 hours, it is failing with the error. Any suggestions or help please?

Apply Query changes

Failed to save modifications to the server. Error returned: There's not enough memory to complete this operation. Please try again later when there may be more memory available

[Phil_Seamark]

Hi @alakiran

Remove any columns you don't need.  Do you have any columns that have highly unique columns?

Do you have any DateTime data values?

What is your data source?

[alakiran]

Yes, I think i selected 6-8 columns out of 30 columns and it is still failing with the error. Yes, There is a datetime column out of those selected columns.

[eniX]

Maybe try to separate your data into buckets with rownum and append the queries in the end?