Large Dataset in refreshing PowerBI Service

[Anonymous]

Hi all,

I'm newbie in this field.

I had created a Customer's reports. I used MySQL Server is my datawarehouse, and connect with power BI. Then I published it online.

It was 15 milion data rows for 6 months, and it took 2-3 hours to refresh data in app.powerbi.com (not refresh in powerbi desktop)

So, here is my 02 problems:

1. My dataset will be get more and more large in the near future, and maybe it will take more time than 3hours to refresh data.

Is there something that I can do to imporve this situation?

2. The refresh usually get fail, for some below common reasons:

- "Before the data import for Total_Identifies finished, its data source timed out. Double-check whether that data source can process import queries, and if it can, try again."
- "Unable to connect to the data source undefined."

Microsoft SQL: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

What can I do to fix this?

Thanks you guy in advance (Y)

[v-lid-msft · Accepted solution]

Hi @Anonymous ,

We can use the Increasement Refresh to reduce the refresh time if the dataset is under premium capacity. Or we can following those tips to reduce the size of dataset or optimize the model of dataset based on this document, some tips may not reduce the time of refresh.

• Remove unused tables or columns, where possible. 
• Avoid distinct counts on fields with high cardinality – that is, millions of distinct values.  
• Take steps to avoid fields with unnecessary precision and high cardinality. For example, you could split highly unique datetime values into separate columns – for example, month, year, date, and so on. Or, where possible, use rounding on high-precision fields to lower cardinality – (for example, 13.29889 -> 13.3).
• Use integers instead of strings, where possible.
• Be wary of DAX functions, which need to test every row in a table – for example, RANKX – in the worst case, these functions can exponentially increase run-time and memory requirements given linear increases in table size.
• When connecting to data sources via DirectQuery, consider indexing columns that are commonly filtered or sliced again. Indexing greatly improves report responsiveness.  

For second question, please increase the timeout value in connector function, such as following:

MySQL.Database(server, database, [CommandTimeout = #duration(0,2,0,0)])

we may also need to increase the timeout value in data source.

Best regards,

[Seth_C_Bauer]

@Anonymous The quick things you can do are ensure that only the data you are using is being loaded, remove the rest. Use the PowerQuery Analyzer to tune the ingestion if possible.

Another alternative is to use Analysis Services as a stand alone model. I love working in Power BI more, but sometimes if you have to scale up you need to jump over and use that with live connection to offload the long processing times.