How to reduce the time the flow needs to load (parquet files)?

[Anonymous]

In PowerBI Desktop I want to get data from Azure blob storage. The data is in the parquet format, the size is about 80 MiB.

I could connect PowerBI desktop directly to the file, but then I would have use the account key. I haven't seen alternatives. Since a lot of people will have to connect this way and sharing the key with so many people isn't secure, I'm looking for a different option.

So I've created a data flow that takes the data from the storage account using a gateway. It works fine although is time-consuming. A flow refresh in PowerBI service takes 1h.

The problem is, when I then connect to the flow from PowerBI desktop, the amount of the data is huge. In this very moment I'm trying to load the data and the process hasn't finished after 30 minutes. It shows that it has loaded 30 GB but is still loading. 

Any ideas how I could shape the architecture in order not to make the key available but also to get the data within, let's say, 15 minutes?

[v-lionel-msft · Accepted solution]

Hi @Anonymous ,

You can use Direct Query mode to connect to DataFlow, so you don't need to import data into the dataset. In doing so, even if your data is large, it will not take too much time.

Power BI Dataflows Direct Query Support | Microsoft Power BI Blog | Microsoft Power BI

Best regards,
Lionel Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.