Data flow temp files

[ambarishkunte]

Hi,

When a large dataflow (time taken for completion approx 4 hours) is executed against a On-prem data source via a gateway , does it create any kind of temp file or anything else which can cause disk space (storage) related issues on the gateway server ?

It was observed on the gateway server that C:\ was more than 90% utilized for a shorter period ( 5mins) while the installation of gateway is on D:\ . So not sure if there's any relation there

Regards,

Ambarish Kunte

[v-eqin-msft · Accepted solution]

Hi @ambarishkunte ,

To my knowledge, you cannot set the spooling location to a custom directory. 

I have re-installed gateway(enterprise) on D disk , it is still located in C disk:

So as I mentioned before, when installing the gateway,we recommend you use a solid-state drive (SSD) as the spooling storage.

Refer to :
https://community.powerbi.com/t5/Service/Gateway-spooling-storage/m-p/941070

https://sharepointviews.com/setup-on-premises-data-gateway/

Best Regards,
Eyelyn Qin
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[v-eqin-msft]

Hi @ambarishkunte ,

According to the official document about On-premises data gateway architecture ,You could know how the gateway works:

1. The cloud service creates a query and the encrypted credentials for the on-premises data source. The query and credentials are sent to the gateway queue for processing. For more information about credential encryption in Power BI, see Power BI security whitepaper.
2. The gateway cloud service analyzes the query and pushes the request to Azure Service Bus Messaging.
3. Azure Service Bus sends the pending requests to the gateway.
4. The gateway gets the query, decrypts the credentials, and connects to one or more data sources with those credentials.
5. The gateway sends the query to the data source to be run.
6. The results are sent from the data source back to the gateway and then to the cloud service. The service then uses the results.

In step 6 , queries like Power BI refreshes and Azure Analysis Services refreshes can return large amounts of data. For such queries, data is temporarily stored on the gateway machine. This data storage continues until all data is received from the data source. The data is then sent back to the cloud service. This process is called spooling. We recommend you use a solid-state drive (SSD) as the spooling storage.

Best Regards,
Eyelyn Qin
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[ambarishkunte]

Thanks Eyelyn . 

The On-Prem Data Gateway is installed on a D:\ while it seems like the temp files are created in C:\ . Is there a way the location of generated temp file could be configured/changed ?

Regards,

Ambarish Kunte