Spooling Data Gateway

[D1234]

Hi,

We are just testing a new data gateway on an Azure VM, the data volume used in the current test data flow is much bigger than we'd tried previously (50 million records coming from an Azure SQL database). It seems that the whole dataset is spooling before its moved to the Power BI Service,taking a few hours,  is this the expected behaviour? I'd previously thought spooling was for a situation where overflow occurred and data wasn't moving fast enough through the gateway, the machine spec is inline with those recommended in the documentation, though there is only one virtual NIC so perhaps throughput may be a bottleneck.

Appreciate any clarification/advice!

Thanks

[v-yangliu-msft · Accepted solution]

Hi  @D1234 ,

Data is temporarily stored on the gateway machine. This data storage continues until all data is received from the data source. The data is then sent back to the cloud service. This process is called spooling.

And we can't set the spooling location to a custom directory, which is located in the installation directory of the gateway:

C：\ Users \ PBIEgwService \ AppData \ Local \ Microsoft \ In-premises data gateway \ Spooler

Therefore, when installing the gateway,We recommend you use a solid-state drive (SSD) as the spooling storage.

Links of related gateways and spools:

https://docs.microsoft.com/en-us/data-integration/gateway/service-gateway-onprem-indepth

Best Regards,

Liu Yang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.