Virtual Network Gateways not publishing data into Lakehouse

[LouisSh]

Hi Fabric Community,

I have a virtual gateway which is able to read data from an Azure SQL database through Gen2 Dataflow however will not then publish the data into a destination, which is a Lakehouse. I can view the data inside the workflow but cannot get it to the destination inside fabric. If I open up the dataflow itself and try to add the connection, this is the error.

When I click the red error triangle outside of the dataflow (in the workspace where the dataflow lives), this is the error I get.

[SidJay]

Hi @LouisSh,

Could you please confirm that your VNet is set up to allow these endpoints?

• Protocol: TCP
• Endpoints: *.api.powerbi.com, *.datawarehouse.pbidedicated.windows.net, *.datawarehouse.fabric.microsoft.com, *.dfs.fabric.microsoft.com
• Port: 1433

Thanks

[v-cboorla-msft]

Hi @LouisSh 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

[v-cboorla-msft]

Hi @LouisSh 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thanks.

[v-cboorla-msft]

Hi @LouisSh 

Thanks for using Microsoft Fabric Community.

Apologies for the inconvenience that you are facing here.

The error message suggests that the dataflow is having trouble communicating with the Fabric endpoint via the on-premises data gateway. Is your on-premises data gateway machine behind a firewall, and, if so, can you check your firewall logs to see if there was a failed connection attempt made to https://api.powerbi.com/powerbi/globalservice/v201606/clusterdetails?

For more details you can refer to this thread: Problem to import on premise data onto a lakehouse using a DataflowGen2

I hope this information helps. Please do let us know if you have any further questions.

Thanks

[LouisSh]

Hi @v-cboorla-msft 

Just to confirm, this is a virtual gateway that is being used and not an on premise. There is no firewall for this. How can I check the firewall logs anyways for virtual gateways, is this in the Azure portal?

[v-cboorla-msft]

Hi @LouisSh 

I have tried to repro the same scenario but didn't find any difficulty in publishing the data into a destination which is a Lakehouse using Dataflow Gen2. Please find the screenshots for reference.

You can also try connecting to the Azure SQL database uisng onprem-gateway. 

For more details please refer to the below mentioned links:

Install an on-premises data gateway.

How to access on-premises data sources in Data Factory for Microsoft Fabric.

On-premises data gateway considerations for data destinations in Dataflow Gen2.

I also tried connecting using on-prem gateway:

Please try connecting as stated above and let us know if you have any further issues.

This might seems to be an intermittent issue. Can you please try again after a day and let me know if you are still facing this issue?

Hope this helps.

Thanks.

[LouisSh]

Hi @v-cboorla-msft 

As I understand from your first set of screenshots to recreate the issue, you have not used a virtual-network gateway as the data-gateway = none, so won't be able to recreate the issue I am seeing. As for it being a intermittent issue, we have been trying overcome this for 3-4 weeks now. We would prefer to continue to use the virtual gateway rather than an on premise gateway. The virtual gateway works fine connecting to the Azure SQL, but fails pushing it into the Lakehouse inside Fabric, this is where the problem lies.

Is there anything additional I can provide to help understand the issue? 

[pochuang · Accepted solution]

Hi @LouisSh ,

We checked trace with the ReqeustId you provided and noticed that your gateway cannot resolve api.powerbi.com. It seems that there is an incorrect DNS record in the private DNS zone probably becauase you had a PowerBI privatelink setup that hasn't being cleaned up correctly.

In the future, you can get more insight of the failure by checking the diagnostic logs (Download logs on the VNet data gateway | Microsoft Learn), and use the network troubleshooting tool for networking problems (Troubleshoot a virtual network (VNet) network | Microsoft Learn).

Thanks.