Direct Lake refresh failes after dropping and reloading tables in Warehouse

[SnoekL]

I'm using a custom semantic model on a Fabric Warehouse and want I created a stored procedure that drops the tables in the warehouse and reloads the tabels with using SELECT * INTO [Table]. It seems that the automated refresh of the Semantic model with Direct Lake mode is triggered the moment the tables are dropped and therefore the refresh is failing as the tables that should be in the Model are missing at that point in time. When I manually refresh the semantic model after the pipeline is finished it all works fine.

Is there a way to resolve this without using the scheduled refresh?

In other words is there a way to trigger the automated refresh after the stored procedure (via pipeline) is finished?

[SnoekL]

I will create the ticket tomorrow.

[v-gchenna-msft]

Hi @SnoekL ,

Thanks for using Fabric Community.
Instead of dropping existing tables, consider using a technique like "TRUNCATE TABLE" or similar methods that remove data without affecting the table structure. This allows the semantic model to maintain its connection and refresh successfully.

Will this helps in your scenario?

[SnoekL]

Thanks for the responds. I also tried to delete all the records from the table and then do a Insert into in the empty table instead of dropping the table. But than the refresh isn't automatically triggered eventhought the automatic refresh when changes are detected is turned on. Any idea what might be wrong?

[v-gchenna-msft]

Hi @SnoekL ,

Is it working properly after doing manual refresh, when you are deleting the records?

[SnoekL]

Yes, that works correct but that also works correct when dropping the tables. But I don't want to do the refresh manually or scheduled. I want the automatic refresh to work 😋

It now seems that the automated detect of changes in the direct lake doesn't work.

[v-gchenna-msft]

Hi @SnoekL ,

Apologies for the issue you are facing. 
If its a bug, we will definitely would like to know and properly address it. Please go ahead and raise a support ticket to reach our support team: Link 

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thanks.

[v-gchenna-msft]

Hi @SnoekL ,

If you are interested to refresh Dataset (Semantic Model) from Pipeline.
We can actually do it in two ways via RestAPI (using Web Activity) or sempy library (using Notebook Activity).

Following Article will provide some insights for Dataset Refresh

1. using RestAPI - Datasets - Refresh Dataset - REST API (Power BI Power BI REST APIs) | Microsoft Learn
2. using sempy - Refreshing Individual Tables and Partitions With Semantic Link (fabric.guru)

Hope this is helpful. Please let me know incase of further queries.

[SnoekL]

I created a notebook and with sempy I could indeed trigger the refresh at the end of the pipeline. This solved my problem.

Should I still raise a ticket as the refresh should be triggered automatically?

[v-gchenna-msft]

Hi @SnoekL ,

I suggest to raise a support-ticket and share your issue, so team can check and confirm whether it is a bug or not. It also helps team in product improvement in case if it is identified as a bug.

I hope your query was answered. Please continue using Fabric Community for your further queries.

[v-gchenna-msft]

Hello @SnoekL ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

[SnoekLaurens]

I raised the Ticket (2402270050000383)