How to implement Change Data Capture

[Mohida]

Hi Team,

I have several big query tables, from which I need to bring the data to my fabric warehouse. I have created a control table in my warehouse where i maintain the last update dates pertaining to a table. I use a dataflow to bring data from bigquery table to warehouse table. Now i want to include a filter here which filters the data for - last_update-date >= (value from control table). 

How should we implement this in dataflow ? Or how should one implement this using data pipelines ? Please suggest. 

Also , Currently i do not see a way to pass a parameter from datapipeline to dataflow. Please let me know the correct approach for the above scenario ? 

Regards

Mohida

[miguel · Accepted solution]

Hi!

Can you create a query which yields the value for (value from control table)? if yes, you can pass that as a filter of the query that you want to use it with as part of a "Filter rows" or Table.SelectRows function

There is currently no way to pass a parameter from a pipeline to a Dataflow. You can vote for this idea to be implemented using the link below:

https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=2f761603-a400-ee11-a81c-6045bdb98602 

[v-nikhilan-msft]

Hi @Mohida 
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks