Hi,
We are working with manual excel files as source. (from a Sharepoint folder).
Have a scenario wherein we need to...
1.) Create dataflow to load historic data into a "Staging" table. ---(for example data till Dec-2022......this is working as expected)
2.) Create another dataflow to load the monthly data and append it to the "Staging" table created in the first dataflow. (for example data for Jan-2023)
3.) The second dataflow has a defined logic wherein it will only pick the latest file from the Sharepoint Folder to be appended with the history dataset and all previous months data.
When I test with single file (data for Jan-2023) in the Sharepoint folder...it works as expected.
However, If I add Feb-2023 data file (for testing), the dataflow only appends data for Feb-2023 to the Historic "Staging" table...and the Jan-2023 data is no longer there...
The expected result was that the "Staging" table would have ---> Historic data + Jan 2023 + Feb 2023
Am I missing a step somewhere ?
Can anyone please help/guide as to what should be the proper approach.
Thanks.
