I've setup a pipeline that pulls data from a lakehouse (with shortcuts to tables in another workspace) into a data-warehouse via two stored procedures.
It succeeds when I run the pipline manually, or when I schedule it for the near future (5 - 15 minutes from the current time). However, every overnight run reports success, runs for the appropriate time, but does not seem to actually modify the data-warehouse (I'm checking via the max time-stamp in the source vs. the destination tables).
Has anyone else observed anything similar, or know an appropriate workaround for the time being? I will file a bug as soon as I find the appropriate place to do so (I'm relatively new to Fabric, and these forums).
More info - I've tried having the data pipleine use scripts instead of stored procedures, adding a step that runs the same time-stamp check in the warehouse (to check if it may be a sync issue, and my test queries were forcing an endpoint refresh), and tried deleting and recreating the pipeline already. I've also tried increasing the time between the ingest in the source workspace and the pipeline, but still saw the same behavior (the original period I tried was a 2 hour difference between the two, increased to 4 hours to test if that was an issue).
