Mirroring in Microsoft Fabric

[Safwan]

Hi,

I am some ambiguity in understanding Fabric Mirroring. I know that it enables users to effortlessly replicate external data warehouses or databases. But I want to understand does it copy the physical data to Fabric and if no then how does it work?

Hoping for a reply.

[v-cboorla-msft]

Hi @Safwan 

Thanks for using Microsoft Fabric Community.

No, Microsoft Fabric Mirroring does not physically copy the data to Microsoft Fabric. Instead, it employs a sophisticated process to replicate data without moving it physically. Here’s how it works:

Change Data Capture (CDC): Fabric's Mirroring feature uses a technology called Change Data Capture (CDC) to act like a detective. It constantly monitors your external data sources (like databases) and keeps a watchful eye for any changes, such as new entries, updates, or deletions. 

Delta Tables: Instead of copying everything, Mirroring only captures these identified changes. It then cleverly transforms them into a format specifically designed for analysis, often using Delta tables. These tables are like condensed summaries, focusing only on the differences.

OneLake Storage Layer: The transformed changes are then placed in a secure storage area within Fabric called OneLake. OneLake is designed for efficiency, ensuring the data is organized and readily accessible for your analysis needs.

Intelligent Logic: To optimize performance, Fabric employs intelligent logic. It can figure out when your source data has changed and only triggers synchronization for those specific updates. This minimizes unnecessary processing power usage.

Granular Controls: Mirroring offers you granular control. You can choose which specific data (like tables or entities) you want to be reflected in Fabric. This allows you to focus on the most relevant information for your analysis.

For more details please refer : Introducing Mirroring in Microsoft Fabric.

                                               : What's new in Microsoft Fabric?

I hope this information helps. Please do let us know if you have any further questions.

Thanks.

[Safwan]

Hi @v-cboorla-msft ,

I get the CDC part of mirroring but how is the inital snapshot of data handeled by Fabric. I just want to develop an understanding of it.

Hoping for a reply.

[v-cboorla-msft]

Hi @Safwan 

You're right, the initial snapshot in Fabric Mirroring is a crucial step to understand. While CDC handles ongoing changes, the initial snapshot brings your external data into Fabric for the first time. Here's how it's likely handled:

Prioritizing Efficiency: Fabric Mirroring is likely designed for speed and minimal resource usage during the initial data transfer. This could involve:

• Bulk Loading: Transferring large chunks of data at once instead of individual records, significantly accelerating the process.
• Partitioned Transfers: Particularly for very large datasets, dividing the data into smaller subsets allows for faster and more efficient transfer.

Potential Configurability:

• Partial Snapshots: Depending on your configuration, you might have the option to specify which tables or subsets of data to include in the initial snapshot. This allows you to focus on specific data relevant to your needs.

Leveraging Source Capabilities:

• It's possible that Fabric Mirroring utilizes existing export functionalities within your source database (like Azure Cosmos DB or SQL Server) to expedite the initial data transfer process. This can leverage existing tools optimized for your specific source.

Additional Considerations:

• Temporary Storage (Possible): The initial data transfer might be staged in temporary storage optimized for speed before being transformed and stored in OneLake, Fabric's data storage solution. This temporary storage would facilitate a faster initial transfer.
• Gradual Population (Speculative): For exceptionally large datasets, the initial data population might occur gradually to minimize any impact on the source database's performance. This would ensure a smooth initial synchronization process.

I hope this information helps. Please do let us know if you have any further questions.

Thanks.

[v-cboorla-msft]

Hi @Safwan 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

[Safwan]

Hi @v-cboorla-msft ,

So during the inital snapshot Mirroring does copy the data from source to Fabric, correct me if I am wrong.

[v-cboorla-msft]

Hi @Safwan 

Yes, during the initial snapshot, mirroring establishes a synchronized copy of the source data within the Microsoft Fabric Lakehouse. This process ensures that the data in the lakehouse is synchronized with the source system. It’s like creating a mirror image of the data, capturing its current state. Once this initial snapshot is complete, the mirroring process efficiently captures and integrates incremental data updates, enabling efficient data synchronization and ongoing data management. 

For details please refer : Fabric Mirroring - public preview

I hope this information helps. Please do let us know if you have any further questions.

Thanks.

[Safwan]

Hi @v-cboorla-msft ,

So mirroring does physically copy the data during the initial snapshot and then ensures that the data is synchronized with the source, right?

[v-cboorla-msft · Accepted solution]

Hi @Safwan 

I have checked with the internal team. The internal team has confirmed that:

Mirroring does physically copy the data from source database to OneLake delta parquet format as well as maintain any data changes in the OneCopy in Fabric OneLake.

I hope this information helps. Please do let us know if you have any further questions.

Thanks.

[v-cboorla-msft]

Hi @Safwan 

Glad that your query got resolved.

Please continue using Fabric Community for any help regarding your queries.

Thanks.