Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.
How can I copy/clone a parquet table stored in ADLSg2 into a managed table in a Lakehouse keeping partitions? I've created a shortcut below Files in my lakehouse, where I can see my tables in parquet format, partitioned in some folders. I click on ··· > Load to tables > New Table > Including Subfolder and then I create new table bellow Tables folder. The issue is that my table bellow Files is partitioned, and the new table bellow Tables folder isn't, all the data is in one parquet file not partitioned. Is there any way I can keep the partitions in the new Delta table?
I guess I can use a notebook to read table in Files and write it as Delta in Tables. But I'd like to know if there is a more simple way to achieve this using Fabric features. Or at least, maybe there's a python script or whatever that reads each table one by one and converts it into Delta bellow Tables folder keeping partitions?
Thanks.
Solved! Go to Solution.
So, I guess I have 2 options:
- Read the table with a notebook from ADLSg2 and write it partitioned into Fabric lakehouse
- Read the table with Data Pieline COPY activity from ADLSg2 connection and partition it in Destination (Fabric Lakehouse) with advanced options
Hi @amaaiia ,
Thanks for using Fabric Community.
As I understand you want to create a Delta table that maintains the partition structure of the original parquet table.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.
Hi @amaaiia ,
There isn't a direct way in Fabric to clone a partitioned Parquet table to a managed Delta table in the lakehouse while preserving partitions. You can simply read the file and save the file with option partitionBy.
Docs to refer -
PySpark partitionBy() - Write to Disk Example - Spark By {Examples} (sparkbyexamples.com)
Hope this is helpful.
So, I guess I have 2 options:
- Read the table with a notebook from ADLSg2 and write it partitioned into Fabric lakehouse
- Read the table with Data Pieline COPY activity from ADLSg2 connection and partition it in Destination (Fabric Lakehouse) with advanced options
Ask questions in Data Engineering, Data Science, Data Warehouse and General Discussion.
Check out the April 2024 Fabric update to learn about new features.
User | Count |
---|---|
4 | |
3 | |
2 | |
1 | |
1 |