Data pipeline to download zip file containing multiple csv files from web

[jgrundah]

I am attempting to create a pipeline or pipelines to do the following:

• Download a zip file from a website  (https://www150.statcan.gc.ca/n1/en/tbl/csv/14100287-eng.zip?st=0Sbmw1wJ)
• Store the binary zip to a lakehouse in the Files/Zip
• Unzip the downloaded file into multiple CSV files in Files/CSV (folder in the lakehouse)
• Dataflow Gen2 to perform the ETL functions

[jgrundah]

I would really like a data pipeline solution to this.  It seems like the ability to unzip delimitated files is built-in, but it fails due to the lack of proper handling of multiple files in the zip.  I would love to see a pipeline procedure that unzips multiple files to the Files folders.  Other pipelines would process the csv and combine them.  Any chance that notebooks could solve my problem?

[jgrundah]

I am attempting to create a pipeline or pipelines to do the following:

• Download a zip file from a website  (https://www150.statcan.gc.ca/n1/en/tbl/csv/14100287-eng.zip?st=0Sbmw1wJ)
• Store the binary zip to a lakehouse in a folder - Files/Zip
• Unzip the downloaded file into multiple CSV files in Files/CSV (folder in the lakehouse)
• Dataflow Gen2 to perform the ETL functions to create a table in the Data lake

The problem is that when I unzip the file into delimited format, there is no way to unzip multiple files to the Files/CSV folder.  The get data in the pipeline wants to create a single file.  I have attempted to go directly from binary zip to table, but the procedure gives me an error because the two CSV are not in the same format. 

I am looking for help on the unzip pipeline into multiple CSV files.

[Brendan151]

You should be able to do this in a Gen2 Dataflow, following tackytechtom's answer from https://community.fabric.microsoft.com/t5/Power-Query/zipped-file-how-to-unzip-in-power-query-more-e... 

Run the function with your url and then you can create separate queries in Power Query for each CSV file.

[v-cboorla-msft]

Hi @jgrundah 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

[v-cboorla-msft]

Hi @jgrundah 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thanks.