How to store/archive historical data in bronze layer in a lakehouse

[CGrunberg]

We are in the process of moving our setup from a single Power BI semantic model to a lakehouse medallion setup in Fabric.

One of our requirements is that we store historical copies of our files (tables) in a folder date hierarchy (year-month-date) in raw format (the bronze layer). Initially, we did this in blob storage simply storing in folders representing current year, then month, and then date, and if data was imported multiple times in a day, it was simply overwriting the data from the current day.

We've noticed that in our setup, onelake data storage is approx. the same price as azure blob data storage. Therefore, we would like to avoid using an additional system and wanted to make the hierarchical date folder structure in our lakehouse. Is this possible? I am unable to create directories in the lakehouse for normal tables.

Thanks

[v-gchenna-msft · Accepted solution]

Hi  @CGrunberg ,

Thanks for using Fabric Community.

Currently you can only create sub folders in File portion of lakehouse. If you are expecting to create folders within the Tables portion of a Lakehouse, for example:

.
├── Tables/
│   ├── Dims/
│   │   ├── Customers
│   │   └── Products
│   └── Facts/
│       └── Sales
└── Files/
    ├── Subfolder1/
    │   └── File1
    └── Subfolder2/
        └── File2

Unfortunately that's not possible. However, schemas are going to be supported soon which will allow you to group tables together based on the schema.  E.G

• Raw.TableOne
• Raw.TableTwo
• Cleanses.EnhancedTableOne

etc

Fabric roadmap shows this feature is planned for Q2 2024
What's new and planned for Synapse Data Engineering in Microsoft Fabric - Microsoft Fabric | Microso...

At present you can only create sub folders in File portion of lakehouse -

Hope this is helpful. Please let me know incase of further queries.

[v-gchenna-msft]

Hi @CGrunberg ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

[v-gchenna-msft]

Hi @CGrunberg ,

Glad to know that your query got resolved. Please continue using Fabric Community for your further queries.