Real-Time Ingestion and KQL

[DennesTorres]

Hi,

Is there a document which explains why the KQL database is being recommended (or it seems so) for eventstream ingestion?

One detail I noticed is that data warehouses and data lakes are directly stored in delta tables, but KQL isn't. There is a special configuration in a KQL database to make it available to onelake and if I understand correctly, this configuration is related to creating a shortcut in onelake to the KQL database.

Could we conclude the KQL database has its own internal storage, and it's only shared with onelake using a shortcut?

Are these two questions related to each other?

Kind Regards,

Dennes

[v-nikhilan-msft · Accepted solution]

Hi @DennesTorres ,
Thanks for using Microsoft Fabric Community.

Yes, the two questions are related.
The KQL database is recommended for eventstream ingestion because it is a fast and scalable database that is designed to handle large volumes of data. It also provides a number of features that make it well-suited for eventstream ingestion, such as:

1) Support for real-time data ingestion: KQL databases can ingest data from eventstreams in real time, allowing you to analyze data as it is being generated.
2) Support for schema-on-read: KQL databases do not require you to define a schema for your data before ingesting it. This allows you to start ingesting data immediately, even if the schema is not finalized.
3) Support for streaming queries: KQL databases allow you to query data as it is being ingested. This allows you to get insights into your data quickly and easily.
4) Integration with OneLake: KQL databases can be integrated with OneLake, a data lakehouse that provides a unified view of your data. This allows you to analyze data stored in both KQL databases and OneLake using the same query language.

OneLake is a data lakehouse that provides a unified view of your data, regardless of where it is stored. When you configure a KQL database to be available to OneLake, OneLake creates a shortcut to the KQL database. This shortcut allows you to query the data in the KQL database using the OneLake query engine. KQL databases also have their own internal storage, which is separate from the storage used by OneLake. This allows KQL databases to provide a number of performance and scalability benefits.

You can refer these documents:

Document1 

Document2 

Hope this helps. Please let us know if you have any further queries.