Eventstream and other architectures

[DennesTorres]

Hi,

On previous architectures, we are used to two core objects to build a real time data architecture: event hub and stream analytics.

Stream analytics is capable to summarize the data received by event hub and send this data to a streaming dataset.

streaming dataflows received the data directly from the event hub, but they had the same summarization capabilities than stream analytics, so in many way they where replacing stream analytics. We could use one or the other.

Streaming Dataflows are deprecated. In my personal opinion this was a big clue for the fact Eventstreams are about to replace streaming dataflows.

BUT

Eventstreams don't have the same summarization capabilities than streaming dataflows. They accept multiple sources and destinations, but the summary capabilities are very limited.

If it stays as it is, we are adding a new layer to the architecture, "event hub layer 2" but still in need to make the data summarization by ourselves. I'm not sure what advantages we are getting in compasion to what we had before.

My guess it that eventstream may evolve to include the summarization capabilities we had in the streaming dataflows, totally replacing them.

Would it be possible to confirm if my guess is correct, or explain what benefits are we getting by having a "event hub layer 2" but still need to summarize the data by ourselves?

I understand part of the answer may still be under NDA, but any clue that could tell me if my guesses are in the correct direction would help.

Kind Regards,

Dennes

[v-nikhilan-msft · Accepted solution]

Hi  @DennesTorres ,

Eventstream offers same transformations as were available in streaming dataflows. The transformations are available when you configure a destination for Eventstream. Currently, those set of transformations are available for Lakehouse destination. 
The transformations for KQL can be expected in the upcoming release of MS Fabric, as it is already on roadmap.
Hope this information is helpful. Please let me know if you have any other questions.

Please refer this document for more information :

Process event data with the event processor editor - Microsoft Fabric | Microsoft Learn

[DennesTorres]

Hi,

I can confirm this solution works, but it doesn't seems a good architectural idea. 

Each destination will need to develop their own transformations, duplicating the development time and creating a problem of parity, when one destination advances on the transformation more than others.

The streaming dataflows could do the transformations as part of the dataflow, independent of the destination. My feedback is that this seems like an architectural throwback and implementing the transformations more like the streaming dataflow could bring advantages.

We can notice this exactly now, when the transformations were already implemented for the lakehouse and need to be implemented again for the Kusto database, while it could have happened only once.

Kind Regards,

Dennes

[v-nikhilan-msft]

Hi @DennesTorres ,
Thanks for the ask and using the Fabric Community. 
At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.

[DennesTorres]

Hi,

Thank you! It's a great (and surprising) feedback. I will make some tests and I'm quite sure this is the solution for the question.

I would only provide an additional feedback: It's very surprising to notice that on the internal architecture, the transformations depend on the destination. There is the danger of the transformations to evolve in different rhythm for each destination. The common and expected scenario is the transformation being done before reaching the destination and in this way, independent of it.

Thank you!

Kind Regards,

Dennes

[v-nikhilan-msft]

Hi @DennesTorres ,
Glad that your issue got resolved. Please continue using Fabric community for any help regarding your queries.

[v-nikhilan-msft]

Hi @DennesTorres ,
Hope your query got resolved. Please do let us know if you have any further queries.
Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.