Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
Hello PBI Community
Here's the the issue:
I have created a streaming dataflow that uses Event Hub to stream ticket sales data. It uses single Event Hub, multiple Manage Fields, multiple Filters and then creates multiple output tables.
Event Hub schema:
It uses 2 partitions and then free fields (not sure the exact name for those fields.): partition 0, partition 1 and then scan_entry in this case.
Each partition contains other fields:
When I use manage fields to pull the fields I need, it works for free fields, but not for the fields in each partition.
Some more details.
My Streaming Dataflow:
Appreciate anyone who took time to view this post - You're the best!
HELP!!! 😄
Solved! Go to Solution.
@v-yetao1-msft we (data engineer from my team and myself) found a solution. Since our event hub consisted of partition 0, partition 1 and those "free fields" and I mainly needed the data from partitions, the data engineer created a new event hub where partition 0 and partition 1 was combined into one and the output looks like only free columns, no second level columns.
So our guess is that PBI SDF (streaming dataflows) cannot for now handle a bit more complex partitions that include lists, dictionaries.
Contact me if you need more details.
Thank You for your help!
How the output looks now:
So have you compared the difference between the fields that can be pulled and the fields that cannot be pulled ?
Best Regards,
Community Support Team _ Ailsa Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
@v-yetao1-msft the fields that cannot be pulled are from valid json files. The fields that can - simple columns. From many examples that I have seen, it should be possible to pull individual fields from json and free fields together. That does not happen in this case.
@v-yetao1-msft we (data engineer from my team and myself) found a solution. Since our event hub consisted of partition 0, partition 1 and those "free fields" and I mainly needed the data from partitions, the data engineer created a new event hub where partition 0 and partition 1 was combined into one and the output looks like only free columns, no second level columns.
So our guess is that PBI SDF (streaming dataflows) cannot for now handle a bit more complex partitions that include lists, dictionaries.
Contact me if you need more details.
Thank You for your help!
How the output looks now:
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the April 2024 Power BI update to learn about new features.