cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
MiguelMartinez
Microsoft
Microsoft

Share your thoughts on streaming dataflows aka SDF (preview feature)

Back at MBAS, our CVP Arun Ulag gave everybody a sneak peek of streaming dataflows, the new experience for real-time data preparation and comsumption in Power BI. Later at BUILD, we showed everyone an extended demo of the current bits being tried by several of our customers as part of the private preview. And today, finally streaming dataflows is released to the world for public preview with even more updates a new UI.

 

We would love to hear your feedback and opinions to help us decide what comes next and how to improve this functionality in general. Thanks you so much in advance for your feedback and we will be alert to answer any questions you might have and listen to any ideas you might come up with.

 

Cheers

The streaming dataflows team

Data for Everyone! We mean it
27 REPLIES 27
Oliver2020
Helper I
Helper I

Can not see cold table in Power BI Desktop, only hot table (which works perfect).
Is there any configuration? Retention is set to 7 days, could this be switched off in PowerBI Admin?

Can not see cold table in Power BI Desktop, only hot table (which works perfect).

Re: Hi @Oliver2020, can you confirm that you're using "Power Platform dataflows" connector and not "Power BI dataflows"? If yes, is there an error that you see or it takes a long time to get data? Usually, loading cold entities take a little while since it pulls in all the historical data.

Shivan_1-1629313064385.png

 

Retention is set to 7 days, could this be switched off in PowerBI Admin?

Re: Unfortunately no, the max retention policy that is allowed is 7 days (which only pertains to hot entities). You should still be able to tap into historical data using cold entities for data past that 7 day time window.

@Shivan, @MiguelMartinez , @Eklavya  - I do not see the option for Power Platform Dataflows, is there a preview feature I am missing

 

MarkEndicott_0-1633611735788.png

 

It has now been renamed to "Dataflows"

 

Eklavya_0-1633620483952.png

 

Someone needs to update their documentation in that case 😂

 

MarkEndicott_0-1633623204041.png

 

Oliver2020
Helper I
Helper I

Hi!
I receive all messaage fields from the IoT Hub, but not the internal attributes as "iothub-enqueuedtime"!
So you do not have any timestamp and is useless data! (when it is not in the message)
With Stream Analytics there is no probelm with that because all IoT-columns are provided.
Oliver

Ok, figured it out, you have to manually add this fields:
EventProcessedUtcTime
EventEnqueuedUtcTime

 

Preview does not show content, but it works on output table

AndyDDC
Frequent Visitor

Hi,

 

I'm trying to work out the "hot" and "cold" storage process.  I have a Streaming Dataflow that I have set the Retention Duration in the Dataflow settings tab to 1 Day.  I have then run that Streaming Dataflow and can see data in the Hot table when connected via Power BI Desktop.  However, after several days I can certainly see the same data in the Cold storage table but the data remains in the Hot storage table.  I have tried stopping/resuming the Streaming Dataflow but the data remains in the Hot table - I would have thought this would not be visible after being moved to Cold storage?

Hi Andy,

 

The idea is slightly different:
Hot - Keeps data only for the retention duration. While refreshing, data is actively dropped from Hot entities as it moves out of retention.

 

Cold - Data is always kept in cold, regardless of retention.

 

Data is never moved from hot to cold.

AndyDDC
Frequent Visitor

Hi,

 

The process still isn't very clear to me. 

So does a message always exist in Cold storage no matter the retention period?

 

What defines the hot retention period? I would not expect to still see data in the hot table several days after the "1 Day" retention period setting in the Dataflow.

 

thanks

> So does a message always exist in Cold storage no matter the retention period?

Yes.

 

> I would not expect to still see data in the hot table several days after the "1 Day" retention period setting in the Dataflow.

Dataflows enforce retention policy only while refresh is running. So if your refresh is not running, you will still see data that is out of retention.
Do you have a use case where you need the data to be dropped out of retention even when the refresh is not running?

 

AndyDDC
Frequent Visitor

Hi,

 

OK I understand now after testing.  I see both the same data in the hot and cold storage when I set-up a new Streaming Dataflow,

 

I have started the older Streaming Dataflow and after a few minutes I now see significantly less rows in the hot table - there are still around 90 rows (which is strange).

 

I don't have a specific use case but I do think the documentation could be clearer in terms of the differences between hot and cold and under what conditions data will be retained further than the retention point (eg when it's no running).

 

thanks for the responses, much appreciated

markusr
Regular Visitor

Hi Miguel, I'm just evaluating SDF in a premium per user workspace. I was able to create the dataflow based on IoT Hub, start it, create a report on top of it in Power BI Desktop, and the report refreshes automatically after I configured change detection. Great! Now when I try to publish this report to exactly the same (ppu) workspace, publishing never finishes. Same applies when I try to upload the pbix file to the workspace: "circle of death" forever, and no error messages. I can easily upload to other non-premium workspaces, but of course I lose the streaming functionality. Do you have any hints for me...?

OK, I have to modify my post a little, after all, this is preview cloud software, and it's changing fast! Today, publishing the report works fine. But as I try to access the report via browser in my ppu workspace, the error message is "The data source Extension is missing credentials and cannot be accessed". I was able to publish there, and I do have a Premium per User license. What can I do?

Thanks for the flag @markusr Let me check with the team and get back you on this. Doesn't seem we have seen this error before but I'll confirm 👍

Data for Everyone! We mean it

Hi again @markusr . Some quick checks before we go further down the road here:

  1. Did you go into the dataset settings in the service to enter and confirm your credentials?
  2. Might be unrelated, but make sure you used the new Power Platform dataflows connector

If is not any of these 2, then it might be something else we need to take a look at 👍

Data for Everyone! We mean it

Hi @MiguelMartinez , you found it! I really had to go into the dataset settings in the service to enter and confirm my credentials, now the report works and refreshes. I had to modify the dataset settings of the dataset that automatically gets created along with the publishing of the report, and with the same name as the report! I completely ignored that! So make sure this clearly visible in the documentation (it probably is already, I didn't check that). Great work, thanks for your help! 

 

Thanks for confirming @markusr Great idea on including that in docs. I assumed that people would know because of regular dataflows but other users might come directly to streaming. Will add that note ASAP. Thanks again for the flag for using this awesome feature! 👍🙏

Data for Everyone! We mean it
HDK
Frequent Visitor

Hi Miguel,

 

I am trying to use Streaming Dataflows with an Event Hub connection to bring live weather data into Power BI. When creating the streaming flow, I get data in the data preview for the Event Hub item, but for all others it remains blank (Manage Fields / Filter / Output Table). In all documentation I've seen, you should be able to click on each item in the flow and see a static data view, but somehow it doesn't work for me. Have you got any tips?

 

Thanks.

 

Apologies for the late reply @HDK Totally missed this message. Let's start checking did you click on the "Get statis preview" button on top of the data preview table? If yes, we might need more details to figure out what's going on. I showed it on the monthly update video too in case you want to take a look.

Data for Everyone! We mean it

Helpful resources

Announcements
PBI_User Group Leader_768x460.jpg

Manage your user group events

Check out the News & Announcements to learn more.

Power BI October Update 2021.jpg

Power BI Release

Click here to read more about the October 2021 Release!

Community Connections 768x460.jpg

Community & How To Videos

Check out the new Power Platform Community Connections gallery!

Teds Dev Camp Oct. 2021 768x460.jpg

Power BI Dev Camp - October 28th, 2021

Mark your calendars and join us for our next Power BI Dev Camp!