Missing data between Azure Stream Analytics and Power BI

[fr1337ddy]

Hi,

I have the following results (just an extract, the whole result table has more columns) of a test output in my Azure Stream Analytics job (one event every second):

time	conductivity
"2020-04-16T04:52:36.0000000Z"	977,898418
"2020-04-16T04:52:37.0000000Z"	977,998421
"2020-04-16T04:52:38.0000000Z"	978,098423
"2020-04-16T04:52:39.0000000Z"	978,198425
"2020-04-16T04:52:40.0000000Z"	978,298428
"2020-04-16T04:52:41.0000000Z"	978,398430
"2020-04-16T04:52:42.0000000Z"	978,498432
"2020-04-16T04:52:43.0000000Z"	978,598434
"2020-04-16T04:52:44.0000000Z"	978,698437
"2020-04-16T04:52:45.0000000Z"	978,798439
"2020-04-16T04:52:46.0000000Z"	978,898441
"2020-04-16T04:52:47.0000000Z"	978,998444
"2020-04-16T04:52:48.0000000Z"	979,098446
"2020-04-16T04:52:49.0000000Z"	979,198448
"2020-04-16T04:52:50.0000000Z"	979,298450
"2020-04-16T04:52:51.0000000Z"	979,398453
"2020-04-16T04:52:52.0000000Z"	979,498455
"2020-04-16T04:52:53.0000000Z"	979,598457
"2020-04-16T04:52:54.0000000Z"	979,698460
"2020-04-16T04:52:55.0000000Z"	979,798462
"2020-04-16T04:52:56.0000000Z"	979,898464
"2020-04-16T04:52:57.0000000Z"	979,998466
"2020-04-16T04:52:58.0000000Z"	980,098469
"2020-04-16T04:52:59.0000000Z"	980,198471
"2020-04-16T04:53:00.0000000Z"	980,298473
"2020-04-16T04:53:01.0000000Z"	980,398476
"2020-04-16T04:53:02.0000000Z"	980,498478
"2020-04-16T04:53:03.0000000Z"	980,598480
"2020-04-16T04:53:04.0000000Z-	980,698483

The output is configured as Power BI. In Power BI I have set up a simple table to show the events. As you can see there is some data missing between 16.04.2020 04:52:38 and 16.04.2020 04:52:57.

time	Conductivity
16.04.2020 04:53:08	981,10
16.04.2020 04:53:07	981,00
16.04.2020 04:53:06	980,90
16.04.2020 04:53:05	980,80
16.04.2020 04:53:04	980,70
16.04.2020 04:53:03	980,60
16.04.2020 04:53:02	980,50
16.04.2020 04:53:01	980,40
16.04.2020 04:53:00	980,30
16.04.2020 04:52:59	980,20
16.04.2020 04:52:58	980,10
16.04.2020 04:52:57	980,00
16.04.2020 04:52:38	978,00
16.04.2020 04:52:37	978,00
16.04.2020 04:52:36	977,90
16.04.2020 04:52:35	977,80
16.04.2020 04:52:34	977,70
16.04.2020 04:52:33	977,60

This is happening nearly every minute.

1. Is there a way to debug the connection between Azure Stream Analytics Output and Power BI?
2. Is there a limit of data which I can transfer per second?

Thank you for your help. If any information is missing please do not hesitate to ask for it!

[Greg_Deckler]

So are you using push or streaming? I am guessing push?

https://docs.microsoft.com/en-us/power-bi/service-real-time-streaming

[fr1337ddy]

I am using Azure Stream Analytics to transfer the data to Power BI, so I think I use "push" then (Using Azure Stream Analytics to push data).

I have configured "Power BI" as an output for my Stream Analytics job. In Power BI I have created a report from the dataset which was generated by the Stream Analytics job.

I also have created a dashboard which shows an equal behaviour: The mesaurements refresh every second and at a certain point everything stands still for 20s (like I mentioned in my first post). And so on (measurements do refresh for ~40s and stop for ~20s...).

[V-pazhen-msft]

@fr1337ddy 

There isn't many details in the Power BI document, I would recommend you to follow the Azure Steram Analytic document: https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-power-bi-dashboard

There are 2 types of logs for debugging:

https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-job-diagnostic-logs

For further assistance, you will get more help and confirmed answers here:  Azure Stream Analytics forum.

Paul Zheng _ Community Support Team 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

[fr1337ddy · Accepted solution]

Hello @V-pazhen-msft,

thank you for you reply.

After a further investigation I came to the result that my IoT device collected data until it has reached 20KB or 60s have expired. I decreased the collected data to 4KB so the sending interval is shorter. Now, there is no data lost anymore.

Best Wishes,

HelgeFP