Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
Prefsmaster
Regular Visitor

Pushing data to PowerBI Streaming dataset API fails when historic data analysis is ON

The Topic says it all.

I have successfully created an API and was happily pushing events to it.

Then I created an Azure Webjob that pushes some random data every second or so.

Data flows nicely and I can see live gauges and line-charts!

Next, I enabled historic data analysis so I could try to visualize my events with a custom visual.

BOOM. Immediately I got 'Bad Request' failures:

Making POST request with data: {"timestamp":"2017-02-28T14:48:09.058","queuelength":75.0,"corridoroccupancy":8.0,"maxcorridor":10.0}
Exception thrown: 'System.Net.WebException' in System.dll
System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.GetResponse()

When I disable the historic analysis everything goes back to normal.

Anybody an idea of what I may be doing wrong?

Is there some data field that is mandatory when historic data is on?

If yes, where is that documented?

 

1 ACCEPTED SOLUTION

@Prefsmaster

What is the JSON format in your case?
In my test, before enabling "HISTORIC" I can upload data with 

{
"DATETM" :"2017-03-02T09:04:48.837Z",
"SALES" :5000000
}

However after enabling "HISTORIC", I got a 400 error though the upload data would finally get saved as in your case. The reponse contains  the the error saying something wrong with the dataset JSON. In case you have interest, the tool is POSTMAN

Capture.PNG

 

Based on my test, with a dataset JSON format as below could always work either the "HISTORIC" is enabled or disabled.

[
{
"DATETM" :"2017-03-02T10:04:48.837Z",
"SALES" :45000000
} 
]

You can actually send multiple data in JSON array.

[
{
"DATETM" :"2017-03-02T10:04:48.837Z",
"SALES" :45000000
},

{
"DATETM" :"2017-03-01T09:04:48.837Z",
"SALES" :38000000
}
]

 

View solution in original post

14 REPLIES 14
v-shex-msft
Community Support
Community Support

Hi @Prefsmaster,


I'm not clear how did you push the data to power bi streaming api, perhaps you can try to use below sample application to push data:

Power BI Samples

 

Reference articles:

Real-time streaming in Power BI

Create Dataset

 

In addition, if above are not help, can you please share some detail content of your issue?

 

Regards,

Xiaoxin Sheng

Community Support Team _ Xiaoxin
If this post helps, please consider accept as solution to help other members find it more quickly.

Afterthought: I know those articles you mention, andused them to get as far as I am now 🙂

I am now using this bit from the first referenc article you mention:

API3.png

I understood from the article that BOTH the URL and AAD approaches should work with an API created through the service UI.

Perhaps it is wrong, and is it mandatory to use AAD auth when you want to use Historic data analysis.

 

Ferenc

Hi, I am using the endpoint copied from the PowerBI UI:API1.png

 

 

 

 

 

 

 

 

 

 

 

And I wrote a small Command Line Tool to Send JSON to that endpoint (URL is stored in 'realTimePushURL'):

            Debug.WriteLine($"Making POST request with data: {jsonString}");

            var bodyData = Encoding.UTF8.GetBytes(jsonString);

            WebRequest request = WebRequest.Create(realTimePushURL);
            request.Method = "POST";
            request.ContentLength = bodyData.Length;

            var dataStream = request.GetRequestStream();
            dataStream.Write(bodyData, 0, bodyData.Length);
            dataStream.Close();

            WebResponse response = request.GetResponse();

I can send you the full source code of my small tool, but I think this won't help much: 

The fact is that sending data to the streaming API with it works fine, as long as I don't turn on 'Historic data analysis' in PBI:

API2.png

Ferenc

 

 

 

 

Hi @Prefsmaster,

 

I try to turn off the 'Historic' and the tile still works well. Perhaps you can try to test with the above sample.

 

Regards,

Xiaoxin Sheng

Community Support Team _ Xiaoxin
If this post helps, please consider accept as solution to help other members find it more quickly.

Yes I know.

In my original question I mentioned that it all works fine with Historic data OFF.

It stops when I turn it ON.

 

I will also try sending data using a bearer token, and send you the results.

 

If that solves the problem, the documentation in that reference article (Real-time streaming in Power BI) should be edited, because it seems to suggest that you can send data to such an API without AAD auth.

 

 

@v-shex-msft

 

Just found this in Github: https://github.com/Microsoft/PowerBI-CSharp/commit/8d6bd1b993a6b0d31a8d8d00e2f2e7852c9aef02

So the time format must be a string to support Historic data?

 

I made that change, but then te timestamp can't be used in the line-graph for streaming data, so that visual stops working.

 

Card and Gauge keep working because those don't use the timestamp.

 

When I enable Historic Data analysis I still get the "The remote server returned an error: (400) Bad Request." errors,

So the change to Text doesn't seem to help.

 

And I noticed something else: Data is still received by PowerBI, even though an HTTP error is returned!

So why the error is thrown I have no idea.

 

Next I added an extra field to the API, a timestamp in DateTime format. now I have both a string and a DateTime version of the timestamp.

By using the DateTime field as Axis in the line graph, that came back to life.

 

But I still see no historic data.

 

I will now start working on the AAD approach.

 

In the mean time, perhaps you have any Ideas?

 

Maybe you can pass my findings on to a developer?

@Prefsmaster

What is the JSON format in your case?
In my test, before enabling "HISTORIC" I can upload data with 

{
"DATETM" :"2017-03-02T09:04:48.837Z",
"SALES" :5000000
}

However after enabling "HISTORIC", I got a 400 error though the upload data would finally get saved as in your case. The reponse contains  the the error saying something wrong with the dataset JSON. In case you have interest, the tool is POSTMAN

Capture.PNG

 

Based on my test, with a dataset JSON format as below could always work either the "HISTORIC" is enabled or disabled.

[
{
"DATETM" :"2017-03-02T10:04:48.837Z",
"SALES" :45000000
} 
]

You can actually send multiple data in JSON array.

[
{
"DATETM" :"2017-03-02T10:04:48.837Z",
"SALES" :45000000
},

{
"DATETM" :"2017-03-01T09:04:48.837Z",
"SALES" :38000000
}
]

 

Thank you for your tests and input.

I supplied the data I am sending in the error quote (see original question at top of the thread):

Making POST request with data: {"timestamp":"2017-02-28T14:48:09.058","queuelength":75.0,"corridoroccupancy":8.0,"maxcorridor":10.0}

As you can see my JSON isn't sent as an array (doesn't have the [ ] around it).

I will try to add that, and see what happens.

 

@Prefsmaster

Based on my test, when "HISTORIC" enabled, no matter an array or not, it has to be wrapped with [].

Anonymous
Not applicable

Man, what a save your post was to me today!! Need to perform demo's on an event tomorrow and couldn't get the historical analysis working from Python untill I saw your post. Thanks.

@Eric_Zhang

 

That did the trick!

Thanks a lot. It would make more sense if the error also appears when HISTORIC is OFF though 🙂

Or clearly explain in the documentation and examples what should be sent...

 

 

@Prefsmaster

 

The product team will update the documentation to avoid this confusion.

@Prefsmaster

It is great to hear that your issue has gone. If no further questions, could you please close accept a solution to close this thread?

 

Meanwhile, to reduce confusion, I've already escalated this behavior to the Product team.

I posted this in 'Service' yesterday, but it actually belongs here. Maybe an admin can remove the other one?

The situation is:

I have successfully created an API and was happily pushing events to it.

Then I created an Azure Webjob that pushes some random data every second or so.

Data flows nicely and I can see live gauges and line-charts!

Next, I enabled historic data analysis so I could try to visualize my events with a custom visual.

BOOM. Immediately I got 'Bad Request' failures:

Making POST request with data: {"timestamp":"2017-02-28T14:48:09.058","queuelength":75.0,"corridoroccupancy":8.0,"maxcorridor":10.0}
Exception thrown: 'System.Net.WebException' in System.dll
System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.GetResponse()

When I disable the historic analysis everything goes back to normal.

Anybody an idea of what I may be doing wrong?

Is there some data field that is mandatory when historic data is on?

If yes, where is that documented?

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors