Where is my Historic Data? I can see events from past hour only.

[SergeSemenov]

Components:

- newly created Azure Event Hub that receives events

- newly created Azure Stream Analytics Job that processes events from the event hub with tumbling window of 1 minute and pushes output to Power BI online

- newly created Power BI workspace with a simple report

Events pushed to Power BI:

- 4,500 events per minute

- 500 Kb of event data per minute

Problem:

In any chart I can see events only for the last hour only (51 minutes to be precise), even the explicit filtering on date/time never shows more data.

Additional info:

- The stream analytics job shows up in "Streaming Datasets" - seems like a recent change by Power BI

- The "Historic data analysis" option is on

Update:

- Reducing frequency of sending events to Power BI from 1 minute to 5 minutes resulted in showing data for the past 4 hours only, but still my target is to be able to see a 12 or 24-hour window without sacrificing the granurality of data.

[GilbertQ]

Hi @SergeSemenov, in this scenario I would suggest looking to use SSAS to store your data, and then connect your SSAS instance to Power BI.

In doing so, this can store all the data you require as well as then give you the drill down you need?

You could look at the new SSAS in Auzre as an SaaS?

In the past I have dealt with SCOM data, and created exception reporting, which was based off SSAs and the report would only show exceptions, which made it a lot easier to find issues with the Servers/VMs

[SergeSemenov · Accepted solution]

I finally found what the problem is.

Power BI has a limitation on how many events and/or how much data you can load into a report!

And this setting is not configurable.

In my case we have 1000 VMs running in the Azure Cloud which publish events on various perf counters. And in my report I want to have slicers which helps to drill down to a problem root. That means, for every single point in time I have e.g. 2000 variables I can filter by. Let's say every point of time is 1 minute, so if Power BI shows about 50 minutes of data, that means it's limitation is about 100,000 events per report. And it does not matter if you apply a filter for the whole report, seems like it cannot go back in history passing the 100K threshold.

I wish that can be configurable - we don't care if a browser page takes 10 seconds to load and 500MB of memory - we just would like to have a simple tool which allows simple slide and dice feature on huge amount of data.

Added an idea to extend that limit:

https://ideas.powerbi.com/forums/265200-power-bi-ideas/suggestions/17135662-extend-the-number-of-eve...

[v-haibl-msft]

@SergeSemenov

Could you please check the storage you used in Service? You can find it in “Manage storage” to see if there is enough storage remained.

Best Regards,

Herbert