Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
ancyphilip
Employee
Employee

Share your thoughts on the new Anomaly detection feature (preview)

Hit Reply to tell us what you think about the new Anomaly detection feature so we can continue to improve.

-Power BI AI team

59 REPLIES 59

Thanks for reporting the issue, Looking into it..

Capstone
Resolver I
Resolver I

Thank you. Much appreciated.

lisahua46
Employee
Employee

 

Anomaly detection tutorial - Power BI | Microsoft Docs

Check out this technical blog for more details about the Anomaly Detector algorithm.

User should get almost identical results of using Azure AnomalyDetection API vs PowerBI Anomaly Detection. 

 

The sensitivity is a way we allow users to tune the model with the understanding that different business or person could have different tolerance on even the same dataset. The higher the sensitivity, the narrower the band, as a result, you would get more anomalies.

 

The strength is defined as “Explanatory power”

“Explanatory power” is the percentage of change in the aggregate value at an anomaly point that is explained by the explanation value/time series. Concretely, it is calculated as the ratio of deviation (actual minus expected value) between the explanation/component time series (e.g., for Property type = Loft) and the aggregate time series for the anomaly point. See original paper

 

The implementation for AnomalyDetection and Explanation is open-sourced

Source code: AnomalyDetection Explanation

There is no relationship between sensitivity and strength.

 

Roughly speaking, for explain anomalies, we do:

  1. Given an “aggregate time series” (e.g., Listing count by date) with an anomaly on a given date, find anomalies in all “component time series” (Listing count by date where Property type = Loft, Listing count by date where Property type = Cabin, Listing count by date where Neighborhood = Capitol Hill etc.)
  2. Find the dimension (i.e., Property type or Neighborhood in the example) for which the component time series most likely show a root cause on the given date. This is based done using a decision tree (with classes being anomaly vs not anomaly) and the split criterion is Entropy Gain Ratio and additional filtering
  3. For the chosen dimension (e.g., Property type) rank dimension values (Loft, Cabin etc.) with anomalies on the given date according to an “explanatory power”
Capstone
Resolver I
Resolver I

Hi,

Can you provide some resources which give an intuitive understanding of the algorithm which would be easier to understand for business users. Questions like

 

What is sensitivity and how does change in sensitivity change the anomolies

How is the range calculated

What is strength %

What is the relationship between sensitivity and strength (if any)

Ex: If we are analysing anamolies for sales data and using Customer as Explain by and sensitivity at 70%. What does it mean when there is an anamoly at 70% sensitivity and strength is 45%.

 

It would greatly help as right now these terms are unclear. Without an intuitive understanding of these terms it would be difficult for business users to explain the data. 

 

unais
Helper V
Helper V

how possible explanation Get Works?

 

unais_0-1606296725914.png

 

1. You could specify the explain by fields by yourself  https://powerbi.microsoft.com/en-us/blog/anomaly-detection-preview/

2. If no field is specified in explainBy fields, we try some dimensions based on heurisitics, considering the performance. 

3. We invoke the Explanation API with these dimensions to perform a max-entropy decision tree trying to find explanation, machinelearning/time-series-root-cause-localization.md at master · dotnet/machinelearning (github.co... 

4. It can happen that none of the dimensions significantly impact the anomaly. Either because the dimension might not be selected (which user can input their insights by specifying dimensions), or because these dimensions are not significantly contribute to the anomaly based on decision-tree.

5. If you can share a repro pbix to us, we can look into details. (you can refer to a previous reply in this post with calculation group repro)

 

 

configure-explanations.gif

kushsoni
New Member

Hi Team, 

 

As this currently doesn't now support Directly Query from SQL, I have calculated my own upper and lower limits and wanted to use the chart that is being used here to show anomaly. Can you please share how I can create a similar chart where upper and lower limits are being shows as a range/band on actual value?

 

thanks,

Kush

Hi, can you share us a repro for  Directly Query from SQL? anomaly detection is supposed to work in this case. And we want to make sure we fix it if not working.

JakeStone
Frequent Visitor

Love this feature! Any chance of being able to use it with a visual pinned to a dashboard in the near future? Building out holiday reporting for next week and had to remove the anomaly detection from any visuals I was adding to the report since they wouldn't load, I'm assuming because it's not supported yet.

Hi JakeStone,

 

You can expect line charts containing anomaly detections to render in dashboards in early December (most likely the week of November 30th). 

yakovleva_j
Regular Visitor

Hi. I just wanted to say that this is a great feature)

deanismael
Advocate II
Advocate II

I'm observing the same behaviour @tez@ancyphilip - can you please advise and also let me know about support of this feature in Embedded. Thanks.

We haven't support Embedded scenario yet, we are planning to support it in next SU, i.e., next month. 

That's fantastic news. Thanks for confirming.

deanismael
Advocate II
Advocate II

Have you tried running anomaly detection with the date hierarchy set to date value only and then switching it back to the hierarchical view (ensuring the drilldown path for testing purposes has more than 12 data points - i.e. day of month)?

Hi @deanismael 

yes, I´ve tried; if I only use a date field (without hierarchy), it works. As soon as I use any hierarchy (independent of the hierarchy layer, more than 12 data points), the message appears as posted before.

tez
Resolver I
Resolver I

I get an error message when using a date hierarchy from my individual date table:

tez_0-1605517167370.png

Any clues, what could be the problem? How has a date hierarchy to be constructed, to be usable for anomaly detection?

Regards,

Thomas.

I think date hierarchy may look like below

lisahua46_0-1605542222592.png

lisahua46_1-1605542282311.png

And you can drill down using drill down icon. 

 

Your setting seems different from the setting above?

Hi @lisahua46 

no, it´s the same; the screenshot only shows a different part of the screen.

I suggest to choose the date and change it to date hierarchy as my screenshot above. Directly select part of date hierarchy might not contain the date as scalar key - which is needed for anomaly detection.

 

lisahua46_0-1605631561485.png

 

Hi @lisahua46 ,

this is the way I did it from the beginning.

Regards,

Thomas.

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.