Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.

Reply
donalmcnamee
New Member

Ideas on how to smooth out a noisy data set

I've got a data set that's a little noisy and I'm looking for ideas as to how it can be smoothed-out.
 
The chart below plots the volume of a liquid in a tank on a moving vehicle.
The volume of liquid in the tank reduces as we move through the day.
 
graph.png
The sensor in the tank is recording the volume approx. every 6 minutes or so.
However, as the vehicle is moving, the liquid is inclined to slosh around a bit. And if a sample is triggered when the vehicle is moving, it's inclined to read either a slightly elevated or slighted reduced reading.
 
I'm wondering if there's an elegant way to smooth-out these peaks and troughs in the data set?
 
 
Ideas appreciated.

 

 

1 ACCEPTED SOLUTION
d_gosbell
Super User
Super User

One way to smooth out a line like this is to use a moving average.  The following code creates a moving 20 minute average. You can change the number of minutes for the moving average by changing the variable called _minsToAvg

Tank Level Smoothed = 
var _minsToAvg = 20
var _dayFraction = _minsToAvg / (24*60)
var _currentDateTime = MAX( data[EventDate])
var _startOfAvg = _currentDateTime - _dayFraction
var result = AVERAGEX (
    FILTER (
        ALL ( data ),
        data[EventDate]  > _startOfAvg 
            && data[EventDate]  <= _currentDateTime  
    ),
    CALCULATE(SUM(data[TankLevel]))
)
return  result

 The above measure produces the following output

2020-05 moving average.png

 

 

View solution in original post

4 REPLIES 4
d_gosbell
Super User
Super User

One way to smooth out a line like this is to use a moving average.  The following code creates a moving 20 minute average. You can change the number of minutes for the moving average by changing the variable called _minsToAvg

Tank Level Smoothed = 
var _minsToAvg = 20
var _dayFraction = _minsToAvg / (24*60)
var _currentDateTime = MAX( data[EventDate])
var _startOfAvg = _currentDateTime - _dayFraction
var result = AVERAGEX (
    FILTER (
        ALL ( data ),
        data[EventDate]  > _startOfAvg 
            && data[EventDate]  <= _currentDateTime  
    ),
    CALCULATE(SUM(data[TankLevel]))
)
return  result

 The above measure produces the following output

2020-05 moving average.png

 

 

I am trying this with the data set and i get a circular dependency error?

Could you explain what i am doing wrong?

 

Rogiervanweert_0-1596025768856.png

 

@Rogiervanweert  Use a measure instead of a calculated column.

/ J


Connect on LinkedIn

@d_gosbell 

Thanks a million for that. Exactly what I was looking for!

 

Much appreciated.

Helpful resources

Announcements
RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

MayPowerBICarousel

Fabric Monthly Update - May 2024

Check out the May 2024 Fabric update to learn about new features.

LearnSurvey

Fabric certifications survey

Certification feedback opportunity for the community.