cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
learn2fly
Helper I
Helper I

Power query fuzzy clustering with large data sets

Hello,

 

Wondering if anyone has any suggestions or best practice regarding Power Query Clustering Values with large dataset(dataset has over 1.5million rows). I've tryied just about everything to reduce the lag and improve performance but with no luck.

learn2fly_0-1669935958905.png

it buffers or gets stock for hours without any results:

learn2fly_1-1669936138924.png

 

Any advice or recommandation is greatly appriacate, thank you!

 

 

1 ACCEPTED SOLUTION
yingyinr
Community Support
Community Support

Hi @learn2fly ,

Before applying the fuzzy cluster, we can refer the following link to optimize the data model to improve the performance.

Optimizing the data model

 

As any fuzzy operation on the dataset will bring a huge performance impact to the data processing. The reason is that every text value has to be compared with every other text value in the table, the similarity threshold of the two has to be calculated (based on the algorithm above). This process is a very time-consuming process. Especially with more rows in the data table, you will feel it much more... Please find more details about Fuzzy Clustering...

Fuzzy Clustering in Power BI using Power Query: Finding similar values

Best Regards

Community Support Team _ Rena
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

1 REPLY 1
yingyinr
Community Support
Community Support

Hi @learn2fly ,

Before applying the fuzzy cluster, we can refer the following link to optimize the data model to improve the performance.

Optimizing the data model

 

As any fuzzy operation on the dataset will bring a huge performance impact to the data processing. The reason is that every text value has to be compared with every other text value in the table, the similarity threshold of the two has to be calculated (based on the algorithm above). This process is a very time-consuming process. Especially with more rows in the data table, you will feel it much more... Please find more details about Fuzzy Clustering...

Fuzzy Clustering in Power BI using Power Query: Finding similar values

Best Regards

Community Support Team _ Rena
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Helpful resources

Announcements
Carousel_PBI_Wave1

2023 Release Wave 1 Plans

Power BI release plans for 2023 release wave 1 describes all new features releasing from April 2023 through September 2023.

Power BI Summit Carousel 2

Global Power BI Training

Make sure you register today for the Power BI Summit 2023. Don't miss all of the great sessions and speakers!

BizApps LATAM 2023

Business Application LATAM Summit 2023

Join the biggest FREE Business Applications Event in LATAM this February.

Power Platform Bootcamp

Global Power Platform Bootcamp

In this bootcamp we will deep-dive into Microsoft’s Power Platform stack with hands-on sessions and labs, delivered to you by experts and community leaders.