Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Grow your Fabric skills and prepare for the DP-600 certification exam by completing the latest Microsoft Fabric challenge.

Reply
nuhasan
Helper IV
Helper IV

Add additional RAM into my laptop can make my query load faster?

Hi all,

 

Currently my laptop have 16gb of RAM and I working with million of data in Powerbi. It take forever to load 100million of data now. So my action is to add additional RAM into my laptop from 16gb to 64gb.

 

Do you think that the data load of power query will be faster than before after I upgrade my RAM?

 

Regards,

Nuha

1 ACCEPTED SOLUTION
watkinnc
Super User
Super User

I can offer you some intuitive advice, just from long experience with Power Query.

Thats a lot of filters, before an unpivot and then some more filters, and then removing duplicates.  There's no way query folding is surviving that--I guarantee it's folding no further than after you remove duplicates, 
It looks like these are all tables from the same database--they may even be properly keyed and indexed.  I would do all of the table joins after one or two of the filters that cut down the most data--but as far upstream as possible.  I would left join them as Table.Join, as opposed to Table.NestedJoin, so you may have to adjust some column names for your joins.  Nevertheless, I am often surprised at how easily you can join quite a few tables without issue, because it's all folding to SQL, which can handle it easily.  "Do all the joins while you are still folding, no matter how many rows it returns."
You could probably still fold after applying the current month fillter--which you should calculate as a single date value in another query Named ThisMonthStart, so that you have that single date for your main query to use as a filter, and consequently for Power Query to turn into SQL, instead of however you might be doing it now.
ThisMonthStart = let DayNumber = (Date.Day(DateTime.LocalNow())*-1) in Date.AddDays(Date.From(DateTime.LocalNow()), DayNumber)

 

And then you can make your main query's #"Filter Only Current Month" step can be

Table.SelectRows(PriorStepName, each [DateColumnName] > ThisMonthStart)

 

So apply the date filter before the joins and see if that helps.

 

If you are using some of these join steps merely to select a list of values in your table against another table (as in, you are just getting rid of all the columns in the other table right away), and its not too many values or a lot of strings, consider filtering instead, using a buffered list, like
Table.SelectRows(PriorStepName, each List.Contains(List.Buffer(OtherTableQuery[ValuesToKeepColumn]), [ThisTableColumn]))

 

That gives SQL an IN statement with those values in it.

 

Make sure you are extracting dates using Date.From, not DateTime.Date.  They claim DateTime.Date folds, but it will fail you in places where Date.From succeeds.
The Unpivot steps are not necessarily hit or miss as far as folding, but it's not something I can define here.
The group will likely always be slow, so do it as far down as possible, like you already have it.
OK, good luck!

 

--Nate


I’m usually answering from my phone, which means the results are visualized only in my mind. You’ll need to use my answer to know that it works—but it will work!!

View solution in original post

6 REPLIES 6
watkinnc
Super User
Super User

I can offer you some intuitive advice, just from long experience with Power Query.

Thats a lot of filters, before an unpivot and then some more filters, and then removing duplicates.  There's no way query folding is surviving that--I guarantee it's folding no further than after you remove duplicates, 
It looks like these are all tables from the same database--they may even be properly keyed and indexed.  I would do all of the table joins after one or two of the filters that cut down the most data--but as far upstream as possible.  I would left join them as Table.Join, as opposed to Table.NestedJoin, so you may have to adjust some column names for your joins.  Nevertheless, I am often surprised at how easily you can join quite a few tables without issue, because it's all folding to SQL, which can handle it easily.  "Do all the joins while you are still folding, no matter how many rows it returns."
You could probably still fold after applying the current month fillter--which you should calculate as a single date value in another query Named ThisMonthStart, so that you have that single date for your main query to use as a filter, and consequently for Power Query to turn into SQL, instead of however you might be doing it now.
ThisMonthStart = let DayNumber = (Date.Day(DateTime.LocalNow())*-1) in Date.AddDays(Date.From(DateTime.LocalNow()), DayNumber)

 

And then you can make your main query's #"Filter Only Current Month" step can be

Table.SelectRows(PriorStepName, each [DateColumnName] > ThisMonthStart)

 

So apply the date filter before the joins and see if that helps.

 

If you are using some of these join steps merely to select a list of values in your table against another table (as in, you are just getting rid of all the columns in the other table right away), and its not too many values or a lot of strings, consider filtering instead, using a buffered list, like
Table.SelectRows(PriorStepName, each List.Contains(List.Buffer(OtherTableQuery[ValuesToKeepColumn]), [ThisTableColumn]))

 

That gives SQL an IN statement with those values in it.

 

Make sure you are extracting dates using Date.From, not DateTime.Date.  They claim DateTime.Date folds, but it will fail you in places where Date.From succeeds.
The Unpivot steps are not necessarily hit or miss as far as folding, but it's not something I can define here.
The group will likely always be slow, so do it as far down as possible, like you already have it.
OK, good luck!

 

--Nate


I’m usually answering from my phone, which means the results are visualized only in my mind. You’ll need to use my answer to know that it works—but it will work!!

Hi @watkinnc ,

 

Woow what a long statement and thank you for all of your suggestion.

But can you explain more what is "folding to SQL" means? What is "fold" even means?

 

Regards,

Nuha

Food means "to send a query to a data source by transforming the M query into the language of the data source". For example, if you query a SQL Server database, Power Query can fetch and transform the data using SQL, and the work gets done at the data source--which is almost always faster and less work for your computer. If your query is too complex, or uses certain transforms, then Power Query will instead import the data and transform it on your machine, usually in chunks and at the cost of your limited computer resources.

 

I've run thousands of complex queries, against just about every data source, and 16GB of RAM has pretty much always been enough.

 

Try my strategies!

 

--Nate


I’m usually answering from my phone, which means the results are visualized only in my mind. You’ll need to use my answer to know that it works—but it will work!!

Hi @watkinnc ,

 

Okay sir. Noted. Thank you for your explanation and advice.

 

Regards,

Nuha

watkinnc
Super User
Super User

"It depends." Can we have a look at your query?

 

--Nate


I’m usually answering from my phone, which means the results are visualized only in my mind. You’ll need to use my answer to know that it works—but it will work!!

Hi @watkinnc ,

 

I'm not sure from which view you want to see my query. I have some screenshot for you below. Currently I'm loading 50million of data (already more than 4 hour).

 

Regards,

Nuha

 

nuhasan_1-1687328749982.png

 

 

 

nuhasan_0-1687328723096.png

 

nuhasan_2-1687328769973.png

 

 

Helpful resources

Announcements
RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

MayPowerBICarousel1

Power BI Monthly Update - May 2024

Check out the May 2024 Power BI update to learn about new features.

Top Solution Authors
Top Kudoed Authors