Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more.
Get startedGrow your Fabric skills and prepare for the DP-600 certification exam by completing the latest Microsoft Fabric challenge.
Hi all,
Currently my laptop have 16gb of RAM and I working with million of data in Powerbi. It take forever to load 100million of data now. So my action is to add additional RAM into my laptop from 16gb to 64gb.
Do you think that the data load of power query will be faster than before after I upgrade my RAM?
Regards,
Nuha
Solved! Go to Solution.
I can offer you some intuitive advice, just from long experience with Power Query.
Thats a lot of filters, before an unpivot and then some more filters, and then removing duplicates. There's no way query folding is surviving that--I guarantee it's folding no further than after you remove duplicates,
It looks like these are all tables from the same database--they may even be properly keyed and indexed. I would do all of the table joins after one or two of the filters that cut down the most data--but as far upstream as possible. I would left join them as Table.Join, as opposed to Table.NestedJoin, so you may have to adjust some column names for your joins. Nevertheless, I am often surprised at how easily you can join quite a few tables without issue, because it's all folding to SQL, which can handle it easily. "Do all the joins while you are still folding, no matter how many rows it returns."
You could probably still fold after applying the current month fillter--which you should calculate as a single date value in another query Named ThisMonthStart, so that you have that single date for your main query to use as a filter, and consequently for Power Query to turn into SQL, instead of however you might be doing it now.
ThisMonthStart = let DayNumber = (Date.Day(DateTime.LocalNow())*-1) in Date.AddDays(Date.From(DateTime.LocalNow()), DayNumber)
And then you can make your main query's #"Filter Only Current Month" step can be
Table.SelectRows(PriorStepName, each [DateColumnName] > ThisMonthStart)
So apply the date filter before the joins and see if that helps.
If you are using some of these join steps merely to select a list of values in your table against another table (as in, you are just getting rid of all the columns in the other table right away), and its not too many values or a lot of strings, consider filtering instead, using a buffered list, like
Table.SelectRows(PriorStepName, each List.Contains(List.Buffer(OtherTableQuery[ValuesToKeepColumn]), [ThisTableColumn]))
That gives SQL an IN statement with those values in it.
Make sure you are extracting dates using Date.From, not DateTime.Date. They claim DateTime.Date folds, but it will fail you in places where Date.From succeeds.
The Unpivot steps are not necessarily hit or miss as far as folding, but it's not something I can define here.
The group will likely always be slow, so do it as far down as possible, like you already have it.
OK, good luck!
--Nate
I can offer you some intuitive advice, just from long experience with Power Query.
Thats a lot of filters, before an unpivot and then some more filters, and then removing duplicates. There's no way query folding is surviving that--I guarantee it's folding no further than after you remove duplicates,
It looks like these are all tables from the same database--they may even be properly keyed and indexed. I would do all of the table joins after one or two of the filters that cut down the most data--but as far upstream as possible. I would left join them as Table.Join, as opposed to Table.NestedJoin, so you may have to adjust some column names for your joins. Nevertheless, I am often surprised at how easily you can join quite a few tables without issue, because it's all folding to SQL, which can handle it easily. "Do all the joins while you are still folding, no matter how many rows it returns."
You could probably still fold after applying the current month fillter--which you should calculate as a single date value in another query Named ThisMonthStart, so that you have that single date for your main query to use as a filter, and consequently for Power Query to turn into SQL, instead of however you might be doing it now.
ThisMonthStart = let DayNumber = (Date.Day(DateTime.LocalNow())*-1) in Date.AddDays(Date.From(DateTime.LocalNow()), DayNumber)
And then you can make your main query's #"Filter Only Current Month" step can be
Table.SelectRows(PriorStepName, each [DateColumnName] > ThisMonthStart)
So apply the date filter before the joins and see if that helps.
If you are using some of these join steps merely to select a list of values in your table against another table (as in, you are just getting rid of all the columns in the other table right away), and its not too many values or a lot of strings, consider filtering instead, using a buffered list, like
Table.SelectRows(PriorStepName, each List.Contains(List.Buffer(OtherTableQuery[ValuesToKeepColumn]), [ThisTableColumn]))
That gives SQL an IN statement with those values in it.
Make sure you are extracting dates using Date.From, not DateTime.Date. They claim DateTime.Date folds, but it will fail you in places where Date.From succeeds.
The Unpivot steps are not necessarily hit or miss as far as folding, but it's not something I can define here.
The group will likely always be slow, so do it as far down as possible, like you already have it.
OK, good luck!
--Nate
Hi @watkinnc ,
Woow what a long statement and thank you for all of your suggestion.
But can you explain more what is "folding to SQL" means? What is "fold" even means?
Regards,
Nuha
Food means "to send a query to a data source by transforming the M query into the language of the data source". For example, if you query a SQL Server database, Power Query can fetch and transform the data using SQL, and the work gets done at the data source--which is almost always faster and less work for your computer. If your query is too complex, or uses certain transforms, then Power Query will instead import the data and transform it on your machine, usually in chunks and at the cost of your limited computer resources.
I've run thousands of complex queries, against just about every data source, and 16GB of RAM has pretty much always been enough.
Try my strategies!
--Nate
"It depends." Can we have a look at your query?
--Nate
Hi @watkinnc ,
I'm not sure from which view you want to see my query. I have some screenshot for you below. Currently I'm loading 50million of data (already more than 4 hour).
Regards,
Nuha