cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
fengel
New Member

Flatten a giant Json DB.

Hello there,

Before everything a big thanks, for any help received.
Well, I'm not a BI developer, I'm actually a C# game developer adventuring here.

I have an extensive data set, from one of the games I'm developing, registering some data about the gameplay for each match for each user.

With a small data set, I could flatten all the matches, as I needed, with the code:

WyBf0XCTmF.png

 my issues appear applying this to the real production data, which has more than 26k users (in this code, the total rows there, and then each user could have N number of matches recorded).

The query runs(is running) for more than 1hour, and I don't know the end of it.
Is there a way to optimize this first query, just flattening the records into one big table that I'll use later to generate insights ?

Thanks in advance

1 REPLY 1
otravers
Super User
Super User

There's probably a way to optimize your M code, but in your situation where you're in charge of the app, I'd flatten the JSON into one or more tables (depending on data structure) in a staging database which I'd use as the source for Power BI. Power Query is convenient but it's not fast. Also you won't be able to do incremental refresh with a file source so things will only get worse as your dataset grows over time.

------------------------------------------------
1. How to get your question answered quickly - good questions get good answers!
2. Learning how to fish > being spoon-fed without active thinking.
3. Please accept as a solution posts that resolve your questions.
------------------------------------------------
More tips and guidance in my Power BI architecture and development blog

Helpful resources

Announcements
September Update

Check it Out!

Click here to learn more about the September 2022 updates!

Power BI Dev Camp Session 26

Check it Out!

Mark your calendars and join us on Thursday, September 29 at 11a PDT for a great session with Ted Pattison!

Top Solution Authors