Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
ppalacin
Frequent Visitor

FHIR Connector is loading slowly large data sets

We are experiencing huge delays when loading big datasets from our FHIR Server.

I am talking about 1 Mio+ records, which take close to a day to fully load.

 

The main problem seems to be the way how the FHIR Connector is loading the data:

- Small sets with small offset (about 200 for each request)

- No utilization of FHIR export feature

 

Is there any way to tweak the connector e.g. by overwriting the default offset size?

Our server would have no problems to serve sth. like 25k records with each request.

 

 

 

3 REPLIES 3
ppalacin
Frequent Visitor

Any update on this issue?

The problem is still existing.

I honestly don't understand why the PBI FHIR Connector does not ship with an option to set the page size. Queryfolding is supported which is much harder to implement.

 

Providing a parameter to set the page size should be easy to be tackled by the team, would love to see it on the roadmap.

ppalacin
Frequent Visitor

Hi @DataInsights ,

 

thanks for the advice. It's pretty nice explained, but only works for local datasets.

However the problem with FHIR is that it takes the most time to load all datapoints 1by1 (buckets of 100-500) from the server instead of incresing the offset size. So you loose a lot of time with establishing unnecessary requests.

DataInsights
Super User
Super User

@ppalacin,

 

A colleague alerted me to this interesting technique, and got a 10x improvement in his query. Let me know if it works for you.

 

https://blog.crossjoin.co.uk/2020/05/14/speed-up-data-refresh-performance-in-power-bi-desktop-using-... 





Did I answer your question? Mark my post as a solution!

Proud to be a Super User!




Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.