Incremental refresh custom solution

[ezarebska]

Hi,
I have a question if it is possible to set up an incremental refresh in Power BI for a data source that does not support query folding.
In our case data source for Power BI is Elastic Search. As there is no native connector we implemented REST API endpoint + power query M script that query data in batches using elastic scroll API. Currently, one-time refresh takes about 1,5 hour and consumes ~ 5GB of memory which becomes problematic.
Is it possible to somehow combine custom data source which is API endpoint and incremental refresh?

BR

[v-shex-msft]

Hi @ezarebska ,

Current power bi does not support to do custom on incremental refresh feature, maybe you can submit an idea for this requirement.

Regards,

Xiaoxin Sheng

[ezarebska]

Hi,

thanks for your reply @v-shex-msft . How about a solution that power query M script would call my API endpoint with parameters RangeStart and RangeEnd? Question is if the received data batch will override existing data or just add new entries. 

BR

[v-shex-msft]

HI @ezarebska ,

If the performance issue is caused by invoking web connectors with API to many times, I also not find effective solutions to increase performance. 

Maybe you can try to add buffer functions to confirm if they work on your scenario.

How and When to use List & Table Buffer? 

Regards,

Xiaoxin Sheng