Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
Hi,
I have a question if it is possible to set up an incremental refresh in Power BI for a data source that does not support query folding.
In our case data source for Power BI is Elastic Search. As there is no native connector we implemented REST API endpoint + power query M script that query data in batches using elastic scroll API. Currently, one-time refresh takes about 1,5 hour and consumes ~ 5GB of memory which becomes problematic.
Is it possible to somehow combine custom data source which is API endpoint and incremental refresh?
BR
Hi @ezarebska ,
Current power bi does not support to do custom on incremental refresh feature, maybe you can submit an idea for this requirement.
Regards,
Xiaoxin Sheng
Hi,
thanks for your reply @v-shex-msft . How about a solution that power query M script would call my API endpoint with parameters RangeStart and RangeEnd? Question is if the received data batch will override existing data or just add new entries.
BR
HI @ezarebska ,
If the performance issue is caused by invoking web connectors with API to many times, I also not find effective solutions to increase performance.
Maybe you can try to add buffer functions to confirm if they work on your scenario.
How and When to use List & Table Buffer?
Regards,
Xiaoxin Sheng
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the April 2024 Power BI update to learn about new features.
User | Count |
---|---|
104 | |
101 | |
79 | |
72 | |
64 |
User | Count |
---|---|
142 | |
108 | |
101 | |
81 | |
74 |