I have a problem/question about importing data from a Sharepoint List and I found similar topics but not quite what I was looking for.
Problem: I need to import data from a Sharepoint list, which is generally running smoothly, but has one big disadvantage: With additional rows the list grows exponentially in size. For example, I have a list that barely has 10.000 rows (15 columns or so) but needs an import of 50MB. There are also larger lists to import (100k rows) and this poses a real problem for the scheduled refresh, because Sharepoint does not provide the most stable connection. So with an import of several 100 MBs, the refresh would fail quite often. The reason for this large import seems to be some relations in the background of Sharepoint, but I'm not too familiar with the structure of SP. Anyway, I need a smaller import size 🙂
Solution using REST API?: So I read somewhere, that it's maybe an option to import the data using the REST API. Could anyone explain to me how this would work? And apart from that: are there any other suggestions on how I could reduce the refresh size? Incremental refresh seems not to be an option, since Sharepoint does not support query folding.