Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Grow your Fabric skills and prepare for the DP-600 certification exam by completing the latest Microsoft Fabric challenge.

Reply
Anonymous
Not applicable

Count number of records in a table power query

Hi, I have a power query that is extracting the data using the rest API. 

My requirement is to run the query first to extract all the records using the below query:

let
  Pagination = List.Skip(List.Generate( () => [WebCall=[result = {0}], Page = 0, Counter=0], // Start Value
    each List.Count([WebCall][result])>0 or [Counter] =0// Condition under which the next execution will happen
    [Query=[sysparm_offset =Text.From([Counter])]])),
    Page = [Page]+1,
    Counter = [Counter]+1000// internal counter
    ]
    ) ,1),

this will extract all the records from the table. Now I want to run the same query but only want to extract records updated in the last 7 days. What changes I need to make in the query so it should check if there are already records in the table(power query) and

1.if not it will extract the entire dataset 

2. if there are records in the table then it should only extract the updated dataset. Something like this

https://<host>/api/now/table/incident?sysparm_query=sys_created_onONToday%40javascript%3Ags.beginningOfToday()%40javascript%3Ags.endOfToday()%5EORsys_updated_onBETWEENjavascript%3Ags.beginningOfToday()%40javascript%3Ags.beginningOfLast7Days()

 

 

1 ACCEPTED SOLUTION
Mariusz
Community Champion
Community Champion

Hi @Anonymous 

 

Power Query does not have a capability to store the data, so every time you run the script it will pull the data preview egain or it will pull whole data when you load into the model.

You could use dataflows to simulate your scenerio with incremental refresh if your sorce suports it and you are lucky enough to be on premium.

You can think of Power Query as a ETL tool not a storage db.

Please see the below for the ref.

https://docs.microsoft.com/en-us/power-query/power-query-what-is-power-query

 

 

Best Regards,
Mariusz

If this post helps, then please consider Accepting it as the solution.

Please feel free to connect with me.
LinkedIn

 

View solution in original post

1 REPLY 1
Mariusz
Community Champion
Community Champion

Hi @Anonymous 

 

Power Query does not have a capability to store the data, so every time you run the script it will pull the data preview egain or it will pull whole data when you load into the model.

You could use dataflows to simulate your scenerio with incremental refresh if your sorce suports it and you are lucky enough to be on premium.

You can think of Power Query as a ETL tool not a storage db.

Please see the below for the ref.

https://docs.microsoft.com/en-us/power-query/power-query-what-is-power-query

 

 

Best Regards,
Mariusz

If this post helps, then please consider Accepting it as the solution.

Please feel free to connect with me.
LinkedIn

 

Helpful resources

Announcements
RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

MayPowerBICarousel1

Power BI Monthly Update - May 2024

Check out the May 2024 Power BI update to learn about new features.

Top Solution Authors
Top Kudoed Authors