cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Paul_keijzers
Frequent Visitor

help iterate trhough webpages till no webpage

Hi 

 

I am trying to loop trough webpages and have this code below but it gives me a error on next.. so i am doing something wrong here. I hope someone can help me fix it. And it would be great if i can schedule refres in the end.

 

Below the query i am using:

let

    Source = List.Generate( ()=>
[Result = Json.Document(Web.Contents("https://tenantx.api-us1.com/api/3/accounts", [Headers=[#"Api-Token"="123456789"]])), Counter=0, Next = Json.Document(Web.Contents("https://tenantx.api-us1.com/api/3/accounts", [Headers=[#"Api-Token"="123456789"]]))[next]],
each [Next] <> null,
each [
Next = [Result][next],
Result = let
        next = [Result][next],
        offset = Text.BetweenDelimiters( next, "offset=", "&"),
        limit = Text.AfterDelimiter(next, "limit="),
        StartWithEmptyTable = Json.Document(Web.Contents("https://tenantx.api-us1.com/api/3/accounts/?offset=0&limit=20", [Query=[offset=Text.From(offset), limit=Text.From(limit)]],[Headers=[#"Api-Token"="123456789"]]))
    in
        StartWithEmptyTable,
Counter = [Counter] + 1
]
),
    #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    #"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Result", "Counter"}, {"Result", "Counter"}),
    #"Expanded Result" = Table.ExpandRecordColumn(#"Expanded Column1", "Result", {"count", "next", "previous", "results"}, {"count", "next", "previous", "results"}),
    #"Expanded results" = Table.ExpandListColumn(#"Expanded Result", "results"),
    #"Expanded results1" = Table.ExpandRecordColumn(#"Expanded results", "results", {"name", "url"}, {"name", "url"})
in
    #"Expanded results1"

 

and this is the result i get when using postman to query directly.

{
    "accounts": [
        {
            "name": "Rivalss",
            "accountUrl": "",
            "createdTimestamp": "2021-03-22T13:41:41-05:00",
            "updatedTimestamp": "2021-03-22T13:41:42-05:00",
            "contactCount": "0",
            "dealCount": "0",
            "links": {
                "notes": "https://tenantx.api-us1.com/api/3/accounts/4/notes",
                "accountCustomFieldData": "https://tenantx.api-us1.com/api/3/accounts/4/accountCustomFieldData",
                "accountContacts": "https://tenantx.api-us1.com/api/3/accounts/4/accountContacts",
                "emailActivities": "https://tenantx.api-us1.com/api/3/accounts/4/emailActivities",
                "contactEmails": "https://tenantx.api-us1.com/api/3/accounts/4/contactEmails"
            },
            "id": "4"
        },
        {
            "name": "Joske",
            "accountUrl": null,
            "createdTimestamp": "2021-03-23T11:32:05-05:00",
            "updatedTimestamp": "2021-03-23T11:32:05-05:00",
            "contactCount": "0",
            "dealCount": "0",
            "links": {
                "notes": "https://tenantx.api-us1.com/api/3/accounts/9/notes",
                "accountCustomFieldData": "https://tenantx.api-us1.com/api/3/accounts/9/accountCustomFieldData",
                "accountContacts": "https://tenantx.api-us1.com/api/3/accounts/9/accountContacts",
                "emailActivities": "https://tenantx.api-us1.com/api/3/accounts/9/emailActivities",
                "contactEmails": "https://tenantx.api-us1.com/api/3/accounts/9/contactEmails"
            },
            "id": "9"
        },
        {
            "name": "spxvb",
            "accountUrl": "",
            "createdTimestamp": "2021-03-24T02:33:42-05:00",
            "updatedTimestamp": "2021-03-24T02:33:43-05:00",
            "contactCount": "0",
            "dealCount": "0",
            "links": {
                "notes": "https://tenantx.api-us1.com/api/3/accounts/10/notes",
                "accountCustomFieldData": "https://tenantx.api-us1.com/api/3/accounts/10/accountCustomFieldData",
                "accountContacts": "https://tenantx.api-us1.com/api/3/accounts/10/accountContacts",
                "emailActivities": "https://tenantx.api-us1.com/api/3/accounts/10/emailActivities",
                "contactEmails": "https://tenantx.api-us1.com/api/3/accounts/10/contactEmails"
            },
            "id": "10"
        },
        {
            "name": "yep",
            "accountUrl": null,
            "createdTimestamp": "2021-03-24T13:59:49-05:00",
            "updatedTimestamp": "2021-03-24T13:59:49-05:00",
            "contactCount": "0",
            "dealCount": "0",
            "links": {
                "notes": "https://tenantx.api-us1.com/api/3/accounts/13/notes",
                "accountCustomFieldData": "https://tenantx.api-us1.com/api/3/accounts/13/accountCustomFieldData",
                "accountContacts": "https://tenantx.api-us1.com/api/3/accounts/13/accountContacts",
                "emailActivities": "https://tenantx.api-us1.com/api/3/accounts/13/emailActivities",
                "contactEmails": "https://tenantx.api-us1.com/api/3/accounts/13/contactEmails"
            },
            "id": "13"
        },
        {
            "name": "Org",
            "accountUrl": null,
            "createdTimestamp": "2021-03-29T01:34:02-05:00",
            "updatedTimestamp": "2021-03-29T01:34:02-05:00",
            "contactCount": "0",
            "dealCount": "0",
            "links": {
                "notes": "https://tenantx.api-us1.com/api/3/accounts/2175/notes",
                "accountCustomFieldData": "https://tenantx.api-us1.com/api/3/accounts/2175/accountCustomFieldData",
                "accountContacts": "https://tenantx.api-us1.com/api/3/accounts/2175/accountContacts",
                "emailActivities": "https://tenantx.api-us1.com/api/3/accounts/2175/emailActivities",
                "contactEmails": "https://tenantx.api-us1.com/api/3/accounts/2175/contactEmails"
            },
            "id": "2175"
        }
    ],
    "meta": {
        "total": "9425"
    }
}

 

1 ACCEPTED SOLUTION

Hi  @Paul_keijzers ,

 

Based on my understanding,if you could get all the data from all the webpages and the data can be refreshed automatically,then you will realize" iterate trhough webpages till no webpage",check below:

https://social.technet.microsoft.com/Forums/en-US/0f9fec02-4469-4c56-92ec-00d46b26b3fc/how-to-automa...

 

https://www.pbiusergroup.com/communities/community-home/digestviewer/viewthread?MessageKey=1c86e2d9-...

 

 

Best Regards,
Kelly

Did I answer your question? Mark my post as a solution!

View solution in original post

4 REPLIES 4
v-kelly-msft
Community Support
Community Support

Hi  @Paul_keijzers ,

 

After researching, you could refer to below blog:

Loop through Multiple Web Pages using Power Query

 

Best Regards,
Kelly

Did I answer your question? Mark my post as a solution!

 

 

 

Thank you for your reply but here i need to state the rows upfront that i want to query. As the rows are expanding each day it needs to be variable.

In C# you have a function for each that would be interesting to have here as well for each till max meta.total

Hi  @Paul_keijzers ,

 

Based on my understanding,if you could get all the data from all the webpages and the data can be refreshed automatically,then you will realize" iterate trhough webpages till no webpage",check below:

https://social.technet.microsoft.com/Forums/en-US/0f9fec02-4469-4c56-92ec-00d46b26b3fc/how-to-automa...

 

https://www.pbiusergroup.com/communities/community-home/digestviewer/viewthread?MessageKey=1c86e2d9-...

 

 

Best Regards,
Kelly

Did I answer your question? Mark my post as a solution!

View solution in original post

I think this would work but have found a new way by just updating the offset. Thanks!

Helpful resources

Announcements
PBI_User Group Leader_768x460.jpg

Manage your user group events

Check out the News & Announcements to learn more.

MBAS on Demand

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

Get Ready for Power BI Dev Camp

Microsoft named a Leader in The Forrester Wave

Microsoft received the highest score of any vendor in both the strategy and current offering categories.

R2 (Green) 768 x 460px.png

Microsoft Dynamics 365 & Power Platform User Professionals

DynamicsCon is a FREE, 4 half-day virtual learning experience for 11,000+ Microsoft Business Application users and professionals.

Top Kudoed Authors