Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.

Reply
Anonymous
Not applicable

script running looping URLs and get data

Hi fellow Power BI users,

 

I know Power BI Desktop has 'From Web' feature to get data, but it can only process one URL at a time.  If I have multiple (but similar) URLs, I think a script running looping process is ideal.  Does anyone know how this can be done and processed in Power BI (sample script is preferred)?  Once I have this script, what are the steps to get Power BI process the script? 

 

URLs: - https://inteligence/apps/1/...

https://inteligence/apps/2/..

https://inteligence/apps/3/...

...

with Authorization - bearer <key>

 

Thanks in advance.

1 ACCEPTED SOLUTION

When you add a custom column (Invoke Custom Function), change the drop down to Column Name and select the app_id column.  Then, the function will pass in the app_id value from each row into the function (into the URL) and perform the query steps you designed earlier.  If you need to pass a bearer token to the URL for each app_id value, if it is the same bearer token for every app_id, you can add an Authorization header to the URL request.

 

For example, I have a dashboard that pulls data from Twitter.  Once I obtain the access token (it is the result of a different query named AccessToken) here is the function I use to query Twitter:

 

(params) =>
let
    
    GetJsonQuery = Web.Contents("https://api.twitter.com/1.1/search/tweets.json" & params,
        [
            Headers = [#"Authorization"=AccessToken]
        ]
     ),
    FormatAsJsonQuery = Json.Document(GetJsonQuery),

    data = try FormatAsJsonQuery[statuses] otherwise null,
    next = try FormatAsJsonQuery[search_metadata][next_results] otherwise null,

    res = [Data=data, Next=next]
in
    res

I have a table with a column of search words, where the column name is "keyword".  The parameter named "params" that started off the function above is defined as:

 

params = "?q=" & keyword & "&count=100"

So you can see how the column of hard coded search terms is passed into the url part defined by "params" which is in turn passed into the function.  The function runs on every row in my table of "keyword", so each row returns a table of results, which I can then expand and Power BI will automatically append the results into one table.

 

Note:  This is not the entire query and will not return results (the "data", "next" and "res" variables are used in another query that iterates through pages of results).  Hopefully this shows you how to structure your web call using the bearer token.

View solution in original post

35 REPLIES 35
Anonymous
Not applicable

I did create a custom function, but that takes appIDs from Parameters.  How do I change it to take appIDs from another query? Changing aa_appid as table generated errors.  Thanks in advance.

 

let
Source = (aa_app_id as text) => let
Source = Json.Document(Web.Contents("https://api.appannie.com/v1.2/intelligence/apps/ios/app/" & aa_app_id & "/history?countries=US&feeds=downloads&device=all&start_date=2016-01-01&end_date=2017-02-28&granularity=monthly&device=iphone", [Headers=[Authorization="bearer <key>"]])),

 

My aa_app_id Query: 

 

let
Source = Excel.Workbook(File.Contents("<filename>.xlsx"), null, true),
Table1_Table = Source{[Item="Table1",Kind="Table"]}[Data],
#"Changed Type" = Table.TransformColumnTypes(Table1_Table,{{"aa_app_id", type text}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "URL request", each "https://api.appannie.com/v1.2/intelligence/apps/ios/app/"&[aa_app_id]&"/history?countries=US&feeds=d... 28&granularity=monthly&device=iphone"),
#"Changed Type1" = Table.TransformColumnTypes(#"Added Custom",{{"URL request", type text}})
in
#"Changed Type1"

When you add a custom column (Invoke Custom Function), change the drop down to Column Name and select the app_id column.  Then, the function will pass in the app_id value from each row into the function (into the URL) and perform the query steps you designed earlier.  If you need to pass a bearer token to the URL for each app_id value, if it is the same bearer token for every app_id, you can add an Authorization header to the URL request.

 

For example, I have a dashboard that pulls data from Twitter.  Once I obtain the access token (it is the result of a different query named AccessToken) here is the function I use to query Twitter:

 

(params) =>
let
    
    GetJsonQuery = Web.Contents("https://api.twitter.com/1.1/search/tweets.json" & params,
        [
            Headers = [#"Authorization"=AccessToken]
        ]
     ),
    FormatAsJsonQuery = Json.Document(GetJsonQuery),

    data = try FormatAsJsonQuery[statuses] otherwise null,
    next = try FormatAsJsonQuery[search_metadata][next_results] otherwise null,

    res = [Data=data, Next=next]
in
    res

I have a table with a column of search words, where the column name is "keyword".  The parameter named "params" that started off the function above is defined as:

 

params = "?q=" & keyword & "&count=100"

So you can see how the column of hard coded search terms is passed into the url part defined by "params" which is in turn passed into the function.  The function runs on every row in my table of "keyword", so each row returns a table of results, which I can then expand and Power BI will automatically append the results into one table.

 

Note:  This is not the entire query and will not return results (the "data", "next" and "res" variables are used in another query that iterates through pages of results).  Hopefully this shows you how to structure your web call using the bearer token.

Anonymous
Not applicable

@dkay84_PowerBI I got it to work.  I was confused about Invoking custom function at first.  It turns out I need to Invoking Custom Function within the app_id query that have all the appIDs, not invoking within the custom function itself.  

 

Thank you all for the tips and guidance regarding this issue.  I've learned a lot!!

Sorry I wasn't able to better explain it earlier!  Glad you got it to work.

Anonymous
Not applicable

I did create a custom function, but that takes appIDs from Parameters.  How do I change it to take appIDs from another query? Changing aa_appid as table didn't create errors.  Thanks in advance.

 

let
Source = (aa_app_id as text) => let
Source = Json.Document(Web.Contents("https://api.appannie.com/v1.2/intelligence/apps/ios/app/" & aa_app_id & "/history?countries=US&feeds=downloads&device=all&start_date=2016-01-01&end_date=2017-02-28&granularity=monthly&device=iphone", [Headers=[Authorization="bearer <key>"]])),

 

My aa_app_id Query: 

 

let
Source = Excel.Workbook(File.Contents("C:\Users\310234735\OneDrive - Philips Lighting\AppResearch\Appannie\AA_app_ids.xlsx"), null, true),
Table1_Table = Source{[Item="Table1",Kind="Table"]}[Data],
#"Changed Type" = Table.TransformColumnTypes(Table1_Table,{{"aa_app_id", type text}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "URL request", each "https://api.appannie.com/v1.2/intelligence/apps/ios/app/"&[aa_app_id]&"/history?countries=US&feeds=d..."),
#"Changed Type1" = Table.TransformColumnTypes(#"Added Custom",{{"URL request", type text}})
in
#"Changed Type1"

Anonymous
Not applicable

I did create a custom function, but that takes appIDs from Parameters.  How do I change it to take appIDs from another query? Changing aa_appid as table didn't create errors.  Thanks in advance.

 

let
Source = (aa_app_id as text) => let
Source = Json.Document(Web.Contents("https://api.appannie.com/v1.2/intelligence/apps/ios/app/" & aa_app_id & "/history?countries=US&feeds=downloads&device=all&start_date=2016-01-01&end_date=2017-02-28&granularity=monthly&device=iphone", [Headers=[Authorization="bearer <key>"]])),

 

My aa_app_id Query

 

let
Source = Excel.Workbook(File.Contents("C:\Users\310234735\OneDrive - Philips Lighting\AppResearch\Appannie\AA_app_ids.xlsx"), null, true),
Table1_Table = Source{[Item="Table1",Kind="Table"]}[Data],
#"Changed Type" = Table.TransformColumnTypes(Table1_Table,{{"aa_app_id", type text}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "URL request", each "https://api.appannie.com/v1.2/intelligence/apps/ios/app/"&[aa_app_id]&"/history?countries=US&feeds=d..."),
#"Changed Type1" = Table.TransformColumnTypes(#"Added Custom",{{"URL request", type text}})
in
#"Changed Type1"

Anonymous
Not applicable

I did create a custom function where appID is defined as Text and it's coming from Parameters, which I don't want.  How do I get it to accept appIDs from another query( aa_app_id query)?  When I changed the 'aa_app_id as table', a dropdown list allowing another query to be chosen.   Does this mean that aa_app_id is searched and processed if it's found in another query? But error after invoking:

 

An error occurred in the ‘new_app_id’ query. Expression.Error: We cannot apply operator & to types Text and Table.
Details:
Operator=&amp;
Left=https://api.appannie.com/v1.2/intelligence/apps/ios/app/
Right=Table

 

My aa_app_id query:

 

let
Source = Excel.Workbook(File.Contents("<filename>.xlsx"), null, true),
Table1_Table = Source{[Item="Table1",Kind="Table"]}[Data],
#"Changed Type" = Table.TransformColumnTypes(Table1_Table,{{"aa_app_id", type text}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "URL request", each "https://api.appannie.com/v1.2/intelligence/apps/ios/app/"&[aa_app_id]&"/history?countries=US&feeds=d..."),
#"Changed Type1" = Table.TransformColumnTypes(#"Added Custom",{{"URL request", type text}})
in
#"Changed Type1"

 

Thanks in advance.

So in an earlier JSON call I get my bearer token.  I then concatenate "Bearer " with the returned token from that earlier REST call and use that concatenated string as my bearer token.  Here is the call using the bearer token...

 

let
    BearerToken = "Bearer "&{AuthBearerToken}{0}{0},

    Source = Json.Document(Web.Contents(SomeURLorURLVariable, [Headers=[Authorization=BearerToken, ContentType="application/json"]])),

Also for your function query remove the quotes from app_id

I believe your problem is that the column of app_id from your excel file is being stored as an integer data type. You need to convert this to string so it can be concantenated with the other url parts

Quick question.  Do you need to get the access token from the web call or do you already have it?

Anonymous
Not applicable

I already have the key and key is part of the URL request.  Thanks.

Okay, the reason I asked is because the answer to your question would change if you need to dynamically obtain the token (i.e. OAuth2).

 

Second question is, do you know how many url's you need to loop through or does it need to be dynamic?

Anonymous
Not applicable

The number of URLs is dynamic, most likely an increasing number.  Thanks.

Helpful resources

Announcements
Microsoft Fabric Learn Together

Microsoft Fabric Learn Together

Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City

PBI_APRIL_CAROUSEL1

Power BI Monthly Update - April 2024

Check out the April 2024 Power BI update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.