Getting REST response headers

[AdamFry]

Hi there!  I have an API that I would like to use in a data pipeline but this API requires me to call a generic URL with the request that I'd like to make and in the response headers, it provides me with a redirect URL.  In postman, it looks like this: 

Since the censored part of the above image is the only thing that changes, you can't really see the difference but I do a GET to the URL at the top and in the response headers, I get a "Location" property back that gives me the URL that I should use to make my request.  

I am able to successfully make this call via PostMan and then take the redirect URL that's provided in the response header and use it in a fabric data pipeline copy activity and make the REST call to get the data for that request.  However, I'd like to perform this call in my pipeline and pass the redirect URL to the next step in the process so that I can ensure that I'm always hitting the right URL if the vendor changes the redirect URL.  But I am unable to see any way to get a handle on the response header. 

If there's good documentation on how to do this, I'd be so grateful for anyone who could point me in the right direction.  If there's another, better approach to handle this situation, I'd love to hear about that as well.  Thanks in advance!

[Kaviraj11]

This will probably help you with that - Azure Data Factory - Copy Activity - Source API Request save header response - Stack Overflow

[v-cboorla-msft]

Hi @AdamFry 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

[AdamFry]

Thank you for checking back on this.  I am trying to exercise the web activity to get the API reponse headers and I wonder if I'm overcomplicating this whole thing.  Based on what I'm seeing, I'm wondering if the fabric REST and/or WEB connection handles URL redirects automatically?  

The API I'm calling requires me to get a URL redirect by calling one URL and on that call, I get an HTTP 302 response with a "Location" field in the response header that provides me the URL that I should call to get my data.  Then I need to call another URL to get a token and pass that into the 3rd request at the redirected URL.  

When I try using the web connection and point to the first URL with anoynmous authentication, I get a 401 Unauthorized back saying the client request has not been completed because it lacks valid authentication credentials for the requested endpoint(url).  

However, when I make this same call via Postman with no authentication provided, I get the HTTP 302 and the redirect URL in the response headers.  

As a test, back in Fabric, I tried adding my token URL and API credentials to the web connection and then instead of getting the HTTP 302 and the redirect URL in the response headers, I got the content that I normally would get on the final call.  This makes me think that Fabric is automatically handling the redirect and I don't need to bother with the web call and parsing the redirect URL from the response and then making another request to get the data.  

Is there any documentation that describes this behavior?  I just want to confirm that the REST connection is actually automatically following the redirect and if that's the case, that's awesome and I can simplify my pipeline.

[v-cboorla-msft]

Hi @AdamFry 

Apologies for the issue you are facing.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.
Appreciate your patience.

Thanks.

[v-cboorla-msft]

Hi @AdamFry 

Apologies for the inconvenience.

Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.

[v-cboorla-msft]

Hi @AdamFry 

We haven’t heard from you on the last response and was just checking back to see if you've had a chance to submit a support ticket. If you have, a reference to the ticket number would be greatly appreciated. This will allow us to track the progress of your request and ensure you receive the most efficient support possible.

Thanks.

[AdamFry]

@v-cboorla-msft would it be possible to edit the title of this thread from "Getting REST response headers" to "Handling REST API with URL Redirection"?  That may help others in the future by giving the topic a more relevant title.  

[AdamFry]

Thank you, I have submitted a support ticket with the following support request id: 
2405100040007201

I posted on the fabric community forums and was directed to create a support request to see if I could get some help understanding the best practice for making a REST API call to a URL that uses URL redirection. Here's my original post: https://community.fabric.microsoft.com/t5/Data-Pipelines/Getting-REST-response-headers/m-p/3904046/highlight/true#M3171 To take a step back and provide some context for the question, I have a 3rd party system that has some of my company's data and I would like to bring that data into Fabric. The data is accessible via REST API. The API uses HTTP Redirects so I am to call one URL that will never change and that URL will provide me with an HTTP 302 response with a Location field in the response headers and that Location field contains the URL that I should call to make the request for my data. My question is whether this is automatically built into the REST connection functionality in Fabric or if I need to call the first URL, parse the redirect location from the response header, and then make the call to the redirect URL. When I tried calling the first URL to get the redirect from the Fabric pipeline, I would get a 401 unauthorized response saying the client request has not been completed because it lacks valid authentication credentials for the requested endpoint (url). When I make the same request via postman, I get an HTTP 302 and the redirect URL in the response headers. When I add the token URL and the API credentials and point to the 1st URL in fabric, instead of getting the 302 and redirect location, I actually got the data back that I had expected to get on the 2nd request. So I suspected that Fabric is automatically handling the URL redirection but I wanted to be sure and so I made the forum post to ask if there is any documentation to confirm this is the case.

[AdamFry · Accepted solution]

I hate to tag the response to my own question as the solution but I was able to get a response on my support ticket that in fact.  This is all I was trying to do with getting the REST response headers was to get a handle on the redirect URL but it turns out it's much simpler than that and I don't even need to bother with it:

... the Fabric REST connection functionality does handle URL redirects automatically. When you configure a REST connection in Fabric, it should follow the redirection and retrieve the data from the final URL.

[Kaviraj11]

Hi!

I have been in the similar situation but different scenarios. So, we stored the total count that comes in the header into the warehouse using Copy Data and use the 'LookUp' to query(SQL) the values from the table in warehouse. Next, we use 'Set Variable' to pass on the value into the Variables declared and further utilized the variable in the next step of the pipeline.

I hope it's helpful.

[AdamFry]

Hi Kaviraj, thank's for the suggestion!  I would be happy to do something similar, storing the header using copy data and then doing a lookup to read that value but what I'm not understanding is how did you get a handle of the REST response header to store it?  When I preview data or write the response to a file, I am just seeing the response body.