fetch response content

[kristoffer-zliide]

Version

v20.15.0

Platform

Microsoft Windows NT 10.0.22631.0 x64

Subsystem

fetch

What steps will reproduce the bug?

fetch('https://www.facebook.com/.well-known/oauth/openid/jwks/')

How often does it reproduce? Is there a required condition?

💯

What is the expected behavior? Why is that the expected behavior?

I expect to get a response from fetch that contains a JSON body, since that's what Firefox's, Edge's, and node-fetch's fetch return. It's also what curl 8.x gets and what browsers show if you paste the URL in the address bar.

What do you see instead?

I response that contains text/html.

Additional information

I also tried fetch('https://www.facebook.com/.well-known/oauth/openid/jwks/', { headers: { accept: 'application/json' } }) (and accept: '*/*'), but it made no difference. I tried the fetch generated by a browser using "copy as fetch," and that worked. After a bit of binary search between the minimal fetch and the browser generated fetch, I found that

fetch('https://www.facebook.com/.well-known/oauth/openid/jwks/') responds with HTML
fetch('https://www.facebook.com/.well-known/oauth/openid/jwks/', { headers: { 'Sec-Fetch-Site': 'none' } }) responds with JSON

I've also seen (closed) APIs from Azure API Management that respond 500 to nodejs' fetch, but works as expected with all the above mentioned alternatives. The issue is also there in 20.15.0 from https://deb.nodesource.com/node_20.x in a debian:bookworm-20240513 container.

[RedYetiDev]

Certain websites have certain settings that make fetching them without certain information return a different response. Node.js doesn't control these sites, so your request must conform to their standards.

[RedYetiDev]

In your case, it appears that Facebook wants the following headers:

fetch('https://www.facebook.com/.well-known/oauth/openid/jwks/', {
  headers: {
    'sec-fetch-site': '<...>'
  }
});

Edit: I see you mentioned that in your issue, sorry to reiterate a known point 😄

[kristoffer-zliide]

Obviously. The point is that fetch without headers work fine in browsers and node-fetch (and curl, and...), so it's kind of surprising it doesn't work with nodejs' built-in fetch.

[kristoffer-zliide]

fetch in Deno v1.44.4 and Bun v1.1.17 also work fine without any extra headers.

[RedYetiDev]

If you'd like to like to suggest an improvement to the default fetch headers, please refer to https://github.com/nodejs/undici and https://github.com/whatwg/fetch