webui: CORS issue on Peers page breaks geoip

[lidel]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

built from source

Version

0.21.0-rc3

Config

default after `ipfs init`

Description

We seem to have regression since Kubo 0.20.0, webui is unable to fetch blocks from gateway port due to CORS:

Occuring during subdomain redirect (localhost):

For TLDR of potential cause see #9983 (comment)

cc #9814, as we don't want to introduce this regression in 0.21

[lidel]

This can be reproduced on empty repo and empty browser cache – confirmed both Firefox and Brave (Chromium), issue was not present in 0.20.0, but appeared in 0.21.0-rc1 and is still present in rc3.
(Peers screen is caching results, so the bug was not obvious if you had geoip results mostly cached).

Firefox provides bit more detail:

[lidel]

Ok, there is the change in subdomain redirect AND trustless gateway behavior we did, and it seems to impact the use case when raw blocks are fetched from the localhost subdomain gateway:

0.20.0 did no redirect when block or car was requested

$ curl 'http://localhost:8080/ipfs/bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy?format=raw' -H 'Accept: application/vnd.ipld.raw' -H 'Origin: http://127.0.0.1:5001' -v
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /ipfs/bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy?format=raw HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.0.1
> Accept: application/vnd.ipld.raw
> Origin: http://127.0.0.1:5001
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Access-Control-Allow-Headers: Content-Type
< Access-Control-Allow-Headers: Range
< Access-Control-Allow-Headers: User-Agent
< Access-Control-Allow-Headers: X-Requested-With
< Access-Control-Allow-Methods: GET
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Content-Length
< Access-Control-Expose-Headers: Content-Range
< Access-Control-Expose-Headers: X-Chunked-Output
< Access-Control-Expose-Headers: X-Ipfs-Path
< Access-Control-Expose-Headers: X-Ipfs-Roots
< Access-Control-Expose-Headers: X-Stream-Output
< Cache-Control: public, max-age=29030400, immutable
< Content-Disposition: attachment; filename="bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy.bin"; filename*=UTF-8''bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy.bin
< Content-Length: 332
< Content-Type: application/vnd.ipld.raw
< Etag: "bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy.raw"
< Location: http://bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy.ipfs.localhost:8080/?format=raw
< X-Content-Type-Options: nosniff
< X-Ipfs-Path: /ipfs/bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy
< X-Ipfs-Roots: bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy
< Date: Tue, 20 Jun 2023 16:47:54 GMT
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* Closing connection 0

0.21.0-rc3 always redirects to subdomain first

> curl 'http://localhost:8080/ipfs/bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy?format=raw' -H 'Accept: application/vnd.ipld.raw' -H 'Origin: http://127.0.0.1:5001' -v
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /ipfs/bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy?format=raw HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.0.1
> Accept: application/vnd.ipld.raw
> Origin: http://127.0.0.1:5001
>
< HTTP/1.1 301 Moved Permanently
< Content-Type: text/html; charset=utf-8
< Location: http://bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy.ipfs.localhost:8080/?format=raw
< Date: Tue, 20 Jun 2023 16:49:39 GMT
< Content-Length: 133
<
<a href="http://bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy.ipfs.localhost:8080/?format=raw">Moved Permanently</a>.

* Connection #0 to host localhost left intact

---

I think the new behavior is valid, we normalize to subdomain, and execute bussiness logic only there.
TBD how to solve CORS problem.

[lidel]

When CORS preflight request is triggered by cross-origin request from webui at 127.0.0.1:5001 to path on localhost:8080, the response is HTTP 301 without Access-Control- headers, and at this point browser blocks regular GET requests.

$ curl -X OPTIONS 'http://localhost:8080/ipfs/bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy?format=raw' -H 'Accept: application/vnd.ipld.raw' -H 'Origin: http://127.0.0.1:5001' -v
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> OPTIONS /ipfs/bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy?format=raw HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.0.1
> Accept: application/vnd.ipld.raw
> Origin: http://127.0.0.1:5001
>
< HTTP/1.1 301 Moved Permanently
< Location: http://bafyreihnpl7ami7esahkfdnemm6idx4r2n6u3apmtcrxlqwuapgjsciihy.ipfs.localhost:8080/?format=raw
< Date: Tue, 20 Jun 2023 22:30:30 GMT
< Content-Length: 0
<

The fix here is to:

• ensure boxo/gateway applies user-provided headers (Kubo passes Access-Control-) headers on 301 returned in NewHostnameHandler – fix(gateway): include CORS on subdomain redirects boxo#395
• have regression tests in Kubo – fix(gateway): include CORS on subdomain redirects #9994
• fill issue in ipfs/specs to create spec for CORS behavior on gateways (lean towards liberal CORS, unless DNSLink) gateway: add CORS to specs specs#423

[BigLep]

2023-06-22 maintainer conversation:
Historically we haven't covered CORS in the specs. Ideally Helia should be able to use any public gateway to fetch blocks. Ideally we should get this into specs, and should be its own spec.
Gateways that provide deserialized data should have relaxed access to stay public.

For now going to do the fix in Boxo/Kubo (for Kubo 0.21) to not introduce regressions.
Then have a backlog item for spec work.