Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CONNECT-UDP in forwarding proxy mode resets stream and fails to send HTTP Datagrams #34836

Open
jeongseokson opened this issue Jun 21, 2024 · 10 comments
Open

CONNECT-UDP in forwarding proxy mode resets stream and fails to send HTTP Datagrams #34836

jeongseokson opened this issue Jun 21, 2024 · 10 comments
Assignees
@jeongseokson
Labels
area/udp bug no stalebot Disables stalebot from closing an issue

Comments

@jeongseokson
Copy link
Contributor

Title: CONNECT-UDP in forwarding proxy mode resets stream and fails to send HTTP Datagrams

Description:
When Envoy is configured as a CONNECT-UDP forwarding proxy, where it forwards the requests to upstream and HTTP Datagrams without terminating HTTP, it fails to forward the HTTP Datagrams and resets the stream.

Repro steps:
Use the example config in configs/proxy_connect_udp_http3_downstream.yaml to run a CONNECT-UDP forwarding proxy, and send CONNECT-UDP request and subsequent HTTP Datagrams to it.

Admin and Stats Output:
See #33775

Config:
configs/proxy_connect_udp_http3_downstream.yaml

Logs:
See #33775

Call Stack:
See #33775
@jeongseokson jeongseokson added bug triage Issue requires triage labels Jun 21, 2024
@jeongseokson
Copy link
Contributor Author

/assign @jeongseokson

@tyxia tyxia added area/udp and removed triage Issue requires triage labels Jun 21, 2024
@github-actions GitHub Actions
Copy link

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale stalebot believes this issue/PR has not been touched recently label Jul 21, 2024
@github-actions GitHub Actions
Copy link

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 28, 2024
@jeongseokson
Copy link
Contributor Author

@RyanTheOptimist Hi Ryan, can you reopen it and mark it as no stale? I am currently working on this issue.

@jeongseokson jeongseokson mentioned this issue Aug 26, 2024
Closed
@DavidSchinazi
Copy link
Contributor

/nostalebot

@repokitteh-read-only repokitteh-read-only bot added the no stalebot Disables stalebot from closing an issue label Aug 26, 2024
@DavidSchinazi
Copy link
Contributor

@RyanTheOptimist while you're reopening this one, please also close #33775 - thanks!

@RyanTheOptimist RyanTheOptimist removed the stale stalebot believes this issue/PR has not been touched recently label Aug 26, 2024
@RyanTheOptimist
Copy link
Contributor

@RyanTheOptimist while you're reopening this one, please also close #33775 - thanks!

Done!

@jeongseokson jeongseokson mentioned this issue Sep 17, 2024
Merged
RyanTheOptimist pushed a commit that referenced this issue Sep 23, 2024
@jeongseokson
http: fix issues in CONNECT-UDP forwarding mode. (#36174)
0e7fdf5 
Commit Message:
When Envoy operates as a CONNECT-UDP forwarding proxy, it was resetting
the upstream stream because it received HTTP Datagrams before receiving
the SETTINGS frame. A new enum has been added in QUICHE to distinguish
this case, so I added handling logic for this and made Envoy drop the
datagrams instead of resetting the stream.

Also, Envoy was dropping Datagrams because the default maximum packet
length for QUIC connections in QUICHE is not large enough for tunneling
use cases such as CONNECT-UDP. I added a new QUIC protocol option called
`max_packet_length` to allow users to adjust the maximum packet length
for upstream QUIC connections to fix this issue.

Additional Description:
Risk Level: Low, this change is only relevant if CONNECT-UDP is enabled
with the forwarding mode.
Testing: Added more unit tests. 
Docs Changes: Added the `max_packet_length` QUIC protocol option and its
explanation.
Release Notes: Added notes about fixing the CONNECT-UDP forwarding mode
and adding the new QUIC protocol option.
Platform Specific Features: N/A
[Optional Fixes #Issue]: #34836

---------

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>
@jeongseokson
Copy link
Contributor Author

Thanks for your patience, @Bfarkiani. We identified the issue thanks to your report. The fix has been merged into the main branch, so could you please try the forwarding mode again on your end after fetching the main branch? I was able to successfully make the two Envoy proxies (one for forwarding and the other for terminating) receive an HTTP/3 response from google.com using masque_client locally. Please let me know if you run into any problems.

@Bfarkiani
Copy link

Dear @jeongseokson . Thank you for your help. I tested with the following configs:

Client

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: UDP
        address: 0.0.0.0
        port_value: 10000
    udp_listener_config:
      quic_options: {}
      downstream_socket_config:
        prefer_gro: true
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.quic
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport
          downstream_tls_context:
            common_tls_context:
              tls_certificates:
              - certificate_chain:
                  filename: /etc/envoy/example.com.crt
                private_key:
                  filename: /etc/envoy/example.com.key
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP3
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog                       
          
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
              - "*"
              routes:
              - match:
                  connect_matcher:
                    {}
                route:
                  cluster: cluster_0
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          http3_protocol_options:
            allow_extended_connect: true
          upgrade_configs:
          - upgrade_type: CONNECT-UDP
  clusters:
  - name: cluster_0
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http3_protocol_options:
            allow_extended_connect: true
            quic_protocol_options:
              # Increase the max packet length for tunneling.
              # The example value matches the default value used by the MASQUE client in QUICHE.
              max_packet_length: 1350
    load_assignment:
      cluster_name: cluster_0
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 192.168.170.129
                port_value: 10000
    transport_socket:
      name: envoy.transport_sockets.quic
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicUpstreamTransport
        upstream_tls_context:
          # SAN of the certs/servercert.pem used for local testing. Update this if using a different certificate.
          sni: example.com

Server:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: UDP
        address: 0.0.0.0
        port_value: 10000
    udp_listener_config:
      quic_options: {}
      downstream_socket_config:
        prefer_gro: true
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.quic
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport
          downstream_tls_context:
            common_tls_context:
              tls_certificates:
              - certificate_chain:
                  filename: /etc/envoy/example.com.crt
                private_key:
                  filename: /etc/envoy/example.com.key
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP3
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog                       
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
              - "*"
              routes:
              - match:
                  connect_matcher:
                    {}
                route:
                  cluster: service_google
                  upgrade_configs:
                  - upgrade_type: CONNECT-UDP
                    connect_config:
                      {}
          http_filters:
          - name: envoy.filters.http.dynamic_forward_proxy
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
              dns_cache_config:
                name: dynamic_forward_proxy_cache_config
                dns_lookup_family: V4_ONLY          
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          http3_protocol_options:
            allow_extended_connect: true
          upgrade_configs:
          - upgrade_type: CONNECT-UDP
  clusters:
  - name: service_google
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: service_google
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: www.google.com
                port_value: 443

Command:
./masque_client --disable_certificate_verification --allow_unknown_root_cert 127.0.0.1:10000 https://www.google.com

Client proxy printed:

[2024-09-26 01:28:26.706][16][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_server_handshaker.cc:976] No hostname indicated in SNI
[2024-09-26 01:28:26.723][16][warning][quic_stream] [source/common/quic/http_datagram_handler.cc:62] SendHttpH3Datagram failed: status = MESSAGE_STATUS_SETTINGS_NOT_RECEIVED, drops the Datagram.
[2024-09-26T01:28:26.718Z] "GET /.well-known/masque/udp/www.google.com/443/ HTTP/3" 101 OM 5439 72019 521 - "-" "-" "700cf726-5f78-4fb9-85fe-e3e2c12f89b5" "www.google.com:443" "192.168.170.129:10000"

Server proxy printed:

[2024-09-26T01:28:26.722Z] "GET /.well-known/masque/udp/www.google.com/443/ HTTP/3" 200 - 4128 72019 516 - "-" "-" "700cf726-5f78-4fb9-85fe-e3e2c12f89b5" "www.google.com:443" "142.250.191.132:443"

Command returned with no error and no data. I think it is the expected behavior?

Thank you.

@jeongseokson
Copy link
Contributor Author

The access log shows that it returned a 200 status code, so it seems to be working fine. You can pass the --stderrthreshold=0 flag to see the response printed out. Here's the command I used:

$ ./masque_client --stderrthreshold=0 --disable_certificate_verification 127.0.0.1:10000 https://www.google.com/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jeongseokson jeongseokson

Labels
area/udp bug no stalebot Disables stalebot from closing an issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jeongseokson @DavidSchinazi @RyanTheOptimist @tyxia @Bfarkiani