SocketIO callback received in the message queue, but the callback isn't executed

[Joe-ChenZ]

Hi,

I am currently having issues with the callback feature of the library. I am using nginx and AWS elastic beanstalk for the load balancing and deployment.

Here is the basic setup of the load balancer:

location / {
    proxy_pass          http://scale_nodes;
    proxy_http_version  1.1;

    proxy_set_header    Connection          $connection_upgrade;
    proxy_set_header    Upgrade             $http_upgrade;
    proxy_set_header    Host                $host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
}

location /socket.io {
    proxy_pass          http://socketio_nodes;
    proxy_http_version  1.1;

    proxy_set_header    Connection          $connection_upgrade;
    proxy_set_header    Upgrade             $http_upgrade;
    proxy_set_header    Host                $host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
}

    upstream scale_nodes {
       ip_hash;

       server 127.0.0.1:8000;
       server 127.0.0.1:8001;
       server 127.0.0.1:8002;
       # to scale the app, just add more nodes here! (Update Procfile)
    }

    upstream socketio_nodes {
       ip_hash;

       server 127.0.0.1:9000;
       server 127.0.0.1:9001;
       # to scale the app, just add more nodes here! (Update Procfile)
    }

This is the flask socketio setup:

mgr = RedisManager(url=get_transport_url(), channel="socketio_mgr")
 socketio = SocketIO(
        app,
        async_mode="gevent",
        cors_allowed_origins="*",
        manage_session=False,
        message_queue=get_transport_url(),
        client_manager=mgr,
    )

get_transport_url() is just a function that returns the redis host and port and it connects fine.

I am using gunicorn and gevent as the workers, and each gunicorn worker is created to bind to one port, with only 1 worker:

web: gunicorn --bind 127.0.0.1:8000 -k geventwebsocket.gunicorn.workers.GeventWebSocketWorker -w 1 application:app
And the other workers look similar to this

The issue:

The code works locally, and also on AWS when using only a single node. Socketio callback functions work in the pure socketio context, aka when the callback is emitted from a socketio context. However, with multiple nodes, it does not work when the server emits an event in the http context. The client responds fine, and both the engineIO logs and the pubsub caught the message, the namespace, sid, and callback id are all matched. There is also one single client in the emit. I have a feeling that it is because it is being handled by a different worker because the code works locally, and also works when there is only one worker. I wonder the more "appropriate way" of handling this is to have the client emit an event instead of using the callback

I am currently using the old flask socketio version of 2.0.1. The change logs don't seem to have updates for this. I probably might be wrong while skimming through the updates. Just want to know the best way of handling this if anyone has encountered similar problems.

[miguelgrinberg]

The issue is probably caused by using two load balancers instead of one, which seems like a strange thing to do. Do callbacks work if you drop the AWS balancer and use only nginx?

[Joe-ChenZ]

The original thought was to separate socketio streams from the http stream. It seems to work after I did a sudo systemctl restart nginx but not sudo nginx -s reload. But either way, thank you so much