Failed database DNS lookup while running in docker

[lucacavazzana]

HI,
I'm running a Kong 3.7.1 instance using docker-compose. In a specific environment, after restarting the services, I started getting a DNS timeout during the startup when Kong resolves the database hostname.
(See the end of the post for a stripped-down version of the compose file I'm using to troubleshoot the problem. It works as expected on other environments, but reproduces the issue on that specific instance).

2024/12/13 14:19:18 [crit] 1336#0: *20 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
// truncated more similar lines like the above
2024/12/13 14:19:18 [crit] 1333#0: *7090 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2024/12/13 14:19:23 [crit] 1341#0: *16 [lua] upstreams.lua:98: could not obtain list of upstreams: [postgres] [cosocket] DNS resolution failed: failed to receive reply from UDP server 127.0.0.11:53: timeout. Tried: ["(short)kdb:(na) - cache-miss","kdb:33 - cache-miss/querying"], context: ngx.timer
2024/12/13 14:19:23 [crit] 1341#0: *16 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: [postgres] [cosocket] DNS resolution failed: failed to receive reply from UDP server 127.0.0.11:53: timeout. Tried: ["(short)kdb:(na) - cache-miss","kdb:33 - cache-miss/querying"], context: ngx.timer

Yet, running dig from inside the kong container (named gw), I get the correct answer:

$ docker exec gw dig kdb

; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> kdb
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62342
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;kdb.                           IN      A

;; ANSWER SECTION:
kdb.                    600     IN      A       172.26.0.2

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11) (UDP)
;; WHEN: Fri Dec 13 15:12:41 UTC 2024
;; MSG SIZE  rcvd: 40

The kong migrate bootstrap container fails similarly:

Error: 
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:101: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: [cosocket] DNS resolution failed: failed to receive reply from UDP server 127.0.0.11:53: timeout. Tried: ["(short)kdb:(na) - cache-miss","kdb:33 - cache-miss/querying"]
stack traceback:
        [C]: in function 'assert'
        /usr/local/share/lua/5.1/kong/cmd/migrations.lua:101: in function 'cmd_exec'
        /usr/local/share/lua/5.1/kong/cmd/init.lua:31: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:31>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/kong/cmd/init.lua:31: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:15>
        (command line -e):7: in function 'inline_gen'
        init_worker_by_lua(nginx.conf:126):44: in function <init_worker_by_lua(nginx.conf:126):43>
        [C]: in function 'xpcall'
        init_worker_by_lua(nginx.conf:126):52: in function <init_worker_by_lua(nginx.conf:126):50>

Removing the dsn_search: . override makes the bootstrap work. The same fix does not work for the main kong service, which still fails in the same way (except for using the inherited host's search: Tried: ["(short)kdb:(na) - cache-miss","kdb.########.org:33 - cache-miss/querying"])

Performing other tests I noticed that if I start Kong without running the boostrap container, (either with the dns_search: . override or without) it complains about the missing schema, suggesting that the service can resolve and connect the database in the early phases of the startup:

$ docker compose up gw
... // log truncated
2024/12/13 13:48:25 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:16: Database needs bootstrapping or is older than Kong 1.0.

To start a new installation from scratch, run 'kong migrations bootstrap'.

To migrate from a version older than 1.0, migrated to Kong 1.5.0 first. 
If you still have 'apis' entities, you can convert them to Routes and Services
using the 'kong migrations migrate-apis' command in Kong 1.5.0.

stack traceback:
  [C]: in function 'error'
  /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:16: in function 'check_state'
  /usr/local/share/lua/5.1/kong/init.lua:667: in function 'init'
  init_by_lua(nginx-kong.conf:42):3: in main chunk
gw exited with code 1

Also: overriding the dns_search: . causes the

Error: 
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:101: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: [cosocket] DNS resolution failed: failed to receive reply from UDP server 127.0.0.11:53: timeout. Tried: ["(short)kdb:(na) - cache-miss","kdb:33 - cache-miss/querying"]
stack traceback:
        [C]: in function 'assert'
        /usr/local/share/lua/5.1/kong/cmd/migrations.lua:101: in function 'cmd_exec'
        /usr/local/share/lua/5.1/kong/cmd/init.lua:31: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:31>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/kong/cmd/init.lua:31: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:15>
        (command line -e):7: in function 'inline_gen'
        init_worker_by_lua(nginx.conf:126):44: in function <init_worker_by_lua(nginx.conf:126):43>
        [C]: in function 'xpcall'
        init_worker_by_lua(nginx.conf:126):52: in function <init_worker_by_lua(nginx.conf:126):50>

Compose file

The following is the minimal configuration used to reproduce the problem:

name: test

x-kong-common:
  &kong-common
  image: kong:3.7.1-ubuntu
  environment:
    &kong-common-env
    KONG_PG_HOST: kdb
    KONG_PG_DATABASE: kong
    KONG_PG_USER: kong
    KONG_PG_PASSWORD: kong
    KONG_DATABASE: postgres

volumes:
  kdata:

networks:
  test:
    name: test
    driver: bridge

services:
  gw:
    <<: *kong-common
    container_name: gw
    restart: always
    command: kong docker-start -v
    healthcheck:
      test: ["CMD", "curl", "-f", "http://kong:8001"]
      interval: 30s
      timeout: 10s
      retries: 10
    dns_search:
      - "."
    dns_opt:
      - attempts:5
    networks:
      - test
    # depends_on:
    #   kdb: {condition: service_healthy}
    #   bootstrap: {condition: service_completed_successfully}

  bootstrap:
    <<: *kong-common
    container_name: bootstrap
    command: kong migrations -vv bootstrap
    restart: on-failure
    networks:
      - test
    dns_search:
      - "."
    # depends_on:
    #   kdb: {condition: service_healthy}

  kdb:
    image: postgres:14.12-alpine3.19
    container_name: kdb
    restart: always
    environment:
      POSTGRES_DB: kong
      POSTGRES_USER: kong
      POSTGRES_PASSWORD: kong
    healthcheck:
      test: ["CMD", "pg_isready", "--username", "kong", "--dbname", "kong"]
      interval: 30s
      timeout: 30s
      retries: 3
    volumes:
      - kdata:/var/lib/postgresql/data
    networks:
      - test

Logs

Here is the complete startup log:

2024/12/13 14:19:12 [verbose] Kong: 3.7.1
2024/12/13 14:19:12 [verbose] no config file found at /etc/kong/kong.conf
2024/12/13 14:19:12 [verbose] no config file found at /etc/kong.conf
2024/12/13 14:19:12 [verbose] no config file, skip loading
2024/12/13 14:19:12 [verbose] prefix in use: /usr/local/kong
2024/12/13 14:19:12 [verbose] preparing nginx prefix directory at /usr/local/kong
2024/12/13 14:19:12 [verbose] SSL enabled on proxy, no custom certificate set: using default certificates
2024/12/13 14:19:12 [verbose] proxy SSL certificate found at /usr/local/kong/ssl/kong-default.crt
2024/12/13 14:19:12 [verbose] proxy SSL certificate found at /usr/local/kong/ssl/kong-default-ecdsa.crt
2024/12/13 14:19:12 [verbose] SSL enabled on admin, no custom certificate set: using default certificates
2024/12/13 14:19:12 [verbose] admin SSL certificate found at /usr/local/kong/ssl/admin-kong-default.crt
2024/12/13 14:19:12 [verbose] admin SSL certificate found at /usr/local/kong/ssl/admin-kong-default-ecdsa.crt
2024/12/13 14:19:12 [verbose] SSL enabled on admin_gui, no custom certificate set: using default certificates
2024/12/13 14:19:12 [verbose] admin_gui SSL certificate found at /usr/local/kong/ssl/admin-gui-kong-default.crt
2024/12/13 14:19:12 [verbose] admin_gui SSL certificate found at /usr/local/kong/ssl/admin-gui-kong-default-ecdsa.crt
2024/12/13 14:19:12 [verbose] generating trusted certs combined file in /usr/local/kong/.ca_combined
2024/12/13 14:19:12 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:7
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:7
2024/12/13 14:19:12 [notice] 1#0: [lua] init.lua:788: init(): [request-debug] token for request debugging: 0272951f-aab8-4b92-a2f8-dde7436af585
2024/12/13 14:19:12 [notice] 1#0: using the "epoll" event method
2024/12/13 14:19:12 [notice] 1#0: openresty/1.25.3.1
2024/12/13 14:19:12 [notice] 1#0: OS: Linux 6.5.0-1025-oracle
2024/12/13 14:19:12 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/12/13 14:19:12 [notice] 1#0: start worker processes
2024/12/13 14:19:12 [notice] 1#0: start worker process 1333
2024/12/13 14:19:12 [notice] 1#0: start worker process 1334
2024/12/13 14:19:12 [notice] 1#0: start worker process 1335
2024/12/13 14:19:12 [notice] 1#0: start worker process 1336
2024/12/13 14:19:12 [notice] 1#0: start worker process 1337
2024/12/13 14:19:12 [notice] 1#0: start worker process 1338
2024/12/13 14:19:12 [notice] 1#0: start worker process 1339
2024/12/13 14:19:12 [notice] 1#0: start worker process 1340
2024/12/13 14:19:12 [notice] 1#0: start worker process 1341
2024/12/13 14:19:12 [notice] 1#0: start worker process 1342
2024/12/13 14:19:12 [notice] 1#0: start worker process 1343
2024/12/13 14:19:12 [notice] 1#0: start worker process 1344
2024/12/13 14:19:12 [notice] 1333#0: *1 [lua] warmup.lua:113: single_dao(): Preloading 'services' into the core_cache..., context: init_worker_by_lua*
2024/12/13 14:19:12 [notice] 1333#0: *1 [lua] warmup.lua:162: single_dao(): finished preloading 'services' into the core_cache (in 0ms), context: init_worker_by_lua*
2024/12/13 14:19:18 [crit] 1336#0: *20 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2024/12/13 14:19:18 [crit] 1334#0: *24 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2024/12/13 14:19:18 [crit] 1342#0: *28 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2024/12/13 14:19:18 [crit] 1339#0: *34 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2024/12/13 14:19:18 [crit] 1344#0: *506 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2024/12/13 14:19:18 [crit] 1343#0: *697 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2024/12/13 14:19:18 [crit] 1337#0: *2566 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2024/12/13 14:19:18 [crit] 1340#0: *4671 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2024/12/13 14:19:18 [crit] 1338#0: *5118 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2024/12/13 14:19:18 [crit] 1335#0: *6011 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2024/12/13 14:19:18 [crit] 1333#0: *7090 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
2024/12/13 14:19:23 [crit] 1341#0: *16 [lua] upstreams.lua:98: could not obtain list of upstreams: [postgres] [cosocket] DNS resolution failed: failed to receive reply from UDP server 127.0.0.11:53: timeout. Tried: ["(short)kdb:(na) - cache-miss","kdb:33 - cache-miss/querying"], context: ngx.timer
2024/12/13 14:19:23 [crit] 1341#0: *16 [lua] init.lua:248: init(): failed loading list of upstreams: failed to get from node cache: [postgres] [cosocket] DNS resolution failed: failed to receive reply from UDP server 127.0.0.11:53: timeout. Tried: ["(short)kdb:(na) - cache-miss","kdb:33 - cache-miss/querying"], context: ngx.timer

[Water-Melon]

Hi @bungle ,
I noticed that the code related to this error message is involved here. Do you have any thoughts or insights on this issue?