Windows daemon container fails to terminate

[jrbe228]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

Continuing from the discussion #13146.

The working sequence with Linux daemons:

• Init container completes.
• Wait and main (server) containers start.
• Wait and main (server) containers terminate when the parent Workflow's client container has completed pinging the main (server) container.

The non-working sequence with Windows daemons:

• Init container completes.
• Wait and main (server) containers start.
• Wait and main (server) containers remain stuck in Running:
  

Also in some cases, deleting the workflow from Argo fails to remove the daemon's pod. The pod can only be removed by kubectl -n argo delete pod <pod-name> --grace-period=0 --force

Version

3.5.7

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: sidecar-nginx-
spec:
  entrypoint: sidecar-nginx-example
  templates:
  - name: sidecar-nginx-example
    steps:
      - - name: start-nginx
          template: nginx-daemon
      - - name: curl-nginx
          template: curl-container
          arguments:
            parameters:
              - name: nginx-ip
                value: "{{steps.start-nginx.ip}}"

#  - name: nginx-daemon
#    daemon: true
#    retryStrategy:
#      limit: 3
#    container:
#      name: nginx
#      image: nginx:1.13
#      command: [nginx, -g, "daemon off;"]
#    nodeSelector:
#      kubernetes.io/os: linux

  - name: nginx-daemon
    daemon: true
    retryStrategy:
      limit: 3
    container:
      name: nginx
      image: claudiubelu/nginx:1.15-1-windows-amd64-ltsc2022
      #image: claudiubelu/nginx:1.15-1-windows-amd64-1809
      #command: ["nginx", "-g", "daemon off;"]
    nodeSelector:
      kubernetes.io/os: windows

  - name: curl-container
    inputs:
      parameters:
        - name: nginx-ip
    container:
      image: appropriate/curl
      command: [sh, -c]
      args: ["until `curl -G 'http://{{inputs.parameters.nginx-ip}}/' >& /tmp/out`; do echo sleep && sleep 1; done && cat /tmp/out"]
    nodeSelector:
      kubernetes.io/os: linux

Logs from the workflow controller

#13146 (reply in thread)

Logs from in your workflow's wait container

#13146 (reply in thread)