Bug: polling health checker will always fail

[humbornjo]

We run into some trouble with client conn control, and trying to use the health check. however, the default health checker will always fail with its context deadline exceeded. I think its because the line 332 in httplb/balancer.go, should use withtimeout instead of withcancel.

connection := conns[i]
connCtx, connCancel := context.WithCancel(b.ctx)
healthChecker := b.healthChecker.New(connCtx, connection, b)
go func() {
	defer connCancel()
	if err := connection.Prewarm(connCtx); err == nil {
		b.warmedUp(connection)
	}
}()

here is my test code

// server.go
package main

import (
	"github.com/gofiber/fiber/v2"
	"github.com/gofiber/fiber/v2/middleware/healthcheck"
	"github.com/gofiber/fiber/v2/middleware/recover"
)

func main() {
	app := fiber.New()

	app.Use(recover.New())
	app.Use(healthcheck.New(
		healthcheck.Config{
			LivenessEndpoint: "/healthz",
			LivenessProbe: func(c *fiber.Ctx) bool {
				c.Send([]byte("OK"))
				return true
			}},
	))
	app.Get("/foo", func(c *fiber.Ctx) error {
		return c.Send([]byte("bar"))
	})
	app.Get("/", func(c *fiber.Ctx) error {
		return c.Send([]byte("bar"))
	})
	if err := app.Listen(":3000"); err != nil {
		panic(err)
	}
}

// client.go
package main

import (
	"fmt"
	"net/http"
	"net/url"
	"time"

	"github.com/bufbuild/httplb"
	"github.com/bufbuild/httplb/health"
)

func main() {

	url := url.URL{
		Scheme: "http",
		Host:   "localhost:3000",
		Path:   "foo",
	}

	client := httplb.NewClient(
		httplb.WithHealthChecks(
			health.NewPollingChecker(
				health.PollingCheckerConfig{
					Timeout:         100 * time.Second,
					PollingInterval: 100 * time.Second,
				},
				health.NewSimpleProber("/healthz"),
			),
		),
	)

	for range 5 {
		req, err := http.NewRequest(http.MethodGet, url.String(), http.NoBody)
		if err != nil {
			fmt.Println(err)
			continue
		}

		resp, err := client.Do(req)
		if err != nil {
			fmt.Println(err)
			continue
		}
		fmt.Println(resp.Status)
		time.Sleep(1 * time.Second)
	}
}

the error log

200 OK
Get "http://localhost:3000/foo": unavailable: no healthy connections
Get "http://localhost:3000/foo": unavailable: no healthy connections
Get "http://localhost:3000/foo": unavailable: no healthy connections
Get "http://localhost:3000/foo": unavailable: no healthy connections

[humbornjo]

btw, I wrote a dummy prober with no use of ctx, and it works well.

[humbornjo]

Speaking of which, about the wired behavior health when the conn dialing timeout. it should report unhealthy, but in fact it still use the timeout conn. I am still trying to comprehend the code logic of this part, maybe discuss it in another thread.