client has run out of available brokers to talk

[yitian108]

When I try disconnect only one node(192.168.2.147) in 3 nodes kafka cluster, and the producer app running on 192.168.2.146, the sarama async client reports following log:
New Sync Producer error: kafka: client has run out of available brokers to talk to: dial tcp 192.168.2.147:9092: i/o timeout

Expect:
If only one node disconnected, the cluster should be ok, rather than exit the producer app

My cluster info:
192.168.2.146:9092,192.168.2.147:9092,192.168.2.148:9092

The versions info:

component	version
kafka	3.6.1
sarama	1.42.2
zk	3.8.3

My Configuration of Producer:

cfg:= sarama.NewConfig()
cfg.Producer.Return.Successes = true
cfg.Producer.Return.Errors = true
cfg.Producer.RequiredAcks = sarama.WaitForLocal

cfg.Producer.Timeout = 3 * time.Second
cfg.Producer.Retry.Max = 0
cfg.Metadata.Timeout = 3 * time.Second
//cfg.Metadata.Retry.Max = 0
cfg.Net.DialTimeout = 3 * time.Second
cfg.Net.ReadTimeout = 3 * time.Second

The sarama log:

[sarama]2024/03/21 15:45:05 Initializing new client
[sarama]2024/03/21 15:45:05 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2024/03/21 15:45:05 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2024/03/21 15:45:05 client/metadata fetching metadata for all topics from broker 192.168.2.147:9092
[sarama]2024/03/21 15:45:07 Failed to connect to broker 192.168.2.147:9092: dial tcp 192.168.2.147:9092: connect: no route to host
[sarama]2024/03/21 15:45:07 client/metadata got error from broker -1 while fetching metadata: dial tcp 192.168.2.147:9092: connect: no route to host
[sarama]2024/03/21 15:45:07 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2024/03/21 15:45:07 client/metadata fetching metadata for all topics from broker 192.168.2.146:9092
[sarama]2024/03/21 15:45:07 Connected to broker at 192.168.2.146:9092 (unregistered)
[sarama]2024/03/21 15:45:07 client/brokers registered new broker #3 at 192.168.2.148:9092
[sarama]2024/03/21 15:45:07 client/brokers registered new broker #1 at 192.168.2.146:9092
[sarama]2024/03/21 15:45:07 client/metadata found some partitions to be leaderless
[sarama]2024/03/21 15:45:07 client/metadata retrying after 250ms... (2 attempts remaining)
[sarama]2024/03/21 15:45:07 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2024/03/21 15:45:07 client/metadata fetching metadata for all topics from broker 192.168.2.146:9092
[sarama]2024/03/21 15:45:07 Connected to broker at 192.168.2.146:9092 (registered as #1)
[sarama]2024/03/21 15:45:07 client/metadata found some partitions to be leaderless
[sarama]2024/03/21 15:45:07 client/metadata retrying after 250ms... (1 attempts remaining)
[sarama]2024/03/21 15:45:07 client/metadata fetching metadata for all topics from broker 192.168.2.146:9092
[sarama]2024/03/21 15:45:07 client/metadata found some partitions to be leaderless
[sarama]2024/03/21 15:45:07 client/metadata retrying after 250ms... (0 attempts remaining)
[sarama]2024/03/21 15:45:07 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2024/03/21 15:45:07 client/metadata fetching metadata for all topics from broker 192.168.2.148:9092
[sarama]2024/03/21 15:45:07 Connected to broker at 192.168.2.148:9092 (registered as #3)
[sarama]2024/03/21 15:45:07 client/metadata found some partitions to be leaderless
[sarama]2024/03/21 15:45:07 Successfully initialized new client
[sarama]2024/03/21 15:45:07 Initializing new client
[sarama]2024/03/21 15:45:07 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2024/03/21 15:45:07 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2024/03/21 15:45:07 client/metadata fetching metadata for all topics from broker 192.168.2.147:9092
[sarama]2024/03/21 15:45:10 Failed to connect to broker 192.168.2.147:9092: dial tcp 192.168.2.147:9092: i/o timeout
[sarama]2024/03/21 15:45:10 client/metadata got error from broker -1 while fetching metadata: dial tcp 192.168.2.147:9092: i/o timeout
[sarama]2024/03/21 15:45:10 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2024/03/21 15:45:10 client/metadata not fetching metadata from broker 192.168.2.148:9092 as we would go past the metadata timeout
[sarama]2024/03/21 15:45:10 client/metadata skipping last retries as we would go past the metadata timeout
[sarama]2024/03/21 15:45:10 Closing Client

I am not sure if this is related to issue #2121, as I mentioned(comments) previously. Do I need to adjust the producer configuration?

Thanks