bentoml cli cannot connect to secure gRPC

[jiyer2016]

The reason I open this as a discussion is that I don't think this is a bug in bento. It might have something to with my environment. Wanted to check if others have encountered similar issues.

I am running the yatai-service (gRPC) behind an F5 load balancer.
When I run a simple bentoml list connecting to yatai, I get the folowing error
Handshake failed with fatal error SSL_ERROR_SSL: error:10000410:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE.

I then followed instructions given under #650 and set
bentoml config set yatai_service.client_certificate_file=<path_to_your_ca_cert.pem>

I also set

export GRPC_TRACE="transport_security,tsi"
export GRPC_VERBOSITY="DEBUG"

I now get a different error:
Cannot check peer: missing selected ALPN property.

It seems like this is related to an open bug in gRPC:
grpc/grpc#23172

The final recommendation from that post is to downgrade to gRPCio 1.19.0
When I do that - It seems to pass the SSL Handshake - but now I get a new error:

HANDSHAKE DONE -                TLS client done
grpc_message":"Trying to connect an http1.x server","grpc_status":14 

What does this final error mean ? It seems like it is well past all the TLS errors.
Does gRPC not like to connect to http1.x ?

Googling up this error throws up the following links:
https://putridparrot.com/blog/category/grpc/
https://newreleases.io/project/9yqbwg6v2jh2zsychkywjykhvg/release/@grpc%2Fgrpc-js@1.0.5

But I don't understand them too well.
I am wondering if my environment requires the following to be set ?
new ChannelOption("grpc.enable_http_proxy", 0)

I just don't know enough about gRPC to interpret the error.

[parano]

Hi @jiyer2016, have you tried it with the latest version of grpcio PyPI package recently? Is this still a problem for you?

[withsmilo]

@jiyer2016
You need to connect to the gRPC server of YataiService, not web UI server. Please check that you are not confused with this part. I also had a similar experience.