Running Fabric's test-network without Docker Desktop

[mbwhite]

Background
A 'Fabric Network' is a distributed system running a number of nodes, of different types. When configured together they provide the cross-enterprise permissioned blockchain.

Fabric is built and supplied as a number of different binary images, along with convenience 'docker images', along with a several example ways these images can be configured and orchestrated to make a fully functional distributed system. The 3 examples in the fabric-samples repo are

• test-network-nano-bash : uses the fabric binaries directly in a CLI context, the docker daemon is used by the Fabric Peer nodes, to create containers for running chaincode
• test-network : a docker-compose based example of running a Fabric-Network
• test-network-k8s : a k8s example currently using KIND showing running a Fabric-Network inside k8s. (same Fabric-Network topology as the test-network)

**Some Techonolgy **

• Docker Engine : the 'engine' that actually runs the containers from the images
• Docker Client : cli to start/stop/managed the images and containers
• Docker Daemon : daemon process exposing API for the cli (and other tools) to use for managaement
• Docker Compose : a separate tool distinct from the Docker Client, defining a 'compose' file with definition of how to run a set of docker containers together
• Docker Desktop : UI Application that includes the above components, specifically for running on MacOS and Windows

Fabric's docker dependency
There are 2 ways Fabric is dependent on Docker itself, over-and-above the Fabric binaries being built into container images.

1. is because the Peer is able to communicate with the Docker-Daemon to create chaincode containers. External Builders and Chaincode-as-a-service are alternative configuration options that remove this requirement
2. The test-network uses both docker and docker-compose, and the test-network-k8s uses the docker engine to start the KIND k8s cluster

The distruptor
Docker Desktop is changing it's license model, so some developers/companies may no longer have access to this.
The core docker-engine/docker-daemon/docker-cli/docker-compose remain available under their original open-source licenses and usable, but the tool that bundles all these together and manages them for MacOS and Windows platforms is changing.

Impact
For a Linux based environment, either Linux running 'on-metal' or in a virtual environment such as WSL2/VirtualBox etc. where it uses as the primary means of development there is no impact.

Docker-Desktop effectively creates a LinuxVM on Windows and MacOS specifically for managing docker containers, and 'hides/smooths' access to these containers from the native CLIs of those platforms.

Without Docker-Desktop therefore, unless MacOS/Windows opt to the Linux VM approach, a Docker-compose-based approach to running a test-network is not available. Note this also affects other tools such as the IBM Blockchain VSCode Extension.

Solutions and Mitigations
If Docker-Desktop is not available then:

• Opt to use a LinuxVM approach to run the docker-engine and docker-compose
  • Multipass is a way to help run LinuxVMs on both MacOS and Windows, this makes it very easier to create machines as needed
  • WSL2 is probably the main solution on Windows. HyperV/VirtualBox also are solutions.
  • MacOS solutions are [please can somebody add precise details - thank you]
• Podman is an alternative to the 'docker' cli and 'docker-engine', Podman-compose is the alternative to docker-compose. On Windows this though still needs to be run within a LinuxVM; it removes the need for docker and docker-daemon it still requires the use of a LinuxVM. On MacOS there is a 'podman-engine' that is available to help
• nerdctl and nerdctl-compose are alternatives to the docker and docker-compose tools
• Rancher Desktop - a complete alternative to Docker-Desktop with also supporting local k8s clusters.

Actions

• For Linux/LinuxVM approaches docker remains a completely viable approach and should continue to be used

• For MacOS/Windows if Docker-Desktop is available to use, then similarly it remains a completely viable approach.

• Rancher-Desktop would be the direct replacement if the Docker-Desktop is not available

• Document the configuration of the peer to 'turn-off' the reliance on the docker-daemon, and how it is used otherwise

• Validate the podman/nerdctl approaches to running the test-network and identify any issues

• Article on how to use the Rancher-Desktop

[mbwhite]

Draft PR that shows the use of Podman and Podman-Compose #596

[davidkel]

I think at the end of the day we shouldn't just be discounting docker-desktop completely. It does offer a very good seamless solution for people to work with containers on Mac so seems reasonable that people would buy it to increase their productivity in the same way developers may buy other development tools. I've only ever seen one request about moving off of docker desktop to something else from the community and that was actually for the IBP vscode extension so this new docker subscription doesn't appear to be a concern for the community at large.
For windows, Microsoft recommend the use of docker-desktop with WSL2. We recommend docker-desktop for windows users as it makes it a simpler introduction to containers in general, but we could easily move that to docker-engine given that we say to use WSL2 for CLI actions but that would require more detailed instructions from us to describe that kind of setup but it still means that docker is the choice for managing and running containers.
For linux users there is no problem here, so I would like us to continue to offer a docker based test-network for fabric users

What about fabric contributors and maintainers who use a mac, as all of our test suites for various components such as the sdks and chaincodes use docker for their test suites and pipelines as well as fabric-test a lot more impact there I feel. Also we use docker as part of the build process to create images etc

[mbwhite]

Yes, docker & docker-desktop are still completely viable and if you have them available, probably the first choice.
Alternatives are available :-)

[davidkel]

So linux users are not a problem, and windows users should be using WSL2 and if docker-desktop is not available I find that docker engine in WSL2 is still as good, just the slight pain of having to modify your bash profile to start docker if it hasn't been started. Both fabric users and fabric contributors and maintainers will have no issues.
So really it just leaves Mac users with a lesser experience if they can't use docker-desktop where they don't have seamless integration of docker from the Mac terminal shells or their filesystem with the source code in

[andrew-coleman]

The scenario tests for fabric-gateway also rely on docker-compose:
https://github.com/hyperledger/fabric-gateway#scenario-tests

[jkneubuh]

On a mac working with an embedded linux VM is the way to go. Rancherdesktop provides a tight integration with the embedded VM and bundles the CLI-compatible nerdctl bridge to avoid hassles getting to/from the containerd running on the VM. Rancher will also optionally provide / manage a symlink from docker -> nerdctl, so in general most of the day-to-day interaction in pulling, building, running, etc. images is invisible if not transparent.

Rancher/nerdctl includes a compose equivalent, covering about 95% "exact match" compatibility with our use of docker-compose in the test network. This provides a route forward for compatibility with the test network, but it's not ideal.

I'll be loud and clear with (Josh) bias, but a good route forward is to build out the Kube test network, and de-emphasise the compose based test networks. This sets up for a good alignment with IBP / production deployments, highlights best practices for Gateway and Chaincode-as-a-Service contracts, and serves as a stable basis for building, debugging, and running code locally in a debugger. The last bit of advocacy here is that the k8s routes are virtually identical across docker, podman, and rancher. For instance I was able to switch between Docker (KIND) and Rancher/nerdctl (k3s) without any impact to the build/edit/test/tag/push/run/etc. pipeline.

[SamYuan1990]

What's the plan for CI? as in current PR(#596) the CI was stopped.

is there any solution that making code(fabric-sample) able to support docker/Rancher/podman?
as test-network-k8s, it supports kind and able to migrate from kind to other k8s(as local k8s)?

I suppose there is interoperability between k8s, kind and minikube?
I am not sure is there any interoperability between docker, rancher or podman?

if there is, no sure if we are able to make the script more flexible?, as docker or podman as alias, and in shell scripts running with the alias.

[jkneubuh]

• OSX + Rancher desktop + dockerd(moby) k8s + docker-compose + test-network => success. (Details in Issue Evaluate the compose test-network using Rancher Desktop and Moby / dockerd #616)

• @SamYuan1990 I think this does not impact the CI flows, which are based on Linux. The Linux VM running the CI test suite can continue to use docker-compose and docker via an installation of docker engine.

• test-network-k8s works well on both Docker+KIND, and better with Rancher+k3s using either moby or containerd.

• In general I like the introduction of ${CONTAINER_CLI} into the sample scripts, which allows a user to swap between podman/nerdctl/docker CLIs. The other option is to rely on the Rancher symbolic link, but this seems less portable overall.

[jkneubuh]

One additional comment before moving away from this issue / thread.

Despite my personal bias to use one specific runtime, I really appreciate and would like to amplify @davidkel 's perspective that we shouldn't just be discounting docker-desktop completely. As a general rule, Fabric should encourage the selection of platforms, runtimes, and tooling such that we are all encouraged and empowered individually to select the right tool, for the task at hand.

In some cases, the "right tool" is to run native binaries.
In other cases, the "right tool" is to run the test-network on Docker compose.
In other cases, the "right tool" is to run a fabric-test-operator network, pointing at a bare metal or AWS k8s.
...

In all cases, let's find routes that work together in an orchestrated fashion. There's no "one size fits all" solution for Fabric.

Cheers and happy coding.
-josh

[bestbeforetoday]

Revisiting Docker Desktop alternatives today on an M1 Mac, I tried this:

1. Installed Colima and docker CLI commands using Homebrew:

   brew install colima docker docker-credential-helper docker-compose

2. Edit ~/.docker/config.json to read as follows:

   {
       "auths": {},
       "credsStore": "osxkeychain",
       "currentContext": "colima"
   }

3. Started Colima:

   colima start

4. Made sure the Colima context docker.sock is available at default location, just in case anything needs it:

   sudo ln -s ~/.colima/default/docker.sock /var/run

5. Installed Fabric binaries and Docker images:

   curl --location https://raw.githubusercontent.com/hyperledger/fabric/main/scripts/install-fabric.sh | bash -s -- binary docker

6. Ran the asset-transfer-basic sample:

   cd test-network && \
       ./network.sh up createChannel -c mychannel -ca && \
       ./network.sh deployCC -ccn basic -ccp ../asset-transfer-basic/chaincode-go/ -ccl go
   cd asset-transfer-basic/application-gateway-go && \
       go run .

It seems to have worked OK. The network stands up, chaincode deploys and the client application runs successfully!

[bestbeforetoday]

Just for reference, I also tried Colima as above but, instead of using the default configuration (qemu and sshfs), configured the MacOS Virtualization.Framework (vz) and virtiofs. That seemed to work OK too.

[celder628]

I tested this out as well. Everything seems to work fine.

If you want to address anything outside of docker, you do need to add the "--network-address" option to "colima start"

colima start --network-address

Without this option, things like the following won't work:

curl -X GET http://127.0.0.1:5984/

[ArvinB]

@bestbeforetoday if you do a docker buildx ls do you get s390x listed using Colima. I don't and am wondering what I'm missing?

[bestbeforetoday]

I don't see s390x listed. When using qemu, docker buildx ls gives me these platforms:

linux/arm64, linux/amd64, linux/amd64/v2

With vz, I get:

linux/arm64

[ArvinB]

ditto 👎

~ % docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT        PLATFORMS
colima    docker                                  
  colima  colima          running v0.10.6+d52b2d5 linux/arm64, linux/amd64, linux/amd64/v2

Even after a qemu reset I get the above:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

My hope was to get a list like this, specifically for s390x

linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

Any ideas if this is a colima limitation?

[SamYuan1990]

maybe dockerx on github agent has s390x support?
for example https://github.com/Hyperledger-TWGC/tape/blob/master/.github/workflows/release.yml#L49
hence if we can easily enable s390x support by change GHA?

[ArvinB]

Is there a config locally I can change to enable this then?

[SamYuan1990]

@denyeart , shall we enable s390x for fabric? if so, I am going to create a PR and test it.