Arkouda Cloud Deployment Study
Arkouda on Kubernetes is deployed as two helm installs: arkouda-locale and arkouda-server for multi-locale deployment.
Deploying Arkouda within a Kubernetes/Docker-based environment ideally will feature the multi-locale Arkouda server deployed as a StatefulSet or ReplicaSet of 1..n pods and a pip install of the Arkouda client in a Jupyter notebook environment like Jupyterhub.
At a minimum, will need to have the arkouda_server process deployable via a Docker container orchestrated by Kubernetes.
The Arkouda server docker image extends the Chapel GASNet smp image for prototyping purposes.
An important detail within the Dockerfile is the following:
RUN export CHPL_GASNET_MORE_CFG_OPTIONS=--enable-force-posix-realtime
WORKDIR $CHPL_HOME
RUN touch third-party/gasnet/Makefile
RUN make -j
Without this set of entries a timing precision error was thrown by the Arkouda server upon deployment as a Docker container to Kubernetes.
The Arkouda server docker build command is as follows;
docker build -f arkouda-server-docker -t hokiegeek2/arkouda-server:0.0.1
The Arkouda server docker image for single-locale Arkouda-on-k8s is located here.
The single-host Arkouda Server is deployed as a Docker container on Kubernetes via a Helm Chart. The values.yaml file encapsulates cluster-specific config parameters:
######################## Pod Settings ########################
releaseVersion: 0.0.5
imagePullPolicy: IfNotPresent
numServers: 1
################ Arkouda Server Configuration ################
server:
host: 0.0.0.0
port: 5555
numLocales: 1
authenticate: false
verbose: false
memTrack: true
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 250m
memory: 256Mi
An example Helm install command from the Arkouda project root directory is as follows:
helm install -n arkouda arkouda-server arkouda-server-chart/
In this example, the Arkouda server is deployed to the arkouda namespace.
The Arkouda client can be deployed on Kubernetes as a Docker container as part of another deployment or via a pip install within a Jupyter notebook environment such as Jupyterhub.
The initial Arkouda Kubernetes deployment consisted of a single locale Arkouda server Docker deployment coupled with a pip install of the Arkouda client within a Dask Jupyterhub deployment.
While single-locale Arkouda-on-k8s provides a proof-of-concept of deploying Arkouda on Kubernetes, a key issue is that this represents a single locale Arkouda deployment and therefore does not leverage the horizontal scalability built into Arkouda via it's Chapel foundation. Consequently, an alternative Docker image is needed that enables multi-locale deployments. Ideally, a dockerized, multi-locale Arkouda would utilize the Kubernetes scheduler--either wholly or in part--so that the entire Arkouda client-server stack can be deployed on Kubernetes.
In order to deploy multi-locale Arkouda on Kubernetes, the following tech elements are required:
- Multi-locale, multi-host Docker--an image that enables launching of Arkouda across multiple locales deployed on multiple hosts.
- Non-smp COMM_SUBSTRATE--A GASNET comm substrate is needed that enable locale-locale comms across locales deployed on multiple hosts. At a minimum, the COMM_SUBSTRATE cannot be smp, which is the substrate where locales communicate across shared memory on one machine.
- Helm install of the arkouda_server master --a Helm chart for deploying the Arkouda docker with the arkouda_server master in multi-locale mode
In addition, it would be preferable to schedule multi-host, multi-locale Arkouda via Kubernetes for the arkouda_server-locale0 process as well as all remaining locales to leverage the Kubernetes metrics, monitoring, and lifecycle management capabilities for the entire Arkouda server runtime.
As detailed above, the initial Arkouda Kubernetes deployment features a docker image that extends the Chapel gasnet smp image. Since this image uses the smp COMM_SUBSTRATE, it simply simulates a multi-locale, multi-host Arkouda deployment on a single host.
A Docker image that enables a multi-locale, multi-host configurations ideally will support an Arkouda Kubernetes deployment where all locales are deployed within Docker containers. Consequently, the GASNET configuration for such an image would specify the following:
- platform-agnostic COMM_SUBSTRATE
- technology-agnostic, remoteable spawner that integrates with the Kubernetes scheduler
Slurm is normally the choice to schedule locales in multi-locale Arkouda deployments where there's one locale per host. Replacing Slurm with Kubernetes to schedule a multi-local Arkouda cluster would enable a single orchestrator/scheduler for HPC as well as microservice workflows.
One possibility would be to launch a job in Kubernetes which uses the GASNET S spawner over udp to launch the locales. In such a scenario, a helper application would be needed to register the servers and corresponding compute resources to prevent any host from being saturated.
To simplify the problem space, a bare-metal multi-locale, multi-host, non-slurm deployment was studied. A combination of the udp COMM_SUBSTRATE and SSH GASNET_SPAWNFN was studied given that (1) udp is platform-agnostic (2) udp enables a non-Slurm launcher (amudprun) and (3) the generic SSH spawner can be used.
Building Arkouda to enable udp-amudprun-ssh, baremetal deployment is as follows:
tar xvf chapel-1.23.0.tar.gz
cd chapel-1.23.0/
source util/quickstart/setchplenv.bash
export CHPL_COMM_SUBSTRATE=udp
export CHPL_COMM=gasnet
export CHPL_LAUNCHER=amudprun #this is actually the default for udp
export GASNET_SPAWNFN=S
export SSH_SERVERS=‘space-delimited list of hostnames/ip addresses’
make
The next step is to scp the resulting arkouda_server and arkouda_server_real files to all hosts specified in SSH_SERVERS. In addition, password-less SSH via certificates for all hosts must be configured to enable communication between the locales.
The runtime env variables on the Arkouda launch node are as follows:
GASNET_MASTERIP=<host where Arkouda is launched from>
GASNET_SPAWNFN=S
SSH_SERVERS=<space-delimited list of hosts Arkouda is to be deployed to>
Arkouda is launched in the standard way:
./arkouda_server -nl <desired number of locales>
Arkouda client view:
Arkouda server view:
The multi-locale, non-slurm Dockerfile is modeled after the Chapel base Dockerfile, but does not extend it due some customization required to launch a non-Slurm configuration of Arkouda that uses a combination of the udp CHPL_COMM_SUBSTRATE, amudprun CHPL_LAUNCHER, and S GASNET_SPAWNFN spawner. Specifically, the GASNET_MASTERIP and the master SSH_SERVERS hostname must be passed into the image build configuration to enable the check-zmq make target with Chapel compiled with the udp CHPL_COMM_SUBSTRATE.
Consequently, the docker image is built as follows:
docker build -f no-slurm-arkouda-server --build-arg GASNET_MASTERIP='<master/launch host>' --build-arg SSH_SERVERS='<master/launch host' --network host -t hokiegeek2/arkouda-server-no-slurm:0.0.5 .
The corresponding container is started as follows:
docker run -it --rm --network host hokiegeek2/arkouda-server-no-slurm:0.0.5
The arkouda_server-locale0 process is started in the Docker container:
Locales 1..n are started as bare-metal Chapel processes:
Running docker in host networking mode is required for arkouda_server-locale0 process to launch locales 1..n in a native, bare-metal manner. Launching locales 1..n as Docker containers via custom GASNET spawner is currently being studied as of 20210525.
The initial version of multi-locale, udp Arkouda on Kubernetes uses a Helm chart that deploys the arkouda_server-locale0 process within the Docker container discussed above. Also as discussed above, locales 1..n are launched as native Chapel processes via a combination of the udp CHPL_COMM_SUBSTRATE, amudprun CHPL_LAUNCHER, and S GASNET_SPAWNFN spawner:
The helm chart is deployed as follows:
helm install arkouda -n arkouda multilocale-arkouda-server-chart/
Consequently, this initial approach fulfills the following requirements:
- Deploy multi-locale Arkouda in non-smp mode on Kubernetes where there is one Arkouda locale per physical box
- Schedule dockerized arkouda_server-locale0 with Kubernetes
- Utilize the udp CHPL_COMM_SUBSTRATE, amudprun launcher, and SSH spawner to obviate the need for the Slurm scheduler
As detailed above, requirement (3) is satisfied by a combination of (1) scheduling arkouda_server-locale0 via Kubernetes and (2) utilizing the udp CHPL_COMM_SUBSTRATE, amudprun launcher, and SSH spawner to launch locales 1..n. Consequently, the Arkouda cluster is scheduled, in part, by Kubernetes via an arkouda_server-locale0 pod. Importantly, while this approach enables deployment of Arkouda on Kubernetes, locales1..n (1) are not launched in Docker containers and (2) are not scheduled by Kubernetes. Accordingly, the initial approach to multilocale Arkouda on Kubernetes is a hybrid solution of k8s orchestration of arkouda_server-locale0 combined with native deployment of locale1..n Chapel processes that represents the possibility of deploying Arkouda in pure Kubernetes environments.
A very important detail of the initial k8s approach is that, analogous to the Docker-bare metal deployment discussed above, the initial Arkouda on k8s plan requires the arkouda_server-locale0 pod to be deployed in host networking mode. Alternatives that would obviate the need for host networking are currently being studied as of 20210525.
A revised approach was studied that would do the following:
- Deploy all Arkouda locales in Docker containers
- Schedule all Arkouda locales with Kubernetes
- Deploy in all Arkouda containers in the Kubernetes control plane
To enable the arkouda_server process to launch locales 1..n in Docker containers, two possibilities were investigated: (1) the GASNET CSPAWN_CMD custom spawner and (2) pre-launching locales 1..n.
The GASNET C spawner approach involves launching locales 1..n from within the arkouda_server-locale0 pod using the following GASNET env variables:
export GASNET_MASTERIP='arkouda_server-locale0 pod IP address'
export GASNET_SPAWNFN=C
export CSPAWN_SERVERS='space-delimited list of locale 1..n host or pod ip addresses'
export GASNET_CSPAWN_CMD='docker run command'
The main problem with this approach is that the arkouda_server-locale0 pod launches the locale 1..n dockers as opposed to Kubernetes.
This approach leverages the k8s scheduler and Docker containers to launch and manage the lifecycles of two sets of Arkouda pods: arkouda_server-locale0 and locale1..n. Consequently, this approach, if successful will utilize Kubernetes to schedule, as well as control the resources allocated for, all Arkouda locales. In addition, since this approach would utilize locale1..n pods, the supplied IP addresses will be within the Kubernetes control plane and therefore all Arkouda locales will be accessible within Kubernetes for monitoring, metrics, and alerting purposes.
The pre-allocated locales approach involves the following steps:
- Deploy via Helm arkouda-locale pods that will host Arkouda locales 1..n
- Deploy via Helm the arkouda-server pod containing the arkouda_server-locale0 process
- Within the arkouda-server pod discover the arkouda-locale pod IP addresses via the k8s API
- Generate the SSH_SERVERS, GASNET_MASTERIP, and NUMLOCALES env variables within the arkouda-server pod
- Launch arkouda_server-local0 in the arkouda-server pod
- Launch locales 1..n in the arkouda-locale pods
To test out whether an approach leveraging a stepwise deployment of arkouda-locale pods followed by an arkouda-server pod deployment is viable, two Helm charts were developed:
- arkouda-server configures the deployment of a pod containing the arkouda_server-locale0 process
- arkouda-locale configures 1..n pods each containing an environment for launching one of the Arkouda locales 1..n
The arkouda-server Helm chart leverages the arkouda-server the Docker image that has the following:
- Chapel configured for the udp via NV CHPL_COMM_SUBSTRATE=udp, CHPL_COMM=gasnet, CHPL_LAUNCHER=amudprun
- Arkouda installation including arkouda_server and arkouda_server_real needed to start arkouda_server in multi-locale mode
- openssh server for using the GASNET S (SSH) spawner
- start-arkouda-server.sh script used to launch the openssh server and then arkouda_server with the configured number of locales
Initial testing used this image for both the arkouda-server and arkouda-locale Helm charts, but a separate Docker image was developed for arkouda-locale.
The docker builds are as follows:
# base image covers the Chapel portion
docker build --build-arg CHPL_VERSION=1.24.0 -f udp-arkouda-server-base -t hokiegeek2/udp-arkouda-server-base:$VERSION .
docker build --network host --build-arg CHPL_VERSION='1.24.0' --build-arg SSH_SERVERS='0.0.0.0' --build-arg GASNET_MASTERIP='0.0.0.0' -f udp-dynamic-arkouda-server -t hokiegeek2/udp-dynamic-arkouda-server:$VERSION .
The process starts with the Helm installation of 1..n arkouda-locales and arkouda-server:
helm install -n arkouda arkouda-locale multilocale-dynamic-arkouda-locale-chart/
helm install -n arkouda arkouda-server multilocale-dynamic-arkouda-server-chart/
The kubectl command is used to confirm successful deployment of the arkouda-locale and arkouda-server pods:
Using the kubectl command to shell into the arkouda-locale pod, retrieve the pod ip address:
In addition, start the arkouda-locale ssh server to enable launching the arkouda-locale arkouda_server process
Similarly, shell into the arkouda-server pod via kubectl to retrieve that pod's ip address. With the IP addresses corresponding to the arkouda-server and 1...n arkouda-locale pods, configure GASNET_MASTERIP (arkouda-server IP address), SSH_SERVERS (arkouda-server and arkouda-locale pod IP addresses), and NUMLOCALES (total number of arkouda pods):
Execute the start-arkouda-server.sh shell script and confirm all locales started up correctly:
The Arkouda python client view shows both configured locales started up correctly within Kubernetes:
Again shelling into the arkouda-locale and arkouda-server pods, respectively, shows that each pod contains an arkouda process:
arkouda-server:
arkouda-locale:
The proof-of-concept detailed above demonstrates that a pure Kubernetes Arkouda deployment--in other words, an Arkouda deployment where all locales are deployed as Docker containers on Kubernetes--is possible. There are a few development, configuration, and testing tasks to convert the manual, stepwise Kubernetes deployment of Arkouda detailed in the proof-of-concept to one-step Helm deployment.
- Utility to retrieve the arkouda-locale pod IP addresses from the Kubernetes API. As discussed above, this list of arkouda-locale pod IP's will be combined with the arkouda-server IP address to generate the SSH_SERVERS list.
- Utility to generate and set SSH_SERVERS env variable
- Logic to cat id_rsa.pub and pupulate ~/.ssh/authorized_keys
- Logic to start openssh server
- Kubernetes secret and container mount to load id_rsa/id_rsa.pub needed to SSH between containers
- Updated Helm charts to mount id_rsa/id_rsa.pub in .ssh directory via Kubernetes secret
- Utility that sets the request as well as limit cpu cores and memory k8s parameters per theqthreads and memMax parameters, respectively
- Unified, streamlined Helm chart that manages the deployments and dependencies of arkouda-locale and arkouda-server
- Updated configuration that uses Multus for Infiniband connections
- Configuration logic that maps arkouda-server and arkouda-locale pods to IB addresses
(1) implemented as a KubernetesDao that queries the Kubernetes API via the Python client:
ubuntu@arkouda-server-6q5h8:/opt/install$ python3
Python 3.8.5 (default, May 27 2021, 13:30:53)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import arkouda as ak
_ _ _
/ \ _ __| | _____ _ _ __| | __ _
/ _ \ | '__| |/ / _ \| | | |/ _` |/ _` |
/ ___ \| | | < (_) | |_| | (_| | (_| |
/_/ \_\_| |_|\_\___/ \__,_|\__,_|\__,_|
Client Version: 2020.7.7
>>> from arkouda.integration import KubernetesDao
>>> dao = KubernetesDao()
>>> ips = dao.get_pod_ips(namespace='arkouda', app_name='arkouda-server')
>>> ips
['172.17.0.20']
>>>
The dao.get_pod_ips method invocation is wrapped in the start-arkouda.sh script:
(2), (3) and (4) implemented in shell scripts
For arkouda-server, the openssh server startup and ssh configuration is followed by logic to set the SSH_SERVERS env variable and start arkouda_server:
sudo service ssh start
mkdir ~/.ssh/
sudo cp ~/ssh-keys/id_rsa* ~/.ssh/
sudo chown -R ubuntu:ubuntu ~/.ssh/*
chmod -R 600 ~/.ssh/*
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
export LOCALE_IPS=$(python3 /opt/client/arkouda/integration.py 'arkouda' 'arkouda-locale')
export SSH_SERVERS="$MY_IP $LOCALE_IPS"
./arkouda_server -nl ${NUMLOCALES:-1} --memTrack=${MEMTRACK:-true} --authenticate=${AUTHENTICATE:-false} \
--logLevel=${LOG_LEVEL:-LogLevel.INFO}
For arkouda-locale, the openssh server startup is followed by ssh configuration and starting a "keep-alive" function that keeps the arkouda-locale container running. in order to host the arkouda_server process until the master arkouda_server-locale0 process terminates in the arkouda-server container:
sudo service ssh start
mkdir ~/.ssh/
sudo cp ~/ssh-keys/id_rsa* ~/.ssh/
sudo chown -R ubuntu:ubuntu ~/.ssh/*
chmod -R 600 ~/.ssh/*
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
/bin/bash -c -- "while true; do sleep 600; done;"
(5) Is implemented via the following kubectl command:
kubectl create secret generic arkouda-ssh --from-file=id_rsa --from-file=id_rsa.pub -n arkouda
Mounted in the helm chart as follows:
volumeMounts:
- name: ssh-keys
mountPath: "/home/ubuntu/ssh-keys"
...
volumes:
- name: ssh-keys
secret:
secretName: arkouda-ssh
Note: the key pair (id_rsa/id_rsa.pub) can be generated as follows:
ssh-keygen -t ed25519 -C
(8) Currently consists of two Helm charts that are executed in succession:
helm install -n arkouda arkouda-locale multilocale-dynamic-arkouda-locale-chart/
helm install -n arkouda arkouda-server multilocale-dynamic-arkouda-server-chart/
It is critical that deployment of the arkouda-locale pods completes prior to deploying arkouda-server. Otherwise, the arkouda-server pod will not discover all of the arkouda-locale IP addresses, which results in less locales than expected. An umbrella chart can be used to handle the dependency between the arkouda-locale and arkouda-server Helm deployments.
Docker Images for Pure Kubernetes Deployment of Arkouda
There are two Docker images for the pure k8s Arkouda deployment: udp-arkouda-server-base and udp-dynamic-arkouda-server. The udp-arkouda-server-base image does the following:
- Creates an ubuntu user w/ sudo privileges
- Adds a ssh key for the ubuntu user
- Installs openssh-server to enabling spawning Chapel locales via SSH
- Builds a Chapel binary configured for a UDP CHPL_COMM_SUBSTRATE:
CHPL_COMM_SUBSTRATE=udp
CHPL_COMM=gasnet
CHPL_LAUNCHER=amudprun
The udp-dynamic-arkouda-server image does the following:
- Builds arkouda_server configured to spawn Chapel locales via SSH over UDP
- Installs the Arkouda client that accesses the Kubernetes API to retrieve the arkouda-locale pods
- Adds the start-arkouda-server.sh and start-arkouda-locale.sh scripts used to launch arkouda_server-locale0 and an ssh server (start-arkouda-server.sh) or a stay alive process and an ssh server (start-arkouda-locale.sh)
Building Pure Kubernetes Deployment Docker Images
Building the Arkouda Server image involves an important and distinct requirement related to the check-zmq and check-hdf5 make targets: an id_rsa/id_rsa.pub ssh key pair that corresponds to an authorized key for the machine the Docker image; the reason for this requirement is that the make build must do a loopback on the build host machine. The ssh key pair is passed into the image and then deleted at the end of the build process as shown in the Dockerfile:
RUN rm -rf ~/.ssh/
ENTRYPOINT sh /opt/start-arkouda-server.sh
Using the helm charts discussed above, deploy arkouda-locale first, confirm all the pods are running, and then deploy arkouda-server. If both the arkouda-locale and arkouda-server deployments proceeded normally, the kubectl command will show both types of pods running:
The kubectl logs with the arkouda-server pod will show (1) that all nodes within the SSH_SERVER list are accessible via SSH and (2) the arkouda_server-locale0 process started up normally:
Accessing via the Python REPL shows ak.get_config() returns list of two Arkouda locales, both of which having Kubernetes pod hostnames:
As of 20210628, there are two issues with Arkouda deployed via UDP/SSH Spawner that must be resolved, both of which are observed in Arkouda on Kubernetes as well as bare-metal Arkouda:
- Strings object generation is exceptionally slow for several Strings object sizes and fails at sizes 1 million and above
- array -> hdf5 writes work correctly, but reading hdf5 files into Arkouda arrays and Strings fail.
Generating a Strings object involves reading in two Chapel arrays as each Strings object consists of two arrays: (1) bytes, a uint8 char array containing string char sequences separated by null uint8 characters and (2) offsets, which is an int array indicating where each string char sequence starts within the bytes array.
Generation of a 100000 size Strings object in a 3-node Arkouda-UDP-SSH Spawner-Kubernetes cluster takes approximately 2.5 minutes:
Attempts at generating a one million-sized Strings object results in the Arkouda server cluster hanging indefinitely.
In contrast, generation of a 100 million size int array took ~ 110 ms and saving to hdf5 took ~ 350 ms:
Reading in a 100 million-sized int array works incorrectly:
Using the native h5ls utility to inspect a set of three hdf5 files corresponding to a ten-member int array shows all the data was written correctly to hdf5:
Importantly, the same hdf5 read error is also observed in a three locale, bare-metal Arkouda-UDP-SSH Spawner cluster, so the observed read errors are not specific to a Kubernetes and/or Docker deployment:
Additional testing w/ Arkuouda-UDP-SSH Spawner threw an HDF5 lib error:
As detailed in the Arkouda Cloud Study issue, it appears that multiple, distributed threads read each HDF5 file into Chapel. Consequently (confirmed by a core Chapel developer), the Chapel HDF5 read logic requires all files to be on shared, distributed file system in order to enable parallel reads across multiple locales.
Another potential option for deploying and accessing Arkouda via Kubernetes is to deploy Arkouda on an external Slurm cluster from Jupyterhub on Kubernetes.
Specifically, there is a python project entitled slurm-jupyter which enables access to a Slurm cluster via a Jupyter notebook environment. Included in slurm-jupyter is a Slurm job submission UI:
The slurm-jupyter kernel can be integrated into Jupyterhub, the combination of which will enable launching Arkouda on an external Slurm cluster from Kubernetes.
Deploying Arkouda to a Slurm cluster requires (1) installing Chapel and Arkouda dependent libraries on all worker nodes OR a shared directory accessible by all worker nodes, (2) building Arkouda on a bare-metal or Docker image matching the worker nodes and (3) deploying the arkouda_server and arkouda_server_real binaries to all Slurm worker nodes OR a shared directory accessible by all worker nodes.
All slurm worker nodes OR a shared directory accessible by all worker nodes must have Chapel, Arkouda, and all Arkouda dependencies following the Chapel and Arkouda build and installation instructions, respectively.
As noted previously, to utilize the gasnet CHPL_COMM and udp CHPL_COMM_SUBSTRATE, the following env variables must be set prior to building Chapel:
export CHPL_COMM=gasnet
export CHPL_COMM_SUBSTRATE=udp
Also as noted above, the following steps are required to build Chapel in order to execute Chapel/Arkouda in Docker via Kubernetes as well as docker-compose:
export CHPL_GASNET_MORE_CFG_OPTIONS=--enable-force-posix-realtime
cd $CHPL_HOME
touch third-party/gasnet/Makefile
make -j
The Slurm batch script, which is used to launch the Arkouda job on Slurm, is as follows for a three-node Arkouda cluster on a docker-compose-slurm example cluster:
#!/bin/bash
#
#SBATCH --job-name=arkouda-3-node
#SBATCH --output=arkouda.out
#
#SBATCH --ntasks=3
#
export CHPL_GASNET_MORE_CFG_OPTIONS=--enable-force-posix-realtime
export SSH_SERVERS='slurmnode1 slurmnode2 slurmnode3'
sbcast -f /home/admin/arkouda_server /home/admin/arkouda_server
sbcast -f /home/admin/arkouda_server_real /home/admin/arkouda_server_real
/home/admin/arkouda_server -nl 3
A few important notes regarding the above Arkouda slurm batch script:
- The SSH_SERVERS env variable listing the hostnames or ip addresses much be provided since CHPL_COMM=gasnet and CCHPL_COMM_SUBSTRATE=udp
- CHPL_GASNET_MORE_CFG_OPTIONS is required since Arkouda is executing within a Docker container
- Number of tasks matches the number of Arkouda locales, in this case 3
- sbcast is used to deploy arkouda_server and arkouda_server_real to all slurm worker nodes
This slurm docker-compose example makes it easy to deploy a Slurm cluster via docker compose. Included within this example Slurm on-docker-compose example is a slurm-jupyter Docker image.
The docker-compose.yml file is as follows:
services:
slurmjupyter:
image: rancavil/slurm-jupyter:19.05.5-1
hostname: slurmjupyter
user: admin
volumes:
- shared-vol:/home/admin
ports:
- 8888:8888
slurmmaster:
image: rancavil/slurm-master:19.05.5-1
hostname: slurmmaster
user: admin
volumes:
- shared-vol:/home/admin
ports:
- 6817:6817
- 6818:6818
- 6819:6819
slurmnode1:
image: rancavil/slurm-node:19.05.5-1
hostname: slurmnode1
user: admin
volumes:
- shared-vol:/home/admin
environment:
- SLURM_NODENAME=slurmnode1
links:
- slurmmaster
slurmnode2:
image: rancavil/slurm-node:19.05.5-1
hostname: slurmnode2
user: admin
volumes:
- shared-vol:/home/admin
environment:
- SLURM_NODENAME=slurmnode2
links:
- slurmmaster
slurmnode3:
image: rancavil/slurm-node:19.05.5-1
hostname: slurmnode3
user: admin
volumes:
- shared-vol:/home/admin
environment:
- SLURM_NODENAME=slurmnode3
links:
- slurmmaster
volumes:
shared-vol:
After cd to the directory containing the docker-compose.yml file, start up the docker compose cluster as follows:
docker compose up -d
Confirm the cluster is up:
(base) hokiegeek2:hokiegeek2-forks kjyost$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d4f27bc52c56 8a546d3d07f5 "/etc/slurm-llnl/doc…" 5 minutes ago Up 5 minutes 6817-6819/tcp hokiegeek2-forks_slurmnode3_1
dfc715f6bd00 8a546d3d07f5 "/etc/slurm-llnl/doc…" 5 minutes ago Up 5 minutes 6817-6819/tcp hokiegeek2-forks_slurmnode2_1
e50fcdcc4e26 8a546d3d07f5 "/etc/slurm-llnl/doc…" 5 minutes ago Up 5 minutes 6817-6819/tcp hokiegeek2-forks_slurmnode1_1
db3544101ef1 b6b194594c39 "/etc/slurm-llnl/doc…" 5 minutes ago Up 5 minutes 0.0.0.0:8888->8888/tcp, :::8888->8888/tcp hokiegeek2-forks_slurmjupyter_1
4853e4587dd6 f560e7d2dcf9 "/etc/slurm-llnl/doc…" 5 minutes ago Up 5 minutes 3306/tcp, 0.0.0.0:6817-6819->6817-6819/tcp, :::6817-6819->6817-6819/tcp hokiegeek2-forks_slurmmaster_1
(base) hokiegeek2:hokiegeek2-forks kjyost$
Now that the cluster is up, log into the slurmnode1 container, install Chapel dependencies, config env variables, and install Chapel using the excellent instructions on the Arkouda README:
sudo apt-get update && sudo apt install openssh-server hdf5-tools curl -y
export CHPL_COMM=gasnet
export CHPL_COMM_SUBSTRATE=udp
# Switch to /home/admin, download, and extract CHPL
cd ~/
wget https://github.com/chapel-lang/chapel/releases/download/1.24.1/chapel-1.24.1.tar.gz
tar -xvf chapel-1.24.1.tar.gz
cd chapel-1.24.1
export CHPL_HOME=$PWD
source $CHPL_HOME/util/setchplenv.bash
# specific configuration for Chapel-Docker-docker compose/Kubernetes
export CHPL_GASNET_MORE_CFG_OPTIONS=--enable-force-posix-realtime
touch third-party/gasnet/Makefile
# Build chapel
make -j
# update PATH
admin@slurmnode1:~/chapel-1.24.1$ export PATH=$PWD/bin/linux64-x86_64/:$PATH
admin@slurmnode1:~/chapel-1.24.1$ chpl --version
chpl version 1.24.1
Copyright 2020-2021 Hewlett Packard Enterprise Development LP
Copyright 2004-2019 Cray Inc.
(See LICENSE file for more details)
admin@slurmnode1:~/chapel-1.24.1$
# clone and build Arkouda
git clone https://github.com/Bears-R-Us/arkouda.git
cd arkouda
# Generate ssh key and add public ssh key to authorized_keys
admin@slurmnode1:~/arkouda$ ssh-keygen -C admin
Your identification has been saved in /home/admin/.ssh/id_rsa
Your public key has been saved in /home/admin/.ssh/id_rsa.pub
dmin@slurmnode1:~/arkouda$ cat ~/.ssh/id_rsa.pub
admin@slurmnode1:~/arkouda$ vi ~/.ssh/authorized_keys
# Set SSH_SERVERS=localhost and start openssh-server to enable check-zmq
export SSH_SERVERS=localhost
sudo service ssh start
# install deps and build Arkouda
make install-deps
make
Now start ssh-server on each worker node (this has to be done via work station CLI)
sudo service ssh start
The slurm cluster is now ready to launch Arkouda
The Slurm Queue Manager is accessible via the familiar Jupyterhub interface:
The arkouda.slurm is into slurm-jupyter docker:
The Arkouda slurm job is launched as follows:
Confirm the Arkouda slurm job is running by pushing the reload button and confirm it shows in the Slurm queue:
View the arkouda.out file to see that the arkouda_server-locale0 process has started as expected:
SSH to one of the worker nodes to confirm the arkouda_server process is running and that there are the expected number of locales:
Finally, launch a Python REPL and execute ak.get_config() to confirm the expected cluster configuration:
There are two ways to take down the docker-compose slurm cluster: down and stop.
The down option removes all containers and the corresponding docker compose network. Importantly, this means any app installations--including the two needed by Arkouda-gasnet-udp, openssh-server and hdf5-tools--are removed. Consequently, if docker-compose down is invoked, openssh-server and hdf5-tools will need to be reinstalled.
IF docker-compose stop is invoked, all containers and the corresponding docker-compose network are saved. Invoking docker start will start the contains back up.
When the sturm-docker-compose cluster is brought back up, either via docker-compose up -d (needed if cluster is stopped with docker-compose down) or docker-compose start, the ssh server needs to be started on each container via the workstation CLI.
Initial functional testing shows that Arkouda on slurm-udp works well with the exception of generation of Strings objects, which is unacceptably slow. In addition, Strings generation hangs for Strings one million or above in size. However, array generation for hdf5 file reads works correctly because docker-compose-slurm features a shared directory accessible by all worker nodes.
