skewer.yaml

title: Redis Multicloud High Availability using Skupper
subtitle: Secure Redis servers across multiple distributed Kubernetes clusters
workflow: null
overview: |
  This example deploys a simple highly available Redis architecture with
  Sentinel across multiple Kubernetes clusters using Skupper.

  In addition to the Redis Server and Sentinel, the example contains
  an additional service:

  * A wiki-getter service that exposes an `/api/search?query=` endpoint. 
    The server returns the result from the Redis cache if present otherwise
    it will retrieve the query via the `wiki api` and cache the content via
    the Redis primary server.

  With Skupper, you can place the Redis primary server in one cluster and 
  the replica servers in alternative clusters without requiring that
  the servers be exposed to the public internet.
prerequisites:
sites:
  west:
    title: West
    platform: kubernetes
    namespace: west
    env:
      KUBECONFIG: ~/.kube/config-west
  east:
    title: East
    platform: kubernetes
    namespace: east
    env:
      KUBECONFIG: ~/.kube/config-east
  north:
    title: North
    platform: kubernetes
    namespace: north
    env:
      KUBECONFIG: ~/.kube/config-north
  podman-west:
    title: Podman West
    platform: podman
    env:
      SKUPPER_PLATFORM: podman
steps:
  - standard: general/install_the_skupper_command_line_tool
  - standard: kubernetes/set_up_your_kubernetes_cluster
  - standard: podman/set_up_your_podman_environment
  - title: Create your sites
    preamble: |
      A Skupper _site_ is a location where components of your
      application are running.  Sites are linked together to form a
      Skupper network for your application.

      In Kubernetes, use `skupper init` to create a site.  This
      deploys the Skupper router and controller.  Then use `skupper
      status` to see the outcome.

      In Podman, use `skupper init` with the option `--ingress none`
      and use `skupper status` to see the result.

      **Note:** If you are using Minikube, you need to [start minikube
      tunnel][minikube-tunnel] before you run `skupper init`.

      [minikube-tunnel]: https://skupper.io/start/minikube.html#running-minikube-tunnel
    commands:
      west:
        - run: skupper init --site-name west
          output: |
            Waiting for LoadBalancer IP or hostname...
            Waiting for status...
            Skupper is now installed in namespace 'west'.  Use 'skupper status' to get more information.
      east:
        - run: skupper init --site-name east
          output: |
            Waiting for LoadBalancer IP or hostname...
            Waiting for status...
            Skupper is now installed in namespace 'east'.  Use 'skupper status' to get more information.
      north:
        - run: skupper init --site-name north
          output: |
            Waiting for LoadBalancer IP or hostname...
            Waiting for status...
            Skupper is now installed in namespace 'north'.  Use 'skupper status' to get more information.
      podman-west:
        - run: skupper init --site-name podman-west --ingress none
          output: |
            It is recommended to enable lingering for currentuser, otherwise Skupper may not start on boot.
            Skupper is now installed for user 'currentuser'.  Use 'skupper status' to get more information.
    postamble: |
      As you move through the steps below, you can use `skupper status` at
      any time to check your progress.
  - title: Link your sites
    preamble: |
      A Skupper _link_ is a channel for communication between two sites.
      Links serve as a transport for application connections and
      requests.

      Creating a link requires use of two `skupper` commands in
      conjunction, `skupper token create` and `skupper link create`.

      The `skupper token create` command generates a secret token that
      signifies permission to create a link.  The token also carries the
      link details.  Then, in a remote site, The `skupper link
      create` command uses the token to create a link to the site
      that generated it.

      **Note:** The link token is truly a *secret*.  Anyone who has the
      token can link to your site.  Make sure that only those you trust
      have access to it.

      In this example, for the purpose of availability, all sites are linked together.
      This is not a requirement as service communications works across intermediate sites
      by way of a shortest path traversal.
    commands:
      west:
        - run: skupper token create ~/west.token --uses 3
          output: Token written to ~/west.token
      east:
        - run: skupper token create ~/east.token --uses 2
          output: Token written to ~/east.token        
        - run: skupper link create ~/west.token
        - run: skupper link status --wait 60
          apply: test
      north:
        - run: skupper token create ~/north.token --uses 1
        - run: skupper link create ~/west.token
        - run: skupper link create ~/east.token
        - run: skupper link status --wait 60
          apply: test
      podman-west:
        - run: skupper link create ~/west.token
        - run: skupper link create ~/east.token
        - run: skupper link create ~/north.token
    postamble: |
      If your terminal sessions are on different machines, you may need
      to use `scp` or a similar tool to transfer the token securely.  By
      default, tokens expire after a single use or 15 minutes after
      creation.
  - title: Deploy Redis Server and Sentinel
    preamble: |
      A _yaml_ file defines the resources for the Redis deployment in each
      site. Each contains:

      - A Server deployment resource
      - A Sentinel deployment resource
      - A Redis config map for configuration

      ** Note ** the `redis-north.yaml` file designates the server to be
      primary while the other sites are designated as replica sites to north.
    commands: 
      west:
        - run: kubectl apply -f redis-west.yaml
          output: |
            deployment.apps/redis-server created
            configmap/redis created
            deployment.apps/redis-sentinel created
      east:
        - run: kubectl apply -f redis-east.yaml
          output: |
            deployment.apps/redis-server created
            configmap/redis created
            deployment.apps/redis-sentinel created
      north:
        - run: kubectl apply -f redis-north.yaml
          output: |
            deployment.apps/redis-server created
            configmap/redis created
            deployment.apps/redis-sentinel created        
    postamble: |
      ** Note ** the Sentinel deployments in each site use an init container
      that waits for the service definitions of the Redis servers in all
      sites to exist before execution. This will be satisfied by the next
      step.
  - title: Expose Redis Server and Sentinel to Application Network
    preamble: |
      We will skupper expose the server and sentinel deployments in each namespace
    commands: 
      west:
        - await_resource: deployment/redis-server
        - run: skupper expose deployment redis-server --address redis-server-west
          output: |
            deployment redis-server exposed as redis-server-west
        - run: skupper expose deployment redis-sentinel --address redis-sentinel-west
          output: |
            deployment redis-sentinel exposed as redis-sentinel-west
      east:
        - await_resource: deployment/redis-server
        - run: skupper expose deployment redis-server --address redis-server-east
          output: |
            deployment redis-server exposed as redis-server-east
        - run: skupper expose deployment redis-sentinel --address redis-sentinel-east
          output: |
            deployment redis-sentinel exposed as redis-sentinel-east      
      north:
        - await_resource: deployment/redis-server
        - run: skupper expose deployment redis-server --address redis-server-north
          output: |
            deployment redis-server exposed as redis-server-north
        - run: skupper expose deployment redis-sentinel --address redis-sentinel-north
          output: |
            deployment redis-sentinel exposed as redis-sentinel-north
  - title: Observe the set of Redis server and Sentinel services in each Site
    commands:
      west: 
        - run: kubectl get services
          output: |
            NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE
            redis-sentinel-east    ClusterIP   172.21.112.88    <none>        26379/TCP                      3m37s
            redis-sentinel-north   ClusterIP   172.21.67.12     <none>        26379/TCP                      3m35s
            redis-sentinel-west    ClusterIP   172.21.6.3       <none>        26379/TCP                      9m4s
            redis-server-east      ClusterIP   172.21.241.21    <none>        6379/TCP                       3m45s
            redis-server-north     ClusterIP   172.21.194.72    <none>        6379/TCP                       3m42s
            redis-server-west      ClusterIP   172.21.236.8     <none>        6379/TCP                       9m8s
      east: 
        - run: kubectl get services
          output: |
            NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE
            redis-sentinel-east    ClusterIP   172.21.243.22    <none>        26379/TCP                      5m10s
            redis-sentinel-north   ClusterIP   172.21.233.113   <none>        26379/TCP                      5m8s
            redis-sentinel-west    ClusterIP   172.21.179.114   <none>        26379/TCP                      10m
            redis-server-east      ClusterIP   172.21.17.63     <none>        6379/TCP                       5m18s
            redis-server-north     ClusterIP   172.21.224.55    <none>        6379/TCP                       5m15s
            redis-server-west      ClusterIP   172.21.3.180     <none>        6379/TCP                       10m
      north: 
        - run: kubectl get services
          output: |
            NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE
            redis-sentinel-east    ClusterIP   172.21.211.51    <none>        26379/TCP                      5m54s
            redis-sentinel-north   ClusterIP   172.21.66.49     <none>        26379/TCP                      5m52s
            redis-sentinel-west    ClusterIP   172.21.83.179    <none>        26379/TCP                      11m
            redis-server-east      ClusterIP   172.21.54.164    <none>        6379/TCP                       6m2s
            redis-server-north     ClusterIP   172.21.138.118   <none>        6379/TCP                       5m59s
            redis-server-west      ClusterIP   172.21.224.110   <none>        6379/TCP                       11m
  - title: Create Redis services on podman site
    preamble: |
      The podman site does not take part in exchange, service have to be created
    commands: 
      podman-west:
        - run: skupper service create redis-server-north 6379 --host-port 6379
        - run: skupper service create redis-server-west 6379 --host-port 6380
        - run: skupper service create redis-server-east 6379 --host-port 6381
        - run: skupper service create redis-sentinel-north 26379 --host-port 26379
        - run: skupper service create redis-sentinel-west 26379 --host-port 26380
        - run: skupper service create redis-sentinel-east 26379 --host-port 26381
        - run: skupper service status
          output: |
            Services exposed through Skupper:
            ├─ redis-sentinel-east:26379 (tcp)
            │  ╰─ Host ports:
            │     ╰─ ip: * - ports: 26381 -> 26379
            ├─ redis-sentinel-north:26379 (tcp)
            │  ╰─ Host ports:
            │     ╰─ ip: * - ports: 26379 -> 26379
            ├─ redis-sentinel-west:26379 (tcp)
            │  ╰─ Host ports:
            │     ╰─ ip: * - ports: 26380 -> 26379
            ├─ redis-server-east:6379 (tcp)
            │  ╰─ Host ports:
            │     ╰─ ip: * - ports: 6381 -> 6379
            ├─ redis-server-north:6379 (tcp)
            │  ╰─ Host ports:
            │     ╰─ ip: * - ports: 6379 -> 6379
            ╰─ redis-server-west:6379 (tcp)
               ╰─ Host ports:
                     ╰─ ip: * - ports: 6380 -> 6379
  - title: Use Redis command line interface to verify master status
    preamble: |
      Running the `redis-cli` from the podman-west site, attach to the Redis server
      and Sentinel to verfy that the redis-server-north is master.
    commands: 
      podman-west:
        - run: redis-cli -p 6379
        - run: 127.0.0.1:6379> ROLE
          output: |
            1) "master"
            2) (integer) 1531796
            3) 1) 1) "redis-server-west"
                  2) "6379"
                  3) "1531796"
               2) 1) "redis-server-east"
                  2) "6379"
                  3) "1531796"
        - run: 127.0.0.1:6379> exit
        - run: redis-cli -p 26379
        - run: 127.0.0.1:26379> sentinel get-master-addr-by-name redis-skupper
          output: |
           1) "redis-server-north"
           2) "6379"
        - run: 127.0.0.1:26379> exit
  - title: Deploy the wiki-getter service
    preamble: |
      We will choose the north namespace to create a wiki-getter deployment
      and service. The client in the service will determine the 
      Sentinel service to access the current Redis primary server
      for query and cache updates.
    commands:
      north:
        - run: kubectl apply -f wiki-getter.yaml
          output: |
            deployment.apps/wiki-getter created
            service/wiki-getter created
  - title: Get Wiki content
    preamble: |
      Use `curl` to send a request to querty the Wikipedia API via the 
      wiki-getter service. Note the *X-Response-Time* header for the initial
      query. The application will check the redis cache and if not found
      will fetch from the external Wikipedia API. If the content has been 
      stored, the applications will provide the response directly.
    commands:
      north:
        - run:  kubectl exec -it deployment/wiki-getter -- curl -f -I --head http://wiki-getter:8080/api/search?query=Boston
          output: |
            HTTP/1.1 200 OK
            X-Powered-By: Express
            Content-Type: application/json; charset=utf-8
            Content-Length: 132099
            ETag: W/"20403-PfBW245Yreh1Gm27jHeiM01Wox8"
            X-Response-Time: 1545.706ms
            Date: Fri, 29 Mar 2024 12:48:53 GMT
            Connection: keep-alive
            Keep-Alive: timeout=5
        - run:  kubectl exec -it deployment/wiki-getter -- curl -f -I --head http://wiki-getter:8080/api/search?query=Boston
          output: |
            HTTP/1.1 200 OK
            X-Powered-By: Express
            Content-Type: application/json; charset=utf-8
            Content-Length: 132097
            ETag: W/"20401-7u+hiY6DPz+D2DHbukm0QE/L82s"
            X-Response-Time: 5.847ms
            Date: Fri, 29 Mar 2024 12:48:58 GMT
            Connection: keep-alive
            Keep-Alive: timeout=5
  - title: Force Sentinel failover
    preamble: |
      Using the Sentinel command, force a failover as if the master was not reachable. This
      will result in the promotion of one of the slave Redis servers to master role.
    commands:
      podman-west:
        - run: redis-cli -p 26379
        - run: 127.0.0.1:26379> sentinel failover redis-skupper
          output: |
            OK
            (0.66s)
        - run: 127.0.0.1:26379> sentinel get-master-addr-by-name redis-skupper
          output: |
            1) "redis-server-west"
            2) "6379"
        - run: 127.0.0.1:26379> exit
    postamble: |
      Note that `redis-server-east` may have alternatively been elected master role.
  - title: Verify Wiki content
    preamble: |
      Check that cached content is correctly returned from new master.
    commands:
      north:
        - run:  kubectl exec -it deployment/wiki-getter -- curl -f -I --head http://wiki-getter:8080/api/search?query=Boston
          output: |
            HTTP/1.1 200 OK
            X-Powered-By: Express
            Content-Type: application/json; charset=utf-8
            Content-Length: 132097
            ETag: W/"20401-7u+hiY6DPz+D2DHbukm0QE/L82s"
            X-Response-Time: 152.056ms
            Date: Fri, 29 Mar 2024 12:50:45 GMT
            Connection: keep-alive
            Keep-Alive: timeout=5
  - title: Use Redis command line to measure latency of servers from each site
    preamble: |
      To understand latency in a true multicloud scenario, the redis-cli can be used to
      measure the latency of a Redis server in milliseconds from any application network
      site. 
    commands:
      west:
        - run: kubectl exec -it deployment/redis-server -- redis-cli --latency -h redis-server-west -p 6379 --raw
          output: |
            0 10 1.41 88
        - run: kubectl exec -it deployment/redis-server -- redis-cli --latency -h redis-server-east -p 6379 --raw
          output: |
            76 254 96.40 10
        - run: kubectl exec -it deployment/redis-server -- redis-cli --latency -h redis-server-north -p 6379 --raw
          output: |
            33 104 44.00 19
      east:
        - run: kubectl exec -it deployment/redis-server -- redis-cli --latency -h redis-server-west -p 6379 --raw
          output: |
            79 110 85.45 11
        - run: kubectl exec -it deployment/redis-server -- redis-cli --latency -h redis-server-east -p 6379 --raw
          output: |
            0 26 1.28 89
        - run: kubectl exec -it deployment/redis-server -- redis-cli --latency -h redis-server-north -p 6379 --raw
          output: |
            113 358 149.14 7
      north:
        - run: kubectl exec -it deployment/redis-server -- redis-cli --latency -h redis-server-west -p 6379 --raw
          output: |
            33 103 38.71 21
        - run: kubectl exec -it deployment/redis-server -- redis-cli --latency -h redis-server-east -p 6379 --raw
          output: |
            115 161 125.25 8
        - run: kubectl exec -it deployment/redis-server -- redis-cli --latency -h redis-server-north -p 6379 --raw
          output: |
            0 19 0.88 94
      podman-west:
        - run: redis-cli --latency -p 6379 --raw
          output: |
            62 88 68.85 13
        - run: redis-cli --latency -p 6380 --raw
          output: |
            28 38 32.88 24
        - run: redis-cli --latency -p 6381 --raw
          output: |
            110 280 141.43 7
    postamble: |
      The sample period output is latency min, max, average over the number of samples. Note, that the sample
      outputs provided are actual measures across three public cloud locations (Washington DC, London, and Dallas)
  - title: Cleaning up
    preamble: |
      To remove Skupper and other resource from this exercise, use the
      following commands.
    commands:
      west:
        - run: kubectl delete -f redis-west.yaml
        - run: skupper delete
      east:
        - run: kubectl delete -f redis-east.yaml
        - run: skupper delete
      north: 
        - run: kubectl delete -f wiki-getter.yaml
        - run: kubectl delete -f redis-north.yaml
        - run: skupper delete
      podman-west:
        - run: skupper delete
summary: |
  This example locates the Redis Server and Sentinel services in different
  namespaces, on different clusters.  Ordinarily, this means that they
  have no way to communicate unless they are exposed to the public
  internet.

  Introducing Skupper into each namespace allows us to create a virtual
  application network that can connect Redis services in different clusters.
  Any service exposed on the application network is represented as a
  local service in all of the linked namespaces.

  The Redis primary server is located in `north`, but the Redis replica
  services in `west` and `east` can "see" it as if it were local.
  Redis replica operations take place by service name and Skupper
  forwards the requests to the namespace where the corresponding server
  is running and routes the response back appropriately.