Fixes: #244 - Add support for S3 storage (#245)

zammad · Jan 29, 2024 · 4f7ac09 · 4f7ac09
1 parent fe2d2fc
commit 4f7ac09
Show file tree

Hide file tree

Showing 7 changed files with 142 additions and 63 deletions.
diff --git a/.gitignore b/.gitignore
@@ -4,3 +4,4 @@ zammad/charts
 zammad/Chart.lock
 zammad/index.yaml
 zammad/requirements.lock
+values.yaml
diff --git a/README.md b/README.md
@@ -1,41 +1,20 @@
-# Zammad Helm chart
+# Zammad Helm Chart
 
 A [Helm](https://helm.sh) chart to install [Zammad](https://zammad.org) on [Kubernetes](https://kubernetes.io)
 
+Please see [zammad/README.md](zammad/README.md) for detailed information & instructions.
+
 ## Repository Info
 
 [![Releases downloads](https://img.shields.io/github/downloads/zammad/zammad-helm/total.svg)](https://github.com/zammad/zammad-helm/releases)
 [![Release Charts](https://github.com/zammad/zammad-helm/workflows/Release%20Charts/badge.svg)](https://github.com/zammad/zammad-helm/commits/master)
 
-## Add the Helm repository via
-
-```console
-helm repo add zammad https://zammad.github.io/zammad-helm
-```
-
-## Version note
-
-Be aware that the Zammad Helm chart version is different from the actual Zammad version.
-
-Due to this charts dependencies the following minimum version requirements apply:
-
-* Kubernetes 1.19+
-* Helm 3.2.0+
-
 ## Sources
 
-The Helm charts sources can be found here:
-
-* [https://github.com/zammad/zammad-helm](https://github.com/zammad/zammad-helm)
-
-The repository source can be found here:
-
-* [https://github.com/zammad/zammad-helm/tree/gh-pages](https://github.com/zammad/zammad-helm/tree/gh-pages)
-
-Helm releases can be found here:
-
-* [https://github.com/zammad/zammad-helm/releases](https://github.com/zammad/zammad-helm/releases)
+* [Helm chart sources](https://github.com/zammad/zammad-helm)
+* [Helm repository source](https://github.com/zammad/zammad-helm/tree/gh-pages)
+* [Helm releases](https://github.com/zammad/zammad-helm/releases)
 
-## Adding changes to this repository
+## Contributing
 
-See our [contributing guidelines](https://github.com/zammad/zammad-helm/blob/master/CONTRIBUTING.md).
+Please see our [contributing guidelines](https://github.com/zammad/zammad-helm/blob/master/CONTRIBUTING.md).
diff --git a/zammad/Chart.yaml b/zammad/Chart.yaml
@@ -1,6 +1,6 @@
 apiVersion: v2
 name: zammad
-version: 10.2.1
+version: 10.3.0
 appVersion: 6.2.0-1
 description: Zammad is a web based open source helpdesk/customer support system with many features to manage customer communication via several channels like telephone, facebook, twitter, chat and e-mails.
 home: https://zammad.org
@@ -16,17 +16,21 @@ maintainers:
 dependencies:
   - name: elasticsearch
     repository: https://charts.bitnami.com/bitnami
-    version: 19.10.3
+    version: 19.17.0
     condition: zammadConfig.elasticsearch.enabled
+  - name: minio
+    version: 13.2.0
+    repository: https://charts.bitnami.com/bitnami
+    condition: zammadConfig.minio.enabled
   - name: memcached
-    version: 6.5.4
+    version: 6.9.0
     repository: https://charts.bitnami.com/bitnami
     condition: zammadConfig.memcached.enabled
   - name: postgresql
-    version: 12.6.4
+    version: 12.12.10
     repository: https://charts.bitnami.com/bitnami
     condition: zammadConfig.postgresql.enabled
   - name: redis
-    version: 17.11.7
+    version: 17.17.1
     repository: https://charts.bitnami.com/bitnami
     condition: zammadConfig.redis.enabled
diff --git a/zammad/README.md b/zammad/README.md
@@ -1,48 +1,93 @@
 # Zammad Helm Chart
 
-This directory contains a Kubernetes chart to deploy [Zammad](https://zammad.org/) ticket system.
+A [Helm](https://helm.sh) chart to install [Zammad](https://zammad.org) on [Kubernetes](https://kubernetes.io)
 
-## Prerequisites Details
+[Zammad](https://zammad.org/) is a web based open source helpdesk/customer support system with many
+features to manage customer communication via several channels like telephone,
+facebook, twitter, chat and emails. It is distributed under version 3 of the
+GNU AFFERO General Public License (GNU AGPLv3).
 
-- Kubernetes 1.19+
-- Helm 3.x
-- Cluster with at least 4GB of free RAM
-
-## Chart Details
+## Introduction
 
 This chart will do the following:
 
-- Install Zammad statefulset
-- Install Elasticsearch, Memcached, PostgreSQL & Redis as requirements
+- Install a Zammad `StatefulSet`
+- Install Elasticsearch, Memcached, PostgreSQL, Redis & Minio (optional) as requirements
+
+Be aware that the Zammad Helm chart version is different from the actual Zammad version.
+
+## Prerequisites
+
+- Kubernetes 1.19+
+- Helm 3.2.0+
+- Cluster with at least 4GB of free RAM
 
 ## Installing the Chart
 
 To install the chart use the following:
 
 ```console
 helm repo add zammad https://zammad.github.io/zammad-helm
-helm upgrade --install zammad zammad/zammad --namespace zammad
+helm upgrade --install zammad zammad/zammad
+```
+
+## Uninstalling the Chart
+
+To remove the chart again use the following:
+
+```console
+helm delete zammad
 ```
 
+This will uninstall the Zammad `StatefulSet`, but keep the associated `PVC`s.
+Please delete them manually if you're sure.
+
 ## Configuration
 
-See [Customizing the Chart Before Installing](https://helm.sh/docs/intro/using_helm/#customizing-the-chart-before-installing). To see all configurable options with detailed comments, visit the chart's [values.yaml](./values.yaml), or run these configuration commands:
+See [Customizing the Chart Before Installing](https://helm.sh/docs/intro/using_helm/#customizing-the-chart-before-installing).
+To see all configurable options with detailed comments, visit the chart's [values.yaml](./values.yaml), or run this configuration command:
 
 ```console
 helm show values zammad/zammad
 ```
 
-### Important note for NFS filesystems
+### Choosing the Storage Provider
+
+Zammad uses the database as the default storage provider for new systems. This works well for the majority of systems.
+Only if you have a large volume of tickets and attachments, you may need to store attachments in another storage provider.
+
+We recommend the `S3` storage provider using the optional `minio` subchart in this case.
+
+You can also use `File` storage. In this case, you should check:
+
+- If you already use an `externalVolumeClaim` with `ReadWriteMany` access, you can keep using that.
+- If you already use an `externalVolumeClaim` with another access mode, we recommend migrating to S3 storage (see below).
+- If you used the default `PVC` of the Zammad `StatefulSet`, we also recommend migrating to S3 storage (see below).
+
+Background information: a future version of Zammad will increase the scalability by splitting up the current `Statefulset`
+into several `Deployment`s which can be scaled. This means the `PVC` of the current `StatefulSet` will then not be usable any more,
+and any volumes will have to support `ReadWriteMany` access.
 
-For persistent volumes, NFS filesystems should work correctly for **Elasticsearch** and **PostgreSQL**; however, errors will occur if Zammad itself uses an NFS-based persistent volume. Websockets will break completely. This is particularly bad news for receiving notifications from the application and using the Chat module.
+#### How to migrate from `File` to `S3` storage
 
-Don't use an NFS-based storage class for Zammad's persistent volume.
+- In the admin panel, go to "System -> Storage" and select "Simple Storage (S3)" as the new storage provider.
+- Migrate existing `File` store content by running `kubectl exec zammad-0 -c zammad-railsserver -- rails r "Store::File.move('File', 'S3')"`. Example:
 
-This is relevant to **EFS** for AWS users, as well.
+```log
+kubectl exec zammad-0 -c zammad-railsserver -- rails r "Store::File.move('File', 'S3')"
+I, [2024-01-24T11:06:13.501572 #168]  INFO -- : ActionCable is using the redis instance at redis://:zammad@zammad-redis-master:6379.
+I, [2024-01-24T11:06:13.506180#168-5980]  INFO -- : Using memcached as Rails cache store.
+I, [2024-01-24T11:06:13.506246#168-5980]  INFO -- : Using the Redis back end for Zammad's web socket session store.
+I, [2024-01-24T11:06:14.561169#168-5980]  INFO -- : storage remove '/opt/zammad/storage/fs/ab76/81d1/a4177/4c41f/12ddb67/96ee19e/a7e7c780a3227936c507cfbfe946afb9'
+I, [2024-01-24T11:06:14.561654#168-5980]  INFO -- : Moved file ab7681d1a41774c41f12ddb6796ee19ea7e7c780a3227936c507cfbfe946afb9 from File to S3
+I, [2024-01-24T11:06:14.566327#168-5980]  INFO -- : storage remove '/opt/zammad/storage/fs/dbaa/01dd/0df3a/33bce/e87c420/f221f59/6df9db38a402b30fccea09cc444a9fb0'
+I, [2024-01-24T11:06:14.566513#168-5980]  INFO -- : Moved file dbaa01dd0df3a33bcee87c420f221f596df9db38a402b30fccea09cc444a9fb0 from File to S3
+I, [2024-01-24T11:06:14.627896#168-5980]  INFO -- : storage remove '/opt/zammad/storage/fs/e81f/fb09/c5a26/f2081/f93401a/cbe8fff/9983e56c86fccb48d17a2eb1e5900b5b'
+```
 
-### OpenShift
+### Deploying on OpenShift
 
-To run OpenShift unprivileged and with [arbitrary UIDs and GIDs](https://cloud.redhat.com/blog/a-guide-to-openshift-and-uids):
+To deploy on OpenShift unprivileged and with [arbitrary UIDs and GIDs](https://cloud.redhat.com/blog/a-guide-to-openshift-and-uids):
 
 - [Delete the default key](https://helm.sh/docs/chart_template_guide/values_files/#deleting-a-default-key)  `securityContext` and `zammadConfig.initContainers.zammad.securityContext.runAsUser` with `null`.
 - Disable if used:
@@ -78,6 +123,12 @@ memcached:
   containerSecurityContext:
     enabled: false
 
+minio:
+  podSecurityContext:
+    enabled: false
+  containerSecurityContext:
+    enabled: false
+
 redis:
   master:
    podSecurityContext:
@@ -91,28 +142,28 @@ redis:
       enabled: false
 ```
 
-## Using zammad
+## Using Zammad
 
-Once the zammad pod is ready, it can be accessed using the ingress or port forwarding.
+Once the Zammad pod is ready, it can be accessed using the ingress or port forwarding.
 To use port forwarding:
 
 ```console
-kubectl -n zammad port-forward service/zammad 8080
+kubectl port-forward service/zammad 8080
 ```
 
-Open your browser on <http://localhost:8080>
+Now you can open <http://localhost:8080> in your browser.
 
 ## Upgrading
 
-### From chart version 9.x to 10.0.0
+### From Chart Version 9.x to 10.0.0
 
 - all containers uses `readOnlyRootFilesystem: true` again
 - volumePermissions init container config has been moved to initContainers section
   - if you used it before you have to adapt your config
   - it's also enabled by default now to workaround rails world writable tmp dir issues
   - if you don't like to use it you might want to set tmpDirVolume.emptyDir.medium to "Memory" instead
 
-### From chart version 8.x to 9.0.0
+### From Chart Version 8.x to 9.0.0
 
 - Zammads PVC changed to only hold contents of /opt/zammad/var & /opt/zammad/storage instead of the whole Zammad content
   - A new PVC `zammad-var` is created for this
@@ -128,7 +179,7 @@ Open your browser on <http://localhost:8080>
 - auto_wizard.json is placed into /opt/zammad/var directory now
 - All subcharts have been updated
 
-### From chart version 7.x to 8.0.0
+### From Chart Version 7.x to 8.0.0
 
 SecurityContexts of pod and containers are configurable now.
 We also changed the default securitycontexts, to be a bit more restrictive, therefore the major version upgrade of the chart.
@@ -166,7 +217,7 @@ If you wan't the old behaviout use the new `.Values.zammadConfig.initContainers.
 
 We've also set the Elasticsearch master heapsize to "512m" by default.
 
-### From chart version 6.x to 7.0.0
+### From Chart Version 6.x to 7.0.0
 
 - Bitnami Elasticsearch chart is used now as Elastic does not support the old chart anymore in favour of ECK operator
   - reindexing of all data is needed so get sure "zammadConfig.elasticsearch.reindex" is set to "true"
@@ -185,27 +236,27 @@ We've also set the Elasticsearch master heapsize to "512m" by default.
   - livenessProbes and readinessProbe have been adjusted to not be the same
   - config values has been removed from chart readme as it's easier to maintain them at a single place
 
-### From chart version 6.0.4 to 6.0.x
+### From Chart Version 6.0.4 to 6.0.x
 
 - minimum helm version now is 3.2.0+
 - minimum Kubernetes version now is 1.19+
 
-### From chart version 5.x to 6.x
+### From Chart Version 5.x to 6.x
 
 - `envConfig` variable was replaced with `zammadConfig`
 - `nginx`, `rails`, `scheduler`, `websocket` and `zammad` vars has been merged into `zammadConfig`
 - Chart dependency vars have changed (reside in `zammadConfig` too now), so if you've disabled any of them you have to adapt to the new values from the Chart.yaml
 - `extraEnv` var is a list now
 
-### From chart version 4.x to 5.x
+### From Chart Version 4.x to 5.x
 
 - health checks have been extended from boolean flags that simply toggle readinessProbes and livenessProbes on the containers to templated
   ones: .zammad.{nginx,rails,websocket}.readinessProbe and .zammad.{nginx,rails,websocket}.livenessProbe have been removed in favor of livenessProbe/readinessProbe
   templates at .{nginx,railsserver,websocket}. You can customize those directly in your overriding values.yaml.
 - resource constraints have been grouped under .{nginx,railsserver,websocket} from above. They are disabled by default (same as prior versions), but in your overrides, make sure
   to reflect those changes.
 
-### From chart version 1.x
+### From Chart Version 1.x
 
 This has changed:
 

diff --git a/zammad/templates/_helpers.tpl b/zammad/templates/_helpers.tpl
@@ -105,3 +105,18 @@ redis secret name
 {{ template "zammad.fullname" . }}-{{ .Values.secrets.redis.secretName }}
 {{- end -}}
 {{- end -}}
+
+{{/*
+S3 access URL
+*/}}
+{{- define "zammad.env.S3_URL" -}}
+{{- with .Values.zammadConfig.minio.externalS3Url -}}
+- name: S3_URL
+  value: {{ . | quote }}
+{{- else -}}
+{{- if .Values.zammadConfig.minio.enabled -}}
+- name: S3_URL
+  value: "http://zammadadmin:zammadadmin@{{ template "zammad.fullname" . }}-minio:9000/zammad?region=zammad&force_path_style=true"
+{{- end -}}
+{{- end -}}
+{{- end -}}
diff --git a/zammad/templates/statefulset.yaml b/zammad/templates/statefulset.yaml
@@ -88,6 +88,7 @@ spec:
                   key: {{ .Values.secrets.postgresql.secretKey }}
             - name: DATABASE_URL
               value: "postgres://{{ .Values.zammadConfig.postgresql.user }}:$(POSTGRESQL_PASS)@{{ if .Values.zammadConfig.postgresql.enabled }}{{ .Release.Name }}-postgresql{{ else }}{{ .Values.zammadConfig.postgresql.host }}{{ end }}:{{ .Values.zammadConfig.postgresql.port }}/{{ .Values.zammadConfig.postgresql.db }}"
+            {{ include "zammad.env.S3_URL" . | nindent 12 }}
           {{- if .Values.extraEnv }}
             {{- toYaml .Values.extraEnv | nindent 12 }}
           {{- end }}
@@ -132,6 +133,7 @@ spec:
                   key: {{ .Values.secrets.postgresql.secretKey }}
             - name: DATABASE_URL
               value: "postgres://{{ .Values.zammadConfig.postgresql.user }}:$(POSTGRESQL_PASS)@{{ if .Values.zammadConfig.postgresql.enabled }}{{ .Release.Name }}-postgresql{{ else }}{{ .Values.zammadConfig.postgresql.host }}{{ end }}:{{ .Values.zammadConfig.postgresql.port }}/{{ .Values.zammadConfig.postgresql.db }}"          {{- if .Values.autoWizard.enabled }}
+            {{ include "zammad.env.S3_URL" . | nindent 12 }}
             - name: AUTOWIZARD_JSON
               valueFrom:
                 secretKeyRef:
@@ -183,6 +185,7 @@ spec:
                   key: {{ .Values.secrets.postgresql.secretKey }}
             - name: DATABASE_URL
               value: "postgres://{{ .Values.zammadConfig.postgresql.user }}:$(POSTGRESQL_PASS)@{{ if .Values.zammadConfig.postgresql.enabled }}{{ .Release.Name }}-postgresql{{ else }}{{ .Values.zammadConfig.postgresql.host }}{{ end }}:{{ .Values.zammadConfig.postgresql.port }}/{{ .Values.zammadConfig.postgresql.db }}"
+            {{ include "zammad.env.S3_URL" . | nindent 12 }}
           {{- if or .Values.zammadConfig.elasticsearch.pass .Values.secrets.elasticsearch.useExisting }}
             - name: ELASTICSEARCH_PASSWORD
               valueFrom:
@@ -304,6 +307,7 @@ spec:
                   key: {{ .Values.secrets.postgresql.secretKey }}
             - name: DATABASE_URL
               value: "postgres://{{ .Values.zammadConfig.postgresql.user }}:$(POSTGRESQL_PASS)@{{ if .Values.zammadConfig.postgresql.enabled }}{{ .Release.Name }}-postgresql{{ else }}{{ .Values.zammadConfig.postgresql.host }}{{ end }}:{{ .Values.zammadConfig.postgresql.port }}/{{ .Values.zammadConfig.postgresql.db }}"
+            {{ include "zammad.env.S3_URL" . | nindent 12 }}
           {{- if .Values.extraEnv }}
           {{- toYaml .Values.extraEnv | nindent 12 }}
           {{- end }}
@@ -360,6 +364,7 @@ spec:
                   key: {{ .Values.secrets.postgresql.secretKey }}
             - name: DATABASE_URL
               value: "postgres://{{ .Values.zammadConfig.postgresql.user }}:$(POSTGRESQL_PASS)@{{ if .Values.zammadConfig.postgresql.enabled }}{{ .Release.Name }}-postgresql{{ else }}{{ .Values.zammadConfig.postgresql.host }}{{ end }}:{{ .Values.zammadConfig.postgresql.port }}/{{ .Values.zammadConfig.postgresql.db }}"
+            {{ include "zammad.env.S3_URL" . | nindent 12 }}
           {{- if .Values.extraEnv }}
           {{- toYaml .Values.extraEnv | nindent 12 }}
           {{- end }}
@@ -409,6 +414,7 @@ spec:
                   key: {{ .Values.secrets.postgresql.secretKey }}
             - name: DATABASE_URL
               value: "postgres://{{ .Values.zammadConfig.postgresql.user }}:$(POSTGRESQL_PASS)@{{ if .Values.zammadConfig.postgresql.enabled }}{{ .Release.Name }}-postgresql{{ else }}{{ .Values.zammadConfig.postgresql.host }}{{ end }}:{{ .Values.zammadConfig.postgresql.port }}/{{ .Values.zammadConfig.postgresql.db }}"
+            {{ include "zammad.env.S3_URL" . | nindent 12 }}
           {{- if .Values.extraEnv }}
           {{- toYaml .Values.extraEnv | nindent 12 }}
           {{- end }}