Persistent Storage Support (Apprise v1.9.0+) Added (#206)

Dockerfile

@@ -47,17 +47,17 @@ FROM base AS runtime
 COPY ./requirements.txt /etc/requirements.txt
 COPY --from=builder /build/*.whl ./
 RUN set -eux && \
-    echo "Installing cryptography" && \
-        pip3 install *.whl && \
-    echo "Installing python requirements" && \
-        pip3 install --no-cache-dir -q -r /etc/requirements.txt gunicorn supervisor && \
     echo "Installing nginx" && \
         apt-get update -qq && \
         apt-get install -y -qq \
             nginx && \
+    echo "Installing cryptography" && \
+        pip3 install *.whl && \
     echo "Installing tools" && \
         apt-get install -y -qq \
-            curl sed && \
+            curl sed git && \
+    echo "Installing python requirements" && \
+        pip3 install --no-cache-dir -q -r /etc/requirements.txt gunicorn supervisor && \
     echo "Cleaning up" && \
         apt-get --yes autoremove --purge && \
         apt-get clean --yes && \

README.md

@@ -43,8 +43,10 @@ Using [dockerhub](https://hub.docker.com/r/caronc/apprise) you can do the follow
 docker pull caronc/apprise:latest
 
 # Start it up:
-# /config is used for a persistent store, you do not have to mount
-#         this if you don't intend to use it.
+# /config/store  is used for a persistent store, you do not have to mount
+#                this if you don't intend to use it.
+# /config is used for a spot to write all of the configuration files
+#         generated through the API
 # /plugin is used for a location you can add your own custom apprise plugins.
 #         You do not have to mount this if you don't intend to use it.
 # /attach is used for file attachments
@@ -90,6 +92,18 @@ docker run --name apprise \
    -e APPRISE_WORKER_COUNT=1 \
    -v /etc/apprise:/config \
    -d apprise/local:latest
+
+# Change your paths to what you want them to be, you may also wish to
+# just do the following:
+mkdir -p config store
+docker run --name apprise \
+   -p 8000:8000 \
+   -e PUID=$(id -u) \
+   -e PGID=$(id -g) \
+   -e APPRISE_STATEFUL_MODE=simple \
+   -e APPRISE_WORKER_COUNT=1 \
+   -v ./config:/config \
+   -d apprise/local:latest
 ```
 A `docker-compose.yml` file is already set up to grant you an instant production ready simulated environment:
 
@@ -125,6 +139,7 @@ Below is a sample of just a simple text response:
 # one or more of the following separated by a comma:
 #  - ATTACH_PERMISSION_ISSUE: Can not write attachments (likely a permission issue)
 #  - CONFIG_PERMISSION_ISSUE: Can not write configuration (likely a permission issue)
+#  - STORE_PERMISSION_ISSUE: Can not write to persistent storage (likely a permission issue)
 curl -X GET http://localhost:8000/status
 ```
 
@@ -138,6 +153,7 @@ The above output may look like this:
   "attach_lock": false,
   "config_lock": false,
   "status": {
+    "persistent_storage": true,
     "can_write_config": true,
     "can_write_attach": true,
     "details": ["OK"]
@@ -147,6 +163,7 @@ The above output may look like this:
 
 - The `attach_lock` always cross references if the `APPRISE_ATTACH_SIZE` on whether or not it is `0` (zero) or less.
 - The `config_lock` always cross references if the `APPRISE_CONFIG_LOCK` is enabled or not.
+- The `status.persistent_storage` defines if the persistent storage is enabled or not.  If the environment variable `APPRISE_STORAGE_PATH` is empty, this value will always read `false` and it will not impact the `status.details`
 - The `status.can_write_config` defines if the configuration directory is writable or not.  If the environment variable `APPRISE_STATEFUL_MODE` is set to `disabled`, this value will always read `false` and it will not impact the `status.details`
 - The `status.can_write_attach` defines if the attachment directory is writable or not.  If the environment variable `APPRISE_ATTACH_SIZE`. This value will always read `false` and it will not impact the `status.details`.
 - The `status.details` identifies the overall status. If there is more then 1 issue to report here, they will all show in this list.  In a working orderly environment, this will always be set to `OK` and the http response type will be `200`.
@@ -383,6 +400,10 @@ The use of environment variables allow you to provide over-rides to default sett
 | `APPRISE_DEFAULT_THEME` | Can be set to `light` or `dark`; it defaults to `light` if not otherwise provided.  The theme can be toggled from within the website as well.
 | `APPRISE_DEFAULT_CONFIG_ID` | Defaults to `apprise`.   This is the presumed configuration ID you always default to when accessing the configuration manager via the website.
 | `APPRISE_CONFIG_DIR` | Defines an (optional) persistent store location of all configuration files saved. By default:<br/> - Configuration is written to the `apprise_api/var/config` directory when just using the _Django_ `manage runserver` script. However for the path for the container is `/config`.
+| `APPRISE_STORAGE_DIR` | Defines an (optional) persistent store location of all cache files saved. By default persistent storage is written into the `<APPRISE_CONFIG_DIR>/store`.
+| `APPRISE_STORAGE_MODE` | Defines the storage mode to use.  If no `APPRISE_STORGE_DIR` is identified, then this is set to `memory` in all circumtances reguardless what it might otherwise be set to. The possible options are:<br/>📌 **auto**: This is also the default. Writes cache files on demand only. <br/>📌 **memory**: Persistent storage is disabled; local memory is used for simple internal references. This is effectively the behavior of Apprise of versions 1.8.1 and earlier.<br/>📌 **flush**: A bit more i/o intensive then `auto`.  Content is written to disk constantly if changed in anyway. This mode is still experimental.
+| `APPRISE_STORAGE_UID_LENGTH` | Defines the unique key lengths used to identify an Apprise URL.  By default this is set to `8`.  Value can not be set to a smaller value then `2` or larger then `64`.
+| `APPRISE_STATELESS_STORAGE` | Allow stateless URLs (in addition to stateful) to also leverage persistent storage. This defaults to `no` and can however be set to `yes` by simply defining the global variable as such.
 | `APPRISE_ATTACH_DIR` | The directory the uploaded attachments are placed in. By default:<br/> - Attachments are written to the `apprise_api/var/attach` directory when just using the _Django_ `manage runserver` script. However for the path for the container is `/attach`.
 | `APPRISE_ATTACH_SIZE` | Over-ride the attachment size (defined in MB). By default it is set to `200` (Megabytes). You can set this up to a maximum value of `500` which is the restriction in place for NginX (internal hosting ervice) at this time.  If you set this to zero (`0`) then attachments will not be passed along even if provided.
 | `APPRISE_UPLOAD_MAX_MEMORY_SIZE` | Over-ride the in-memory accepted payload size (defined in MB). By default it is set to `3` (Megabytes). There is no reason the HTTP payload (excluding attachments) should exceed this limit.  This value is only configurable for those who have edge cases where there are exceptions to this rule.

apprise_api/api/management/commands/storeprune.py

@@ -0,0 +1,45 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2023 Chris Caron <lead2gold@gmail.com>
+# All rights reserved.
+#
+# This code is licensed under the MIT License.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files(the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and / or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions :
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+from django.core.management.base import BaseCommand
+from django.conf import settings
+import apprise
+
+
+class Command(BaseCommand):
+    help = f"Prune all persistent content older then {settings.APPRISE_STORAGE_PRUNE_DAYS} days()"
+
+    def add_arguments(self, parser):
+        parser.add_argument("-d", "--days", type=int, default=settings.APPRISE_STORAGE_PRUNE_DAYS)
+
+    def handle(self, *args, **options):
+        # Persistent Storage cleanup
+        apprise.PersistentStore.disk_prune(
+            path=settings.APPRISE_STORAGE_DIR,
+            expires=options["days"] * 86400, action=True,
+        )
+        self.stdout.write(
+            self.style.SUCCESS('Successfully pruned persistent storeage (days: %d)' % options["days"])
+        )

apprise_api/api/templates/config.html

@@ -351,8 +351,14 @@ <h4>{% trans "Persistent Store Endpoints" %}</h4>
     let code = document.createElement('code');
     let li = document.createElement('li');
     code.textContent = entry.url;
-    li.setAttribute('class', 'card-panel');
     li.appendChild(code);
+    li.setAttribute('class', 'card-panel');
+    if (entry.id) {
+      let url_id = document.createElement('code');
+      url_id.setAttribute('class', 'url-id');
+      url_id.textContent = entry.id;
+      li.appendChild(url_id);
+    }
 
     urlList.appendChild(li);
     // Store `all` tag

apprise_api/api/tests/test_cli.py

@@ -0,0 +1,35 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2023 Chris Caron <lead2gold@gmail.com>
+# All rights reserved.
+#
+# This code is licensed under the MIT License.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files(the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and / or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions :
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+import io
+from django.test import SimpleTestCase
+from django.core import management
+
+
+class CommandTests(SimpleTestCase):
+
+    def test_command_style(self):
+        out = io.StringIO()
+        management.call_command('storeprune', days=40, stdout=out)

apprise_api/api/tests/test_healthecheck.py

@@ -66,6 +66,7 @@ def test_healthcheck_simple(self):
             'config_lock': False,
             'attach_lock': False,
             'status': {
+                'persistent_storage': True,
                 'can_write_config': True,
                 'can_write_attach': True,
                 'details': ['OK']
@@ -90,6 +91,7 @@ def test_healthcheck_simple(self):
                 'config_lock': True,
                 'attach_lock': False,
                 'status': {
+                    'persistent_storage': True,
                     'can_write_config': False,
                     'can_write_attach': True,
                     'details': ['OK']
@@ -113,6 +115,7 @@ def test_healthcheck_simple(self):
                 'config_lock': False,
                 'attach_lock': False,
                 'status': {
+                    'persistent_storage': True,
                     'can_write_config': False,
                     'can_write_attach': True,
                     'details': ['OK']
@@ -136,6 +139,7 @@ def test_healthcheck_simple(self):
                 'config_lock': False,
                 'attach_lock': True,
                 'status': {
+                    'persistent_storage': True,
                     'can_write_config': True,
                     'can_write_attach': False,
                     'details': ['OK']
@@ -159,6 +163,7 @@ def test_healthcheck_simple(self):
                 'config_lock': False,
                 'attach_lock': False,
                 'status': {
+                    'persistent_storage': True,
                     'can_write_config': True,
                     'can_write_attach': True,
                     'details': ['OK']
@@ -172,6 +177,7 @@ def test_healthcheck_library(self):
 
         result = healthcheck(lazy=True)
         assert result == {
+            'persistent_storage': True,
             'can_write_config': True,
             'can_write_attach': True,
             'details': ['OK']
@@ -180,6 +186,7 @@ def test_healthcheck_library(self):
         # A Double lazy check
         result = healthcheck(lazy=True)
         assert result == {
+            'persistent_storage': True,
             'can_write_config': True,
             'can_write_attach': True,
             'details': ['OK']
@@ -192,35 +199,91 @@ def test_healthcheck_library(self):
             # We still succeed; we just don't leverage our lazy check
             # which prevents addition (unnessisary) writes
             assert result == {
+                'persistent_storage': True,
                 'can_write_config': True,
                 'can_write_attach': True,
                 'details': ['OK'],
             }
 
+        # Force a lazy check where we can't acquire the modify time
+        with mock.patch('os.path.getmtime') as mock_getmtime:
+            mock_getmtime.side_effect = OSError()
+            result = healthcheck(lazy=True)
+            # We still succeed; we just don't leverage our lazy check
+            # which prevents addition (unnessisary) writes
+            assert result == {
+                'persistent_storage': True,
+                'can_write_config': False,
+                'can_write_attach': False,
+                'details': [
+                    'CONFIG_PERMISSION_ISSUE',
+                    'ATTACH_PERMISSION_ISSUE',
+                ]}
+
         # Force a non-lazy check
         with mock.patch('os.makedirs') as mock_makedirs:
             mock_makedirs.side_effect = OSError()
             result = healthcheck(lazy=False)
             assert result == {
+                'persistent_storage': False,
                 'can_write_config': False,
                 'can_write_attach': False,
                 'details': [
                     'CONFIG_PERMISSION_ISSUE',
                     'ATTACH_PERMISSION_ISSUE',
+                    'STORE_PERMISSION_ISSUE',
                 ]}
 
-            mock_makedirs.side_effect = (None, OSError())
+            with mock.patch('os.path.getmtime') as mock_getmtime:
+                with mock.patch('os.fdopen', side_effect=OSError()):
+                    mock_getmtime.side_effect = OSError()
+                    mock_makedirs.side_effect = None
+                    result = healthcheck(lazy=False)
+                    assert result == {
+                        'persistent_storage': True,
+                        'can_write_config': False,
+                        'can_write_attach': False,
+                        'details': [
+                            'CONFIG_PERMISSION_ISSUE',
+                            'ATTACH_PERMISSION_ISSUE',
+                        ]}
+
+            with mock.patch('apprise.PersistentStore.flush', return_value=False):
+                result = healthcheck(lazy=False)
+                assert result == {
+                    'persistent_storage': False,
+                    'can_write_config': True,
+                    'can_write_attach': True,
+                    'details': [
+                        'STORE_PERMISSION_ISSUE',
+                    ]}
+
+            # Test a case where we simply do not define a persistent store path
+            # health checks will always disable persistent storage
+            with override_settings(APPRISE_STORAGE_DIR=""):
+                with mock.patch('apprise.PersistentStore.flush', return_value=False):
+                    result = healthcheck(lazy=False)
+                    assert result == {
+                        'persistent_storage': False,
+                        'can_write_config': True,
+                        'can_write_attach': True,
+                        'details': ['OK']}
+
+            mock_makedirs.side_effect = (OSError(), OSError(), None, None, None, None)
             result = healthcheck(lazy=False)
             assert result == {
-                'can_write_config': True,
+                'persistent_storage': True,
+                'can_write_config': False,
                 'can_write_attach': False,
                 'details': [
+                    'CONFIG_PERMISSION_ISSUE',
                     'ATTACH_PERMISSION_ISSUE',
                 ]}
 
-            mock_makedirs.side_effect = (OSError(), None)
+            mock_makedirs.side_effect = (OSError(), None, None, None, None)
             result = healthcheck(lazy=False)
             assert result == {
+                'persistent_storage': True,
                 'can_write_config': False,
                 'can_write_attach': True,
                 'details': [

apprise_api/api/tests/test_stateless_notify.py

@@ -637,23 +637,25 @@ def test_notify_with_filters(self, mock_send):
 
         # Send our service with the `json://` denied
         with override_settings(APPRISE_ALLOW_SERVICES=""):
-            with override_settings(APPRISE_DENY_SERVICES="json"):
-                # Send our notification as a JSON object
-                response = self.client.post(
-                    '/notify',
-                    data=json.dumps(json_data),
-                    content_type='application/json',
-                )
-
-                # json:// is disabled
-                assert response.status_code == 204
-                assert mock_send.call_count == 0
-
-                # What actually took place behind close doors:
-                assert N_MGR['json'].enabled is False
-
-                # Reset our flag (for next test)
-                N_MGR['json'].enabled = True
+            # Test our stateless storage setting (just to kill 2 birds with 1 stone)
+            with override_settings(APPRISE_STATELESS_STORAGE="yes"):
+                with override_settings(APPRISE_DENY_SERVICES="json"):
+                    # Send our notification as a JSON object
+                    response = self.client.post(
+                        '/notify',
+                        data=json.dumps(json_data),
+                        content_type='application/json',
+                    )
+
+                    # json:// is disabled
+                    assert response.status_code == 204
+                    assert mock_send.call_count == 0
+
+                    # What actually took place behind close doors:
+                    assert N_MGR['json'].enabled is False
+
+                    # Reset our flag (for next test)
+                    N_MGR['json'].enabled = True
 
         # Reset Mock
         mock_send.reset_mock()