Crypto module (#169)

* Crypto module

* Disable acceptance tests

* Remove TypedDict for py3.7 compatibility

* Fix compatibility with py3.7... again

* Remove randbytes for 3.7 compatibility. sigh

* Post review fixes

* Integrate crypto_module with pubnub

* Update test matrix - drop support for py3.7 as it has reached end of life

* Fix type, add missing params, add example

* Add tests

* Fix bug with always riv encrypting files

* reenable acceptance tests for crypto module

* Fix miss of encrypting files

* Update license

* PubNub SDK v7.3.0 release.

---------

Co-authored-by: PubNub Release Bot <120067856+pubnub-release-bot@users.noreply.github.com>

.github/workflows/run-tests.yml

@@ -26,7 +26,7 @@ jobs:
     strategy:
       fail-fast: true
       matrix:
-        python: [3.7.17, 3.8.17, 3.9.17, 3.10.12, 3.11.4]
+        python: [3.8.18, 3.9.18, 3.10.13, 3.11.6]
     steps:
       - name: Checkout repository
         uses: actions/checkout@v3
@@ -78,9 +78,13 @@ jobs:
             cp sdk-specifications/features/access/authorization-failure-reporting.feature tests/acceptance/pam
             cp sdk-specifications/features/access/grant-token.feature tests/acceptance/pam
             cp sdk-specifications/features/access/revoke-token.feature tests/acceptance/pam
+            cp sdk-specifications/features/encryption/cryptor-module.feature tests/acceptance/encryption
+            mkdir tests/acceptance/encryption/assets/
+            cp sdk-specifications/features/encryption/assets/* tests/acceptance/encryption/assets/
 
             sudo pip3 install -r requirements-dev.txt
             behave --junit tests/acceptance/pam
+            behave --junit tests/acceptance/encryption/cryptor-module.feature -t=~na=python -k
       - name: Expose acceptance tests reports
         uses: actions/upload-artifact@v3
         if: always()

.pubnub.yml

@@ -1,5 +1,5 @@
 name: python
-version: 7.2.0
+version: 7.3.0
 schema: 1
 scm: github.com/pubnub/python
 sdks:
@@ -18,7 +18,7 @@ sdks:
         distributions:
           - distribution-type: library
             distribution-repository: package
-            package-name: pubnub-7.2.0
+            package-name: pubnub-7.3.0
             location: https://pypi.org/project/pubnub/
             supported-platforms:
               supported-operating-systems:
@@ -97,8 +97,8 @@ sdks:
           -
             distribution-type: library
             distribution-repository: git release
-            package-name: pubnub-7.2.0
-            location: https://github.com/pubnub/python/releases/download/7.2.0/pubnub-7.2.0.tar.gz
+            package-name: pubnub-7.3.0
+            location: https://github.com/pubnub/python/releases/download/v7.3.0/pubnub-7.3.0.tar.gz
             supported-platforms:
               supported-operating-systems:
                 Linux:
@@ -169,6 +169,13 @@ sdks:
                 license-url: https://github.com/aio-libs/aiohttp/blob/master/LICENSE.txt
                 is-required: Required
 changelog:
+  - date: 2023-10-16
+    version: v7.3.0
+    changes:
+      - type: feature
+        text: "Add crypto module that allows configure SDK to encrypt and decrypt messages."
+      - type: bug
+        text: "Improved security of crypto implementation by adding enhanced AES-CBC cryptor."
   - date: 2023-07-06
     version: 7.2.0
     changes:

CHANGELOG.md

@@ -1,3 +1,12 @@
+## v7.3.0
+October 16 2023
+
+#### Added
+- Add crypto module that allows configure SDK to encrypt and decrypt messages.
+
+#### Fixed
+- Improved security of crypto implementation by adding enhanced AES-CBC cryptor.
+
 ## 7.2.0
 July 06 2023
 

LICENSE

@@ -1,27 +1,29 @@
-PubNub Real-time Cloud-Hosted Push API and Push Notification Client Frameworks
-Copyright (c) 2013 PubNub Inc.
-http://www.pubnub.com/
-http://www.pubnub.com/terms
+PubNub Software Development Kit License Agreement
+Copyright © 2023 PubNub Inc. All rights reserved.
 
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
+Subject to the terms and conditions of the license, you are hereby granted
+a non-exclusive, worldwide, royalty-free license to (a) copy and modify
+the software in source code or binary form for use with the software services
+and interfaces provided by PubNub, and (b) redistribute unmodified copies
+of the software to third parties. The software may not be incorporated in
+or used to provide any product or service competitive with the products
+and services of PubNub.
 
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
+The above copyright notice and this license shall be included
+in or with all copies or substantial portions of the software.
 
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
+This license does not grant you permission to use the trade names, trademarks,
+service marks, or product names of PubNub, except as required for reasonable
+and customary use in describing the origin of the software and reproducing
+the content of this license.
 
-PubNub Real-time Cloud-Hosted Push API and Push Notification Client Frameworks
-Copyright (c) 2013 PubNub Inc.
-http://www.pubnub.com/
-http://www.pubnub.com/terms
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF
+ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO
+EVENT SHALL PUBNUB OR THE AUTHORS OR COPYRIGHT HOLDERS OF THE SOFTWARE BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
+CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+https://www.pubnub.com/
+https://www.pubnub.com/terms

examples/crypto_module.py

@@ -0,0 +1,50 @@
+from pubnub.pnconfiguration import PNConfiguration
+from pubnub.pubnub import PubNub
+from pubnub.crypto import AesCbcCryptoModule
+from Cryptodome.Cipher import AES
+
+my_cipher_key = 'myCipherKey'
+my_message = 'myMessage'
+
+# by default no configuration changes is needed
+config = PNConfiguration()
+config.uuid = 'myUUID'
+config.cipher_key = my_cipher_key
+pubnub = PubNub(config)
+
+# message will be encrypted the same way it was encrypted previously
+cbc_message = pubnub.crypto.encrypt(my_message)  # new way of using cryptographic module from pubnub
+decrypted = config.crypto.decrypt(my_cipher_key, cbc_message)
+assert decrypted == my_message
+
+# also no configuration changes is needed if you previously updated the cipher_mode to GCM
+config = PNConfiguration()
+config.uuid = 'myUUID'
+config.cipher_key = my_cipher_key
+config.cipher_mode = AES.MODE_GCM
+config.fallback_cipher_mode = AES.MODE_CBC
+pubnub = PubNub(config)
+
+# message will be encrypted the same way it was encrypted previously
+gcm_message = pubnub.crypto.encrypt(my_message)  # new way of using cryptographic module from pubnub
+decrypted = config.crypto.decrypt(my_cipher_key, gcm_message)
+assert decrypted == my_message
+
+# opt in to use crypto module with headers and improved entropy
+config = PNConfiguration()
+config.uuid = 'myUUID'
+config.cipher_key = my_cipher_key
+config.cipher_mode = AES.MODE_GCM
+config.fallback_cipher_mode = AES.MODE_CBC
+module = AesCbcCryptoModule(config)
+config.crypto_module = module
+pubnub = PubNub(config)
+message = pubnub.crypto.encrypt(my_message)
+# this encryption method is not compatible with previous crypto methods
+try:
+    decoded = config.crypto.decrypt(my_cipher_key, message)
+except Exception:
+    pass
+# but can be decrypted with new crypto module
+decrypted = pubnub.crypto.decrypt(message)
+assert decrypted == my_message

pubnub/crypto.py

@@ -1,11 +1,16 @@
 import hashlib
 import json
 import random
-from base64 import decodebytes, encodebytes
+import logging
 
-from pubnub.crypto_core import PubNubCrypto
+
+from base64 import decodebytes, encodebytes, b64decode, b64encode
 from Cryptodome.Cipher import AES
 from Cryptodome.Util.Padding import pad, unpad
+from pubnub.crypto_core import PubNubCrypto, PubNubCryptor, PubNubLegacyCryptor, PubNubAesCbcCryptor, CryptoHeader, \
+    CryptorPayload
+from pubnub.exceptions import PubNubException
+from typing import Union, Dict
 
 
 Initial16bytes = '0123456789012345'
@@ -80,9 +85,10 @@ def get_secret(self, key):
 
 
 class PubNubFileCrypto(PubNubCryptodome):
-    def encrypt(self, key, file):
+    def encrypt(self, key, file, use_random_iv=True):
+
         secret = self.get_secret(key)
-        initialization_vector = self.get_initialization_vector(use_random_iv=True)
+        initialization_vector = self.get_initialization_vector(use_random_iv)
         cipher = AES.new(bytes(secret[0:32], "utf-8"), self.mode, bytes(initialization_vector, 'utf-8'))
         initialization_vector = bytes(initialization_vector, 'utf-8')
 
@@ -92,9 +98,9 @@ def encrypt(self, key, file):
             initialization_vector=initialization_vector
         )
 
-    def decrypt(self, key, file):
+    def decrypt(self, key, file, use_random_iv=True):
         secret = self.get_secret(key)
-        initialization_vector, extracted_file = self.extract_random_iv(file, use_random_iv=True)
+        initialization_vector, extracted_file = self.extract_random_iv(file, use_random_iv)
         try:
             cipher = AES.new(bytes(secret[0:32], "utf-8"), self.mode, initialization_vector)
             result = unpad(cipher.decrypt(extracted_file), 16)
@@ -103,3 +109,163 @@ def decrypt(self, key, file):
             result = unpad(cipher.decrypt(extracted_file), 16)
 
         return result
+
+
+class PubNubCryptoModule(PubNubCrypto):
+    FALLBACK_CRYPTOR_ID: str = '0000'
+    cryptor_map = {}
+    default_cryptor_id: str
+
+    def __init__(self, cryptor_map: Dict[str, PubNubCryptor], default_cryptor: PubNubCryptor):
+        self.cryptor_map = cryptor_map
+        self.default_cryptor_id = default_cryptor.CRYPTOR_ID
+
+    def _validate_cryptor_id(self, cryptor_id: str) -> str:
+        cryptor_id = cryptor_id or self.default_cryptor_id
+
+        if len(cryptor_id) != 4:
+            logging.error(f'Malformed cryptor id: {cryptor_id}')
+            raise PubNubException('Malformed cryptor id')
+
+        if cryptor_id not in self.cryptor_map.keys():
+            logging.error(f'Unsupported cryptor: {cryptor_id}')
+            raise PubNubException('unknown cryptor error')
+        return cryptor_id
+
+    def _get_cryptor(self, cryptor_id):
+        if not cryptor_id or cryptor_id not in self.cryptor_map:
+            raise PubNubException('unknown cryptor error')
+        return self.cryptor_map[cryptor_id]
+
+    # encrypt string
+    def encrypt(self, message: str, cryptor_id: str = None) -> str:
+        if not len(message):
+            raise PubNubException('encryption error')
+        cryptor_id = self._validate_cryptor_id(cryptor_id)
+        data = message.encode('utf-8')
+        crypto_payload = self.cryptor_map[cryptor_id].encrypt(data)
+        header = self.encode_header(cryptor_id=cryptor_id, cryptor_data=crypto_payload['cryptor_data'])
+        return b64encode(header + crypto_payload['data']).decode()
+
+    def decrypt(self, message):
+        data = b64decode(message)
+        header = self.decode_header(data)
+        if header:
+            cryptor_id = header['cryptor_id']
+            payload = CryptorPayload(data=data[header['length']:], cryptor_data=header['cryptor_data'])
+        if not header:
+            cryptor_id = self.FALLBACK_CRYPTOR_ID
+            payload = CryptorPayload(data=data)
+
+        if not len(payload['data']):
+            raise PubNubException('decryption error')
+
+        if cryptor_id not in self.cryptor_map.keys():
+            raise PubNubException('unknown cryptor error')
+
+        message = self._get_cryptor(cryptor_id).decrypt(payload)
+        try:
+            return json.loads(message)
+        except Exception:
+            return message
+
+    def encrypt_file(self, file_data, cryptor_id: str = None):
+        if not len(file_data):
+            raise PubNubException('encryption error')
+        cryptor_id = self._validate_cryptor_id(cryptor_id)
+        crypto_payload = self.cryptor_map[cryptor_id].encrypt(file_data)
+        header = self.encode_header(cryptor_id=cryptor_id, cryptor_data=crypto_payload['cryptor_data'])
+        return header + crypto_payload['data']
+
+    def decrypt_file(self, file_data):
+        header = self.decode_header(file_data)
+        if header:
+            cryptor_id = header['cryptor_id']
+            payload = CryptorPayload(data=file_data[header['length']:], cryptor_data=header['cryptor_data'])
+        else:
+            cryptor_id = self.FALLBACK_CRYPTOR_ID
+            payload = CryptorPayload(data=file_data)
+
+        if not len(payload['data']):
+            raise PubNubException('decryption error')
+
+        if cryptor_id not in self.cryptor_map.keys():
+            raise PubNubException('unknown cryptor error')
+
+        return self._get_cryptor(cryptor_id).decrypt(payload, binary_mode=True)
+
+    def encode_header(self, cryptor_id: str = None, cryptor_data: any = None) -> str:
+        if cryptor_id == self.FALLBACK_CRYPTOR_ID:
+            return b''
+        if cryptor_data and len(cryptor_data) > 65535:
+            raise PubNubException('Cryptor data is too long')
+        cryptor_id = self._validate_cryptor_id(cryptor_id)
+
+        sentinel = b'PNED'
+        version = CryptoHeader.header_ver.to_bytes(1, byteorder='big')
+        crid = bytes(cryptor_id, 'utf-8')
+
+        if cryptor_data:
+            crd = cryptor_data
+            cryptor_data_len = len(cryptor_data)
+        else:
+            crd = b''
+            cryptor_data_len = 0
+
+        if cryptor_data_len < 255:
+            crlen = cryptor_data_len.to_bytes(1, byteorder='big')
+        else:
+            crlen = b'\xff' + cryptor_data_len.to_bytes(2, byteorder='big')
+        return sentinel + version + crid + crlen + crd
+
+    def decode_header(self, header: bytes) -> Union[None, CryptoHeader]:
+        try:
+            sentinel = header[:4]
+            if sentinel != b'PNED':
+                return False
+        except ValueError:
+            return False
+
+        try:
+            header_version = header[4]
+            if header_version > CryptoHeader.header_ver:
+                raise PubNubException('unknown cryptor error')
+
+            cryptor_id = header[5:9].decode()
+            crlen = header[9]
+            if crlen < 255:
+                cryptor_data = header[10: 10 + crlen]
+                hlen = 10 + crlen
+            else:
+                crlen = int(header[10:12].hex(), 16)
+                cryptor_data = header[12:12 + crlen]
+                hlen = 12 + crlen
+
+            return CryptoHeader(sentinel=sentinel, header_ver=header_version, cryptor_id=cryptor_id,
+                                cryptor_data=cryptor_data, length=hlen)
+        except IndexError:
+            raise PubNubException('decryption error')
+
+
+class LegacyCryptoModule(PubNubCryptoModule):
+    def __init__(self, config) -> None:
+        cryptor_map = {
+            PubNubLegacyCryptor.CRYPTOR_ID: PubNubLegacyCryptor(config.cipher_key,
+                                                                config.use_random_initialization_vector,
+                                                                config.cipher_mode,
+                                                                config.fallback_cipher_mode),
+            PubNubAesCbcCryptor.CRYPTOR_ID: PubNubAesCbcCryptor(config.cipher_key),
+        }
+        super().__init__(cryptor_map, PubNubLegacyCryptor)
+
+
+class AesCbcCryptoModule(PubNubCryptoModule):
+    def __init__(self, config) -> None:
+        cryptor_map = {
+            PubNubLegacyCryptor.CRYPTOR_ID: PubNubLegacyCryptor(config.cipher_key,
+                                                                config.use_random_initialization_vector,
+                                                                config.cipher_mode,
+                                                                config.fallback_cipher_mode),
+            PubNubAesCbcCryptor.CRYPTOR_ID: PubNubAesCbcCryptor(config.cipher_key),
+        }
+        super().__init__(cryptor_map, PubNubAesCbcCryptor)