Thanks very much. I'm currently working on updating Chaquopy to Python 3.10 (#624), so I'll have a look at updating Cryptography after that. The cleanest solution would probably be to take the existing Rust support from the tokenizers package and merge that into the main build-wheel tool, along with any necessary additions from this issue.

If anyone else needs a new version of this package, please click the thumbs up button above, and post a comment explaining why you need it.

First of all, I would like to thank you for creating and sharing Chaquopy. I am truly impressed how easy it is to run Python code on Android using this SDK.
I am currently using Chaquopy (via BeeWare / Briefcase) to develop an Android app in Python that needs to encrypt and decrypt data using cryptographic primitives which are only available in recent versions of the Cryptography package. Therefore, I am interested in getting new versions of the Cryptography package (preferably the latest version) to work with Chaquopy. So far, all my attempts to install version 39 have been without success.
Please let me know if there is an (easy) way to make it work or if a new version of Cryptography for Chaquopy is to be expected soon.

Would version 36.0.2 be new enough for you? If so, you can try using the wheels built by @emanuele-f at https://github.com/emanuele-f/chaquopy-wheels.

If you need a newer version than that, please let me know exactly which features you need it for.

Unfortunately, version 36.0.2 is not suitable for me, given that I have to use AESSIV which was introduced with version 37 (see https://cryptography.io/en/latest/hazmat/primitives/aead/#cryptography.hazmat.primitives.ciphers.aead.AESSIV). The performance improvement of ChaCha20Poly1305 introduced with version 39 would be a bonus, but is not strictly necessary.

I will probably look into building cryptography 38 soon, but I provide no guarantee. In the long term, it would be great to have it as an official supported package.

Having Cryptography version 38 work with Chaquopy would be great.

Just out of curiosity: Is there any particular reason for choosing version 38 rather than the latest version?

I'm trying to run the latest mitmproxy release, which strictly dependends on version 38 https://github.com/mitmproxy/mitmproxy/blob/9.0.1/setup.py#L79

Hi @mhsmith, following the new instructions at https://github.com/chaquo/chaquopy/tree/master/server/pypi to build for python 3.10, I got the same error as before (e.g. see below for PyYAML). I think the problem is that python 3.8 must be used for the conda env (conda create -n build-wheel python=3.8) not python 3.10.

Building wheels for collected packages: PyYAML
  Building wheel for PyYAML (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [565 lines of output]
      In file included from ext/_yaml.c:271:
      ext/_yaml.h:10: warning: "PyString_CheckExact" redefined
         10 | #define PyString_CheckExact PyBytes_CheckExact
            |
      ext/_yaml.c:139: note: this is the location of the previous definition
        139 |   #define PyString_CheckExact          PyUnicode_CheckExact

After switching to python 3.8 in conda, pip install -r server/pypi/requirements.txt still fails. The problem seem to be in the PyYAML version and pip version. I've changed the requirements as follows:

-pip==19.2.3
-setuptools==46.4.0
-wheel==0.33.6
+#pip==19.2.3
+#setuptools==46.4.0
+#wheel==0.33.6
 
 # Other direct requirements
 Jinja2==2.10
 jsonschema==2.6.0
 pyelftools==0.24
-PyYAML==3.12
+PyYAML==3.13

and now installing them works good. I've then tried to build the brotli package just to see if things are set up properly, but it fails:

./build-wheel.py --python 3.10 --abi x86_64 brotli

build-wheel.py: mkdir -p /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel
build-wheel.py: unzip -d /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel -q /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/src/Brotli-1.0.7-cp310-cp310-linux_x86_64.whl
build-wheel.py: Processing native binaries
build-wheel.py: mv /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.cpython-310-x86_64-linux-gnu.so /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.so
build-wheel.py: chmod +w /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.so
build-wheel.py: /home/emanuele/Android/Sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-strip --strip-unneeded /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.so
build-wheel.py: Error: /home/emanuele/src/chaquopy/server/pypi/packages/brotli/build/1.0.7/cp310-cp310-android_21_x86_64/fix_wheel/_brotli.so is linked against unknown library 'libm.so.6'

Kindly request your help to proceed

Please post the full logs for both of those failures: you can attach them as files if they're too large to paste.

And please clarify the following:

• The first failure, with PyYAML: did that happen after the following sequence of commands? Or something different?

  conda create -n build-wheel python=3.10
  conda activate build-wheel
  pip install -r server/pypi/requirements.txt
  

• The second failure, with brotli: did you create that environment with Python 3.8 or 3.10?
  • If 3.8, it's apparently still managed to find a python3.10 executable somewhere to do the build, so where did that come from (which python3.10)?
  • And does the brotli build work with Python 3.8?

I will redo on a clean environment and post the full logs. Do you confirm that to build for python 3.10 I have to use conda create -n build-wheel python=3.10 as of the docs?

Any python3.10 executable on the PATH should be fine: conda is just a convenient way of getting it.

Mmm, but some tools that are used to build the wheels does not seem to support python 3.10 (for example, yaml/pyyaml#416), so I cannot see how this can work, this is why I thought that some step was wrong, so tried using python 3.8 to run build-wheel.

Anyway, here are the steps following the server/pypi instructions, which lead to the error above (using latest chaquopy master):

cd chaquopy
mkdir build-clean
cd build-clean
mkdir -p sysroot overlay workdir
sudo mount -t overlay overlay -o lowerdir=/,upperdir=./sysroot,workdir=./workdir ./overlay
sudo arch-chroot ./overlay

cd /home/emanuele/src/chaquopy
mkdir -p maven/com/chaquo/python/target/3.10.6-1-1
cd maven/com/chaquo/python/target/3.10.6-1-1
wget https://repo.maven.apache.org/maven2/com/chaquo/python/target/3.10.6-1/target-3.10.6-1-x86_64.zip
conda create -n build-clean python=3.10
conda activate build-clean

# gives error, see attached `pip_install_error.txt`
pip install -r server/pypi/requirements.txt

pip_install_error.txt

The second failure, with brotli: did you create that environment with Python 3.8 or 3.10

It happened using python 3.8 in the conda env, but using --python 3.10 in build_sheel.py. I'm on archlinux and python3.10 is in my PATH. I thought build_wheel.py used the python version from the maven folder I've downloaded.

And does the brotli build work with Python 3.8

No. Following server/pypi instructions, it stills fails at the pip install step. Here is my commands:

mkdir -p maven/com/chaquo/python/target/3.8.16-0
cd maven/com/chaquo/python/target/3.8.16-0
wget https://repo.maven.apache.org/maven2/com/chaquo/python/target/3.8.16-0/target-3.8.16-0-x86_64.zip

cd /home/emanuele/src/chaquopy
conda create -n build-wheel python=3.8
conda activate build-wheel

# gives error, see attached `pip_38_install_error.txt`
pip install -r server/pypi/requirements.txt

In essence, without modifying the requirements.txt file, I cannot even reach the build step.

pip_38_install_error.txt

python --version: Python 3.8.16
which python: /opt/miniconda3/envs/build-wheel/bin/python
pip --version pip 23.0.1 from /opt/miniconda3/envs/build-wheel/lib/python3.8/site-packages/pip (python 3.8)

I couldn't reproduce the PyYAML problem at first, because it looks like it falls back to a pure-Python mode when the build machine doesn't have yaml.h available. Anyway, I've now updated it to a version which has manylinux wheels on PyPI, so that shouldn't be a problem anymore. And I've updated several other requirements which also weren't compatible with Python 3.10. Thanks for reporting that.

So it should now be possible to run build-wheel with Python 3.8, 3.9 or 3.10. For Python packages it will then launch a subprocess to do the actual build, with a version matching your --python option.

Hi Malcolm, thanks for the fixes. I confirm that building for python 3.10 works correctly now.

I can successfully build the new cryptography wheels, but I'm running into an issue with OpenSSL at runtime. The following error is thrown after cryptography is imported:

com.chaquo.python.PyException: InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ([_OpenSSLErrorWithText(code=310378599, lib=37, reason=103, reason_text=b'error:12800067:DSO support routines::could not load the shared library'), _OpenSSLErrorWithText(code=310378599, lib=37, reason=103, reason_text=b'error:12800067:DSO support routines::could not load the shared library'), _OpenSSLErrorWithText(code=126615813, lib=15, reason=786693, reason_text=b'error:078C0105:common libcrypto routines::init fail')])

Here is the full log: openssl_error.txt

This is the wheel I've built that throws the above error: cryptography-38.0.4-0-cp310-cp310-android_21_x86.zip (rename to .whl). It uses cffi 1.15.1.

The error is thrown here: binding.py#L176. This is the relevant issue I've found: openssl/openssl#19067 . It seems like OpenSSL is trying to load the "legacy" provider as a shared library via OSSL_PROVIDER_load, which fails (maybe because of the different libcrypto_chaquopy.so name?).

How can we address this?

Can you create a fork of this repository and push the files you used to build this wheel?

It would also be useful to include a README file which includes the instructions above, updated as necessary to reflect how you built this wheel.

I've bypassed the above error by removing the _openssl_assert as explained in pyca/cryptography#7358 . This is not optimal as it disables some ciphers, but for my use case it's fine.

I've uploaded the new wheels to https://github.com/emanuele-f/chaquopy-wheels . @newfix you can also find there the cryptography 39.0.2 modules cooked just for you ;) Please give it a try an let me know

I've quickly verified that the 38.0.4 wheels work correctly on x86 and arm64 in my PCAPdroid-mitm plugin. On armv7, however, I get ImportError: dlopen failed: cannot locate symbol "decode_eht_entry", seems related to libunwind (full log attached). @mhsmith tomorrow I will clean things up and post clean instructions, hopefully you will be able to reproduce the builds. If you have suggestions for the armv7 issue please let me know.

I've bypassed the above error by removing the _openssl_assert as explained in pyca/cryptography#7358 . This is not optimal as it disables some ciphers, but for my use case it's fine.

Thanks: there are some related notes about the OpenSSL legacy provider in #688.

I've uploaded the new wheels to https://github.com/emanuele-f/chaquopy-wheels . @newfix you can also find there the cryptography 39.0.2 modules cooked just for you ;) Please give it a try an let me know

Thank you very much for building this version!

I managed to install it. However, at runtime I get the following error:
ImportError: dlopen failed: library "libpython3.10.so" not found: needed by .../hazmat/bindings/_rust.so in namespace classloader-namespace

Is this maybe because the briefcase toolchain I am using to build my application is referring to a different version (3.8) of python?

Any advice regarding how I could overcome the above error would be appreciated.

Hi, yeah, all these wheel are only for python 3.10. You should consider upgrading your project to 3.10. I could try building for 3.8, but only after I fix the issues with armv7

@newfix: I'm not sure how you managed to install cp310 wheels into a Python 3.8 app. If you renamed them to cp38, that won't work, because different Python versions are not binary compatible.

When using Briefcase, the Python version used to run briefcase will determine the Python version of the app. So to use these wheels, you'll need to create a Python 3.10 environment, install Briefcase into it, then rerun briefcase create android.

Thank you for the feedback.

I did not rename the whl file. I just referenced it from my pyproject.toml file. After modifying my build.gradle file to restrict it to building the "arm64-v8a" abi I managed to build my app incl. the cryptography package. However, at runtime I am getting the error reported above.

I will set up a Python 3.10 environment as you suggested and try again.

Using Python 3.10 I got one step further, but now I am getting the following error:

RuntimeError: OpenSSL 3.0's legacy provider failed to load. This is a fatal error by default, but cryptography supports running without legacy algorithms by setting the environment variable CRYPTOGRAPHY_OPENSSL_NO_LEGACY. If you did not expect this error, you have likely made a mistake with your OpenSSL configuration.

I tried adding the following code snippet, but it did not resolve the issue:

import os
os.environ["CRYPTOGRAPHY_OPENSSL_NO_LEGACY"] = "1"

Any idea how I could solve this?

I just tried version 38 of the cryptography package and it worked 😃

So the issue with the legacy algorithms only exists for version 39.

My mistake, should be fixed in the new cryptography 39 wheels I've just uploaded

Now, version 39 is also working fine 😃

@emanuele-f , @mhsmith : Thank you so much for your great work, for sharing it with others and for the support you are providing!

Happy to share my work and save other people time :)

Pre-built wheels available at https://github.com/emanuele-f/chaquopy-wheels

Instructions and patches available at https://github.com/emanuele-f/chaquopy/blob/cryptography-wheel/README_cryptography_39.md

@mhsmith regarding the issue with armv7, the problem is that the libpython that I build is linked against libunwind.a, whereas the chaquopy libpython is not.

$READELF -Wa armeabi-v7a/sysroot/usr/lib/libpython3.10.so | grep decode_eht_entry
002df850  0002cc16 R_ARM_JUMP_SLOT        002d4989   decode_eht_entry
   716: 002d4989    72 FUNC    GLOBAL DEFAULT    11 decode_eht_entry
 20136: 002d4989    72 FUNC    GLOBAL DEFAULT    11 decode_eht_entry

$READELF -Wa ./requirements/chaquopy/lib/libpython3.10.so | grep decode_eht_entry

$READELF -Wa ./fix_wheel/cryptography/hazmat/bindings/_rust.so | grep decode_eht_entry
0016e838  00008516 R_ARM_JUMP_SLOT        00000000   decode_eht_entry
   133: 00000000     0 FUNC    GLOBAL DEFAULT   UND decode_eht_entry

It seems like I'm forced to link against libunwind.a (Sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/libunwind.a), otherwise armv7a-linux-androideabi21-clang fails to compile even a simple int main() {} source, failing with ld: error: unable to find library -lunwind. Do you know how to prevent armv7a-linux-androideabi21-clang to try to link with -lunwind? I'm using -Wl,--exclude-libs,libunwind.a in the LDFLAGS but it has no effect

Edit: small correction, as explained here, --exclude-libs does not prevent from linking but rather it gives symbols an hidden visibility

Edit2: I was able to workaround this issue by setting the -Wl,--exclude-libs,libunwind.a flag directly in the LDSHARED env and rebuilding python. Now the symbol decode_eht_entry is properly marked as local and the built wheel works correctly on armv7a!

$READELF -Wa libpython3.10.so | grep decode_eht_entry
 15390: 002d43a1    72 FUNC    LOCAL  DEFAULT    13 decode_eht_entry

I've upload the new armv7a wheels. Updating the cryptography build instructions.

Edit3: I've just realized that I mistyped LDFLAGS as LD_FLAGS 🤦‍♂️. This also fixes the issue

Thanks very much.

Hello,
I am currently trying to build a version of tokenizer (tokenizers!=0.11.3,<0.14,>=0.11.1) in order to use upper transformer version.
As i saw on different issue it needs rust too.
Do you think it is possible to do it with the information of this issue and the tutorial for building package (https://github.com/chaquo/chaquopy/tree/master/server/pypi) ?

Yes, it should be possible. In fact, the tokenizers recipe already has a patch for setting up the Rust toolchain, though I haven't tested it with the current version of the build tool.

See #1030 for current notes on Rust.