Crash loading pydantic_core on Android

[caarmen]

Describe the bug

A full bug reproduction project is here: https://github.com/caarmen/beeware-helloworld

The briefcase build android command doesn't seem to package the pydantic_core library inside the app, which causes app crashes at runtime, on app startup.

Steps to reproduce

Clone the bug reproduction project: https://github.com/caarmen/beeware-helloworld

• Install requirements: pip install -r requirements.txt
• Go into the helloworld subdirectory: cd helloworld
• Run the app on your local computer: briefcase dev
• Click on the "Fetch!" button
  • Verify that the screen displays "A New Hope was released in 1977-05-25"
    
• Run the app on an Android device/emulator:
  • briefcase create android
  • briefcase build android
  • Start and Android emulator, or connect an Android device to your computer.
  • adb install -r build/helloworld/android/gradle/app/build/outputs/apk/debug/app-debug.apk
  • Start to capture android logs: adb lolcat -v threadtime > /tmp/logcat.txt 2>&1 &
  • Launch the app.
  • Expected behavior: the app ui appears.
  • Actual behavior: the app crashes. The following appears in /tmp/logcat.txt:

    11-18 22:46:32.665  5171  5171 D AndroidRuntime: Shutting down VM
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: FATAL EXCEPTION: main
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: Process: com.example.helloworld, PID: 5171
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: java.lang.RuntimeException: Unable to start activity ComponentInfo{com.example.helloworld/org.beeware.android.MainActivity}: com.chaquo.python.PyException: ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3676)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3813)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:101)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2308)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.os.Handler.dispatchMessage(Handler.java:106)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.os.Looper.loopOnce(Looper.java:201)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.os.Looper.loop(Looper.java:288)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.app.ActivityThread.main(ActivityThread.java:7898)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at java.lang.reflect.Method.invoke(Native Method)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:936)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: Caused by: com.chaquo.python.PyException: ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.java.chaquopy.import_override(import.pxi:20)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.java.chaquopy.import_override(import.pxi:60)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.pydantic_core.<module>(__init__.py:6)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.java.chaquopy.import_override(import.pxi:60)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.pydantic.main.<module>(main.py:11)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.java.android.importer.exec_module(importer.py:554)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.java.android.importer.exec_module(importer.py:606)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.importlib.import_module(__init__.py:126)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.pydantic.__getattr__(__init__.py:372)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.java.chaquopy.import_override(import.pxi:60)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.graphql_client.async_base_client.<module>(async_base_client.py:9)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.java.chaquopy.import_override(import.pxi:60)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.graphql_client.<module>(__init__.py:3)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.java.chaquopy.import_override(import.pxi:60)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.helloworld.app.<module>(app.py:7)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.java.chaquopy.import_override(import.pxi:60)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.__main__.<module>(__main__.py:1)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.runpy._run_code(<frozen runpy>:88)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.runpy._run_module_code(<frozen runpy>:98)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.runpy.run_module(<frozen runpy>:226)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.chaquopy_java.call(chaquopy_java.pyx:354)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at <python>.chaquopy_java.Java_com_chaquo_python_PyObject_callAttrThrowsNative(chaquopy_java.pyx:326)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at com.chaquo.python.PyObject.callAttrThrowsNative(Native Method)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at com.chaquo.python.PyObject.callAttrThrows(PyObject.java:232)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at com.chaquo.python.PyObject.callAttr(PyObject.java:221)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at org.beeware.android.MainActivity.onCreate(MainActivity.java:85)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.app.Activity.performCreate(Activity.java:8290)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.app.Activity.performCreate(Activity.java:8269)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1384)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3657)
    11-18 22:46:32.667  5171  5171 E AndroidRuntime: 	... 12 more
    

Expected behavior

Expected behavior: the app ui appears.

Screenshots

This is the result on an Android device (Google android emulator here):

Environment

• Build machine: MacOS
• Operating system: Sonoma 14.1, Intel chip.
• Python version: 3.11.6
• Software versions:

  briefcase==0.3.16
  toga==0.4.0
  httpx==0.25.1
  ariadne-codegen==0.10.0
  websockets==12.0
  pydantic==2.5.1
  pydantic_core==2.14.3
  

• Android devices tested:
  • Genymotion Cloud emulator, Android 11, x86_64.
  • Google Android emulator: Pixel 3, Android 11, x86_64.
  • Samsung Galaxy S10+, Android 13.

Logs

briefcase.2023_11_18-23_24_52.create.log
briefcase.2023_11_18-23_25_54.build.log

Additional context

Related:

• Package request: pydantic-core chaquo/chaquopy#995

There's a suggested workaround to use python 3.8 and a 1.x version of pydantic. I didn't try with python 3.8, but the graphql library seems to rely on pydantic 2.

Did a test (still on python 3.11) to downgrade pydantic (and remove the dependency to pydantic_core). The app installs and works fine like that.

Would be nice to have pydantic 2 support though. 🙏🏻

If this is not a problem in briefcase, but in chaquopy, don't hesitate to close this issue :)

[freakboy3742]

Thanks for the report.

The underlying issue here is that Chaquopy doesn't provide a package for pydantic. As of V2, pydantic uses a Rust implementation, which is undeniably faster, but is platform specific, and therefore needs to be compiled specifically for Android (and, for that matter, any other platform - but they provide pre-compiled wheels for Windows, Linux and macOS, so it's not as much of an issue).

Compiling Rust binary packages for Android is possible (as in, the Rust platform tooling exists), but I don't believe the Chaquopy tooling for building binary wheels currently supports Rust packages (@mhsmith can correct me here if I'm wrong on this front). You can check the list of currently supported Android packages here and here; You've already linked the open request to add a recipe for pydantic-core.

It looks like you've discovered the workaround as well - downgrade Pydantic to v1. This makes it a pure-python package, removing the problem. Of course, as you note, you're then not using Pydantic 2 :-)

The thing that is weird from your stack trace is that the build is succeeding, beacuse it is happily installing the macOS binary wheel for pydantic-core... even though it's an Android build. Ordinarily, if you try to install a binary package Chaquopy doesn't support, it raises an error because the dependency can't be satisfied. However, it's satisfying the requirement with a macOS package, so the build succeeds, but then at runtime, there's no Android-compatible binary module present, and the app crashes.

My guess is that the issue is that your explicit dependency is on pydantic, which is a cross-platform wheel; but pydantic-core is a dependency, and resolving that dependency is causing a macOS specific wheel to be selected (because you're running the build on macOS, and for most Python installs, the build machine is the same as the deployment machine). This suggests to me that there's a bug in Chaquopy's dependency resolver; but I'll keep this issue open in case Briefcase is contributing to the problem somehow.

[mhsmith]

Compiling Rust binary packages for Android is possible (as in, the Rust platform tooling exists), but I don't believe the Chaquopy tooling for building binary wheels currently supports Rust packages

That's correct: I've built one Rust package in the past (tokenizers), but we haven't yet added generalized Rust support to the package build tool.

The thing that is weird from your stack trace is that the build is succeeding, beacuse it is happily installing the macOS binary wheel for pydantic-core... even though it's an Android build.

Relevant section of the log:

           > Task :app:generateDebugPythonRequirements                                                                                                             subprocess.py:685
           Chaquopy: Installing for arm64-v8a                                                                                                                      subprocess.py:685
           Looking in indexes: https://pypi.org/simple, https://chaquo.com/pypi-7.0, https://chaquo.com/pypi-13.1                                                  subprocess.py:685
           Collecting httpx==0.25.1                                                                                                                                subprocess.py:685
             Using cached httpx-0.25.1-py3-none-any.whl (75 kB)                                                                                                    subprocess.py:685
[23:25:34] Collecting pydantic==2.5.1                                                                                                                              subprocess.py:685
             Using cached pydantic-2.5.1-py3-none-any.whl (381 kB)                                                                                                 subprocess.py:685
[23:25:36] Processing                                                                                                                                              subprocess.py:685
           /Users/calvarez/Library/Caches/chaquopy/pip/wheels/1d/34/7f/10cb7aee78f14a7b936afe80adf1df7a63a70a549ce8b40ce1/pydantic_core-2.14.3-cp311-cp311-macosx_                  
           10_7_x86_64.whl     

Since it's taking the wheel from pip's wheel cache, rather than its HTTP cache, I think what happened is that it built a macOS wheel locally on a previous attempt, which was only possible because the Rust tools were installed on the build machine. If I try to install pydantic on a machine without the Rust tools, then it fails as expected:

Collecting pydantic-core==2.14.3
  Downloading pydantic_core-2.14.3.tar.gz (359 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 /Users/msmith/git/chaquo/chaquopy/server/pypi/pkgtest/app/build/generated/python/bp/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/2t/th8vxc813_gcch7lkh1sbbmc0000gp/T/tmpom5waxvm
         cwd: /private/var/folders/2t/th8vxc813_gcch7lkh1sbbmc0000gp/T/pip-install-4c6gi9f6/pydantic-core
    Complete output (6 lines):
    
    Cargo, the Rust package manager, is not installed or is not on PATH.
    This package requires Rust and Cargo to compile extensions. Install it through
    the system's package manager or via https://rustup.rs/
    
    Checking for Rust toolchain....
    ----------------------------------------
ERROR: Failed to install pydantic-core==2.14.3 from https://files.pythonhosted.org/packages/4c/ee/b3479b31f47226bae5d9033761971bec215774a6078ce08e8618d6381470/pydantic_core-2.14.3.tar.gz#sha256=3ad083df8fe342d4d8d00cc1d3c1a23f0dc84fce416eb301e69f1ddbbe124d3f (from pydantic).

In the absence of any general way to tell a PEP 517 backend to disable native compilation, I don't think there's much we can do about this. So I'll close this issue, and let's keep any futher discussion in the Chaquopy issue linked above.

[Jzhenli]

Hi, what's the status for support lib implemented with rust on the android platform? @freakboy3742 @mhsmith

[mhsmith]

For pydantic specifically, see chaquo/chaquopy#1017. For Rust in general, see chaquo/chaquopy#1030.