Using nuitka to distribute migration tool

[monchin]

Hello, thank you for this convenient tool.

Here is my situation: we sell our productions each with a server to our customers. The server has no python environment, and our softwares are distributed standalone to get rid of python environment dependency by nuitka. Now I want to use alembic to migrate db when we need to upgrade our softwares, so I write

from alembic.config import Config
from alembic import command
from alembic.runtime.environment import EnvironmentContext
from pathlib import Path
import sys
import os
from alembic.script import ScriptDirectory
from dotenv import load_dotenv

IS_NUITKA_COMPILED: Final = True if "__compiled__" in globals() else False
ORIG_POINT = (
    Path(sys.argv[0]).absolute() if IS_NUITKA_COMPILED else Path(__file__).absolute()
)
PRJ_PATH = ORIG_POINT.parent.parent

from my_proj.sql.db import engine
ALEMBIC_INI_PATH:Final = PRJ_PATH/"deploy"/"alembic.ini"
config = Config(ALEMBIC_INI_PATH)
script = ScriptDirectory.from_config(config)
os.chdir(PRJ_PATH/"deploy")
sys.path.append(str(PRJ_PATH/"deploy"/"alembic"/"versions"))

try:
    with EnvironmentContext(config, script, as_sql=False, StartingRev=None, tag=None) as env_context:
        env_context.configure(connection=engine.connect(), target_metadata=None)
        target_revision = "abcdef"
        command.upgrade(config, target_revision)
except KeyError:
    pass

If I use python to run this script, it would work well; but if I use nuitka to distribute it into a binary file, upgrading would not be executed. I tried to dig in the source code, and found that no matter how I run it, python scripts would be loaded when running script = ScriptDirectory.from_config(config). And if I use python to run it, when executing command.upgrade,
```python

    with EnvironmentContext(
        config,
        script,
        fn=upgrade,
        as_sql=sql,
        starting_rev=starting_rev,
        destination_rev=revision,
        tag=tag,
    ):
        script.run_env()

after script.run_env() which would only load_python_file by loading env.py, it will run _load_revisions to load all scripts in my alembic/versions directory 2nd time. However, if I use nuitka to distribute it into binary form, and then run it, after script.run_env() nothing would be run, even I add print("run_env over") after script.run_env(), nothing would be printed.

I'm curious about why _loading_revisions would be run the 2nd time after script.run_env() with python, how it works, and how to make it run properly in nuitka distributions.

[zzzeek]

Alembic has to load all the revision files every time it runs and has to walk along revisions, since that's how the chain of revisions are represented. the loading occurs as below

https://github.com/sqlalchemy/alembic/blob/e09ef30d879daa00895098d0fca2a2a58065292b/alembic/script/base.py#L130C1-L158C1

and it loads Python files. it has a mode where it can be made to load just .pyc alone but you still need a python interpreter for that. if you were trying to do all this without a python interpreter you'd need to highly modify how alembic loads files like this and there is not a trivial solution to this use case as it stands.

[CaselIT]

Maybe you will have more success using the offline mode using --sql?
This generates an SQL script given a start - end migration that you can run directly.

So your packaging would be something like this:

• define a final revision
• find the previous revision
• run alembic upgrade --sql previous_rev:final_rev
• save the migration sql file somewhere
• write a script that loads that file and applies the migration sql on the db

[monchin]

Thanks! It works