Alembic not loading config member

[MarcoRocco]

We're working on a project which uses Postgresql as a data source which is then managed/initiated using SqlAlchemy to create models, and Alembic for migrations.

The issue is that Alembic cannot auto-generate any new migrations, nor can it upgrade the database to the latest iteration when using the Alembic command. When done programmatically the database version upgrades fine, but we're unable to generate new revisions. There are multiple errors that can show up depending on minor changes in alembics env.py script but essentially hey come down to issues with contexts config member.

Here is the folder structure:

my-project
│   alembic.ini
│
└───alembic
│   │   env.py
│   │   README
│   │   script.py.mako
│   │
│   └───versions
│       │   95ba8c1c3126_some_revision.py
│       │   43700c722f83_other_revision.py
│       │   ...

here are the relevant information from alembic.ini (everything else is default)

[alembic]
script_location = alembic
.
.
.
prepend_sys_path = .
.
.
.

# set this in env.py so we don't need to commit the database password
sqlalchemy.url = NULL

and here is env.py:

from logging.config import fileConfig

from sqlalchemy import engine_from_config
from sqlalchemy import pool

from alembic import context


# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config

# Interpret the config file for Python logging.
# This line sets up loggers basically.
fileConfig(config.config_file_name, disable_existing_loggers=False)

# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata

from core.database.database import Base, get_database_url

config.set_main_option('sqlalchemy.url', get_database_url())
target_metadata = Base.metadata

# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.


def run_migrations_offline():
    """Run migrations in 'offline' mode.

    This configures the context with just a URL
    and not an Engine, though an Engine is acceptable
    here as well.  By skipping the Engine creation
    we don't even need a DBAPI to be available.

    Calls to context.execute() here emit the given string to the
    script output.

    """
    url = config.get_main_option("sqlalchemy.url")
    context.configure(
        url=url,
        target_metadata=target_metadata,
        literal_binds=True,
        dialect_opts={"paramstyle": "named"},
    )

    with context.begin_transaction():
        context.run_migrations()


def run_migrations_online():
    """Run migrations in 'online' mode.

    In this scenario we need to create an Engine
    and associate a connection with the context.

    """
    connectable = engine_from_config(
        config.get_section(config.config_ini_section),
        prefix="sqlalchemy.",
        poolclass=pool.NullPool,
    )

    with connectable.connect() as connection:
        context.configure(
            connection=connection, target_metadata=target_metadata
        )

        with context.begin_transaction():
            context.run_migrations()


if context.is_offline_mode():
    run_migrations_offline()
else:
    run_migrations_online()

this is the error I receive when running something like
alembic revision -m "Add maintenances" --autogenerate from the same directory where 'alembic.ini' is located
error:

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
Traceback (most recent call last):
  File "/home/marco_rocco/.local/lib/python3.9/site-packages/alembic/command.py", line 212, in revision
    script_directory.run_env()
  File "/home/marco_rocco/.local/lib/python3.9/site-packages/alembic/script/base.py", line 490, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/home/marco_rocco/.local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
    module = load_module_py(module_id, path)
  File "/home/marco_rocco/.local/lib/python3.9/site-packages/alembic/util/compat.py", line 184, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "alembic/env.py", line 83, in <module>
    if context.is_offline_mode():
  File "<string>", line 8, in is_offline_mode
AttributeError: 'NoneType' object has no attribute 'is_offline_mode'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/marco_rocco/.local/bin/alembic", line 8, in <module>
    sys.exit(main())
  File "/home/marco_rocco/.local/lib/python3.9/site-packages/alembic/config.py", line 559, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/home/marco_rocco/.local/lib/python3.9/site-packages/alembic/config.py", line 553, in main
    self.run_cmd(cfg, options)
  File "/home/marco_rocco/.local/lib/python3.9/site-packages/alembic/config.py", line 530, in run_cmd
    fn(
  File "/home/marco_rocco/.local/lib/python3.9/site-packages/alembic/command.py", line 212, in revision
    script_directory.run_env()
  File "/home/marco_rocco/.local/lib/python3.9/site-packages/alembic/runtime/environment.py", line 110, in __exit__
    self._remove_proxy()
  File "/home/marco_rocco/.local/lib/python3.9/site-packages/alembic/util/langhelpers.py", line 50, in _remove_proxy
    del globals_[attr_name]
KeyError: 'config'

[zzzeek]

this seems like an environmental problem of some kind. when env.py is loaded, "from alembic import context" will give you the real context object.

questions:

1. Was there some point at which autogenerate stopped working?
2. can you try removing your imports, "from core.database.database import Base, get_database_url", and just set up a blank MetaData(), set your URL as hardcoded config.set_main_option('sqlalchemy.url', "sqlite:///file.db") , and see if that changes the error?
3. all other commands work?
4. version of alembic in use?

[MarcoRocco]

1. I'm unable to tell, this is the problem
2. removing those imports worked, looks like something is wrong with those, what would you suggest to try and debug that?
3. no, they only work programmatically, for example upgrade head is called on process startup
4. 1.6.5

[zzzeek]

OK so something in your imports is doing something with alembic, im not sure what, but you want to make sure that when all of those things go through, nothing from alembic has been imported. This is not necessarily something that's required in general but in this case it seems like something in your imports is affecting the environment.

another approach is to iteratively remove things from the stuff being imported, that is, intentionally go into the code being imported and just comment things out (you would restore from last commit to revert). that way you can narrow down using a divide-and-conquer approach to find what makes it suddenly not work.

alternatively if your environment is open source I would gladly try it out because this seems like something alembic should maybe guard against, whatever it is that's happening.

[MarcoRocco]

Thanks for the answer, thanks to your pointers I was able to find the cause of the issue. In database.py we were defining the Base model which was used by env.py, but also we were programmatically running alembic upgrade head using alembic's commands. I've now moved the commands to upgrade head into a different script which is executed after the fact and everything is working fine. I hope that is enough information for you but I can provide more if required.

[zzzeek]

OK actually running alembic commands in the scope of the imports within env.py is clearly pretty catastrophic, I don't have any immediate ideas on how to guard against that without a lot of complexity, so in any case, glad you solved the problem!