Is there a recommended way to allow "known" differences?

[mgaitan]

I work with a system that has many differences between the database schema and what the models declare. One of the reasons is that the database comes from a legacy system and the models are not 100% representative of the definition of its tables and, on the other hand, the migrations (which fortunately are sporadic) so far are done manually and can introduce small divergences.

A typical example is an index that exists in the base but is not in the model, then Alembic proposes an upgrade with a drop_index operation.

Now we want to start improving and implementing alembic to autogenerate the migrations, but given the situation we need to accept a "0 point" of known differences that should be ignored for the time being, and then gradually correct what is necessary so that the differences do not exist.

I started to explore an approach based on process_revision_directives. From an initial migration, I generate a file "know_diff.json" with the code of every detected operation. Then any migration should keep only the operations that are not in that file.
Something like this

def process_revision_directives(context, revision, directives):
    script = directives[0]
    diff_file = Path(__file__).parent / "known_diff.json"
    
    if os.environ.get("UPDATE_KNOWN_DIFF"):
        logger.info(f"[SPECIAL CASE] Upgrading {diff_file}")
        known_diff = {
            "upgrade": get_lines_of_code_for_ops(script.upgrade_ops.ops),  
            "downgrade": get_lines_of_code_for_all_the_ops(script.downgrade_ops.ops),  
        }
        diff_file.write_text(json.dumps(known_diff, indent=2))
    else:
        known_diff = json.loads(diff_file.read_text())

    script.upgrade_ops.ops[:] = [
        op for op in script.upgrade_ops.ops if get_lines_of_code_for_ops([op]) not in known_diff["upgrade"]
    ]
    script.downgrade_ops.ops[:] = [
        op for op in script.downgrade_ops.ops if get_lines_of_code_for_ops([op]) not in known_diff["downgrade"]
    ]

So, can it be a valid way or is there a correct / better / official way to do what I want?

[zzzeek]

you can do it that way but it seems more difficult than just omitting those items that you dont want considered. The include_object hook is the main hook for just indicating "skip all of these tables / columns / schemas", which is what it sounds like you are doing. include_name may also be used, which might be better when telling it to skip entire schemas to be ignored. the section https://alembic.sqlalchemy.org/en/latest/autogenerate.html#autogenerate-include-hooks has a good rundown over how to omit elements from the DB side as well as the model from the autogenerate process.