Handling Private/Reserved Variables in Mesa

[EwoutH]

This discussion is split of from

• model: Automatically increase steps counter #2223

Background

Mesa models are often subclassed, raising questions about how to handle private or reserved variables to prevent conflicts or unintended overrides.

Key points

1. Naming Conventions: Establishing clear conventions for private and reserved variables.
2. Encapsulation: Balancing the need for encapsulation with the flexibility required for subclassing.
3. Documentation: Clearly communicating which variables are reserved or should not be modified.
4. Python Conventions: Aligning with Python's conventions for private and protected attributes.

Questions for discussion

1. Should Mesa use Python's double underscore name mangling for truly private attributes?
2. How can we balance encapsulation with the need for flexibility in subclassing?
3. Should Mesa provide accessor methods for important internal state to discourage direct attribute access?
4. How can we improve documentation to clearly communicate which attributes should not be modified by subclasses?

[EwoutH]

My solution for this is quite simple, we just make a list.

Model.reserved_variables = ["agents", "step", "running", ...]

If the user tries to overwrite any of these variables on initialization, they get an error with a pretty error message explaining what's happening.

To introduce new reserved variables, we could add a dictionary, which throws a deprecation warning if one of the keys is overwritten, with the Mesa version in which it will be reserved as value.

Model.future_reserved_variables = {"agent_list": "3.0", "time": "3.1"}

We document this list clearly and keep a page on what each reserved variable does.

Agents can also have reserved variables in the same way, since they are often overwritten.

This way we don't have to use the _variables since I find them very ugly.

[quaquel]

Should Mesa use Python's double underscore name mangling for truly private attributes?

Probably overkill

How can we balance encapsulation with the need for flexibility in subclassing?

Keep internal attributes well defined and limited in scope

Should Mesa provide accessor methods for important internal state to discourage direct attribute access?

I am not sure. Using properties might be a convenient way forward. I think it will make it easier to raise warning if people try to overwrite internal attributes. This of course next to clear documentation of all reserved names. I would have to experiment a bit to see what is possible but some kind of metaprogramming seems the most robust way to handle this.

How can we improve documentation to clearly communicate which attributes should not be modified by subclasses?

Probably this needs to be handled in more then one way. For example, a notes in the docstring of Model and Agent highlighting internal attributes is part of it. It should be explicitly covered in tutorials and meaningful warnings/errors should be raised.

[quaquel]

While looking through various libraries for #2291, I came across an example of using __init_subclass__ for explicitly raising a TypeError if any of the attributes in subclass.__dict__ was in a set of forbidden attributes. They made their life easier by having all forbidden attributes start with _<name of library>:

    # taken from psygnal._group.py

    def __init_subclass__(
        cls,
        strict: bool = False,
        signal_aliases: Mapping[str, str | None] = {},
    ) -> None:
        """Collects all Signal instances on the class under `cls._psygnal_signals`."""
        # Collect Signals and remove from class attributes
        # Use dir(cls) instead of cls.__dict__ to get attributes from super()
        forbidden = {
            k for k in getattr(cls, "__dict__", ()) if k.startswith("_psygnal")
        }
        if forbidden:
            raise TypeError(
                f"SignalGroup subclass cannot have attributes starting with '_psygnal'."
                f" Found: {forbidden}"
            )

[EwoutH]

Sounds well thought out. Go ahead with the implementation that's most suitable in your view!

[rht]

I would have wanted to survey 1 or 2 more OOP-heavy libraries (maybe Django), but __init_subclass__ already allows us to have more human-readable protected attributes.

[rht]

Never mind, I suppose from readability (of protected attributes) standpoint, the alternative options are about the same. Psygnal's solution is performant, is simple to implement, and is easier to comprehend. I'm fine with proceeding with this direction.

This is optional, and I will do it in my own time instead: t may be overkill to read other libraries code, but this is a general design question, and so, may have an idiomatic, preferred way of doing it. There is FastAPI, a modern web framework, which probably has fewer legacy patterns than Django.

[quaquel]

So I gave __init_subclass__ a try, but it does not completely do what we want to do. It is a class method and is called with the class. So, you can only check for attributes defined at the class level. However, typically, people put the attributes into the __init__, and this is run after __init_subclass__ has been called.

[rht]

FastAPI uses Pydantic for dataclass validation. Pydantic has a way of informing of collisions with protected namespaces. This feature itself is not used by FastAPI, however. Using Pydantic seems worth it only when ensuring the integrity of Mesa the library is more important than performance. It's a no-go.

[Corvince]

I think we have all the pieces at hand to handle this now

1. Private, internal attributes.
   I think the idea to rename them to start with _mesa is a good solution. Then we only need to document properly that attributes should not start with _mesa. This is already unlikely in itself and the cases where breakage occurs will be very limited. (or even use name mangling. Is there a downside to using name mangling? I have never completely read up on it)

2. Public reserved attributes
   For public, but reserved attributes we can simply use the @property decorator. This will create an error if trying to set an attribute with the same name, even in subclasses. We actually already have this in place for most agent related attributes

3. Public methods
   If users accidently overwrite methods that should not be overwritten we can handle this with __init_sublcass__.

Did I forget anything?

[rht]

For point 3: #2230 (reply in thread)

[Corvince]

For point 3: #2230 (reply in thread)

Point 3 is just for methods. Methods are defined at the class level, so the limitations dont apply luckily

[quaquel]

I think these three points cover everything. It might involve moving a few things into properties. I'll try to find time soonish to get this done.

[Corvince]

Great! Now we just have to discuss which public attributes should be protected.

I think currently we have the following, that are not yet a property "running", "schedule", "steps" and "random".

"schedule" can just be removed in 3.0 (right?)
"running" could maybe be enforced to be boolean (with a setter) ?
"steps" should probably be a read-only property for a private _mesa_steps attribute?
"random" ... Well I think we definitely need a better "random" story.

[quaquel]

yes that is in line with my ideas. Once I dive into it, I might spot a few more.

Regarding random, yes, we need a better solution, but in the meantime, we can probably use some property here with a bit of validation on the setter (e.g., model.random = 5 would mean model._random = random.random(5)).

[EwoutH]

@quaquel I feel you have an idea about an implementation for this one, would you like to create one?

[quaquel]

I agree with the 3 points of@corvince above. It's more a matter of finding a few hours to go through the code and implement it. In fact, you might have seen that in #2292, I already used some of those points. It might also be good to document this approach somewhere in some kind of guide for developers.

[EwoutH]

Python 3.13 includes a new typing.ReadOnly construct to mark an item of a TypedDict as read-only. Not sure if we can use it since we define class-level variables, but it still might be interesting.

__static_attributes__ was also added, which stores the names of attributes accessed through self.X in any function in a class body. Could be useful.