On dataclasses and typing

[dmos62]

Let us discuss the following, somewhat recent, Python features: dataclasses and typing.

I'm not structuring this discussion with a proposal, since I think that right now we could use a general discussion on the subject. When we want to raise and consider a proposal, I recommend opening a dedicated discussion for that.

[dmos62]

I've been using the backend filtering refactor PR as a test bed for dataclasses and typing and I'm a fan. These are my thoughts as summarized there:

• dataclasses eliminated custom boilerplate; I got immutability, _post_init and defaults without writing a single constructor; I think this is great since it's essentially standardized boilerplate;
• typing got me partial typing, which is great:
  • in combination with my LSP server/client (pyright): catches a lot of mistakes and conflicts
  • in that I can annotate when I think it's useful and not when I don't (partial typing)
  • for readability, for example the above method is_ma_type_supported_by_predicate operates on uninstantiated predicate classes, not instances, so to express that I can just say predicate: Type[Predicate]) instead of predicate_subclass
  • doesn't have a cost; another developer can just omit type or write Any if he prefers

Data classes

My plan with this post was to paste the above summary and then write two data class implementations for comparison: one using dataclasses and one using custom boilerplate. But, I stumbled upon a blog post that makes all the points I wanted to make in a concise manner, so I think you'll appreciate me posting that instead: https://towardsdatascience.com/9-reasons-why-you-should-start-using-python-dataclasses-98271adadc66

Partial typing

I find that partial typing, it being optional, is a situation without losers and thus is a no-brainer. For more specific thoughts, see above quote.

[seancolsen]

Take my opinion lightly since I'm not working with Python code in my current role here. But:

• dataclasses: ❤️
• typing: ❤️
  • I'm used to writing Python with full typing and zero type errors from mypy. Very occasionally I'd add # type: ignore, but only as a last resort.
  • My personal opinion... adding types requires more time initially; but the investment pays for itself later by increasing efficiency. The earlier I catch a bug the less time it takes to fix -- and types can uncover some bugs very quickly, especially during refactoring. The more clearly I can see the intent of other devs, the more quickly I can understand their code -- and types help to communicate that intent.
  • I asked about Python types during my job interview here, and Kriti explained that eschewing types in Python was an intentional decision from the start. So with that precedent in mind, I don't want to push much on this topic. I'm happy over here using TypeScript 😁

[mathemancer]

I started writing this comment not caring much about type hints, and became less of a fan the more I thought about them.

I think if we want static typing, we should use a proper statically typed language (e.g., golang) that has the features you'd expect: real function overloading, interfaces, etc. I find working with static types in python a bit tedious, since you don't get any of the real goodies of a static typing system. What you get instead is nagging.

Moreover, I like duck typing and I think it forces the programmer to focus on behavior rather than getting bogged down in type. You often (for example) don't care whether something is a tuple or a list in python, you just care that you can iterate through its members. Encouraging the programmer to specify this in the function signature encourages them to lazily specify whatever the first structure they think of is. If they, for example, specify a tuple of integers in their function signature, it makes it more irritating down the road for a programmer who realizes they can save some memory (and flops in many cases) by using a generator expression instead. While it's possible to specify Generator or Iterable using the typing module for the annotations, this requires knowing those exist in the typing module and importing them. I think it's more likely that someone just puts tuple, list, or (at best) Tuple[int], assuming the implementer knows/cares about the generics available from the typing module. All of this mental overhead is to avoid trying to simply iterate over whatever was passed and handling any errors, which needs to be done anyway, since the type hints aren't enforced.

As far as dataclasses, I have a general suspicion of "magic" annotations that generate a bunch of hidden code in large projects. Maybe too much trauma from diagnosing bugs involving Lombok and SpringBoot in the Java world. The point is, if you can't see the actual __init__, you have to read the docs to see what the magic (newly introduced) syntax is going to produce for that function, rather than just reading an __init__ defined in the class. I do appreciate taking out some boilerplate, but I'm not convinced it's worth the cost of introducing "new" syntax into the codebase, and obscuring the underlying logic.

For example, how is frozen for dataclasses implemented? If I init with a mutable type for an attribute of a class, I can still mutate that attribute, correct? What if the type is mutable not only in membership but in length? Could I specify List[str, str], but then append to that list in a frozen dataclass, violating both the intuitive meaning of "frozen" and the type hint without warning? To know the answer, I need to understand the implementation, which isn't in the docs (from what I can find). So to answer that I need to go to the actual codebase. To contrast, I (as do most people with any python experience) know what to expect if I init an attribute of a non-frozen, non-data class with a list. I can, using duck typing, even initialize any given iterable as a tuple, making it actually immutable (in combo with making it private and using a getter). Doing so would be visible to the next programmer reading the logic, and make it much easier to understand what to expect.

All that said, while I have strong reservations about type hints and data classes, I'm also comfortable using them if the project goes that way.

[dmos62]

I think that being precise and expressive in type specifications (or any specifications) can be important in some cases and it's not particularly hard to do in typing. You have to read the docs to do anything more interesting than List[str] and it makes you think about things you may not have thought of before, but I don't think that that's a problem. If someone doesn't know how to express a type signature, he can add a comment instead, or look it up in the docs.

I feel like you're talking about adding typing annotations to everything. I don't think that's desirable. What I see is developers annotating when and where they think it's useful. By the way, I did find type annotations useful in my workflow.

On the matter of dataclasses, I disagree that it's better to read custom code than the Python docs. By the way, if someone prefers reading code to docs, the code is available: https://github.com/python/cpython/blob/main/Lib/dataclasses.py Maintaining custom boilerplate code is also more work for us. As I said before, I find that standard boilerplate is nice.

[mathemancer]

I feel like you're talking about adding typing annotations to everything.

I'm assuming we won't be requiring that. My concern is less that we'll be forced to add them everywhere, and more that it's easier to specify "this function takes a list" than it is to assume the desired behavior from whatever's passed, and handle any errors.

This means that I'll be trying to improve performance or memory consumption at some point, and get bogged down changing signatures down some function call stack, rather than just changing the passed object from a list to a generator expression (as an example), even if the code would work fine with a generator expression. In this case, I'm not being forced to deal with type hints by fiat, but rather because a previous programmer (maybe myself) assumed that the function would be called with a list, and didn't think about what behavior actually needed to be specified.

Using the type hints, even if we're not requiring them, encourages thinking of the type you expect rather than the required behavior. This isn't just a mental trap for the programmer in the moment, it adds work whenever you want to modify the function in certain ways. TBH, I avoid specifying types in docstrings sometimes for the same reason, and type hints are essentially machine-readable docstrings.

By the way, if someone prefers reading code to docs, the code is available: https://github.com/python/cpython/blob/main/Lib/dataclasses.py

This isn't the code that's running in our project (if we use dataclasses). This is code that generates the code that's running in our project. The actual __init__ of the class created by the @dataclass annotation isn't available to look at unless you figure out how to inspect an in-memory instance of the class in the right way. (same for the __setattr__, or __delattr__, etc.) The advantage (which I acknowledge) of the dataclasses isn't that they somehow avoid the boilerplate, it's that they automatically generate a bunch of boilerplate under the hood. The disadvantage of that is that when debugging, the actual code that runs isn't easily available. I've had some truly nightmarish times in the past diagnosing bugs in automatically-generated boilerplate that wasn't acting as expected (again, in the Java world). Dataclasses are pretty simple, though, so it's possible I'm being over-paranoid.

Maintaining custom boilerplate code is also more work for us.

Fair point, and acknowledged.

[dmos62]

I'm not being forced to deal with type hints by fiat, but rather because a previous programmer (maybe myself) assumed that the function would be called with a list, and didn't think about what behavior actually needed to be specified.

It sounds like someone's made a mess of the type annotations in this hypothetical. I agree that what you should expect is the broadest definition of what is enough (i.e. think about the behaviour), and I think the type annotations should be written with that in mind.

Passing a generator object where a regular collection is expected is dangerous (since reading a generator object mutates it), so you'll probably have to inspect the callstacks anyway. Type annotations help with all this. You just have to think about the right things when writting the annotations. If we couldn't trust ourselves to write useful type annotations, that would be different, but I don't think that we have that problem.

You could also scrub the unuseful annotations.

The actual __init__ of the class created by the @dataclass annotation isn't available to look at

Oh right. But, like you said, maybe dataclasses are simple enough that a decorator works fine?

[kgodey]

It sounds like someone's made a mess of the type annotations in this hypothetical.
[...]
You just have to think about the right things when [writing] the annotations. If we couldn't trust ourselves to write useful type annotations, that would be different, but I don't think that we have that problem.

You could also scrub the unuseful annotations.

We're working on an open source codebase and we're hoping to have a large number of outside contributors (and hopefully code reviewers). We should not make decisions based on what we can trust this group (or senior-level programmers generally) to do.

Needing to review and scrub annotations will give us more ongoing maintenance work to deal with.

[silentninja]

I think if we want static typing, we should use a proper statically typed language (e.g., golang) that has the features you'd expect: real function overloading, interfaces, etc. I find working with static types in python a bit tedious, since you don't get any of the real goodies of a static typing system. What you get instead is nagging.

Exactly what I wanted to say. Moreover frameworks, tools, design patterns for statically typed language are actually built with that in mind, so trying to achieve 100% consistency with typing in python is going to be futile, which would mean that it should be used only when needed with proper reasons. Since a 100% consistency is not possible, it will be hard to enforce or trust it using a static type checker and this would require using tricks like Any

But I do love using it as an alternate to docstrings, provided there is an absolute need for it(which is going to be hard to determine or enforce). For example, when expecting a particular structure as an exception response. So I would want it to be limited to interfaces/ public apis, just like test cases, since interfaces tend to be somewhat better defined. I also prefer to limit to very simple typings rather than complex structures like union of a union

Static typings are great for large and well defined codebases, so the tradeoff for flexibility is worth it. But we would benefit from flexibility rather than a rigid structure as of now.

I really loved static typing when I had to switch to a typed language from a large Django codebase, but later realised that I was not writing pythonic code. Python codebases from what I have seen are very modular(script like), easy to look into, flexible and rely on test cases as they are really cheap compared to other languages. I just find it a worthy compromise

As for dataclasses, I prefer to use primitives as much as possible, to have a simple and expected behaviour and avoid using it for what it was not intended to. But I do find it useful in some cases and see typing as the cost of doing business.

The biggest advantage and disadvantage of python is its flexibility, so it is hard to give a binary answer, so I am leaning towards easy to understand, straightforward vocabulary solution. Right now I see them as a magic tools that python provides which should be used when the situation demands it

[dmos62]

I don't want to be the guy that comments under every comment, but I'd just like to point out that type systems are not binary (static or dynamic) but rather on a spectrum. The extremes of that spectrum are not the only options we have. I'd like us to give ourselves some wiggle room in these things.

@silentninja totally agree that these are tools to be used when useful; anything else wouldn't make sense, after all.

[mathemancer]

The biggest advantage and disadvantage of python is its flexibility,

Ain't that the truth :)

[kgodey]

General thoughts:

• I don't think any of us are opposed to less boilerplate code, but dataclasses are not the only way to achieve that.
• @mathemancer has made some excellent points about generally why not to use type annotations and dataclasses.
• We're largely talking about hypotheticals in this thread, and I'm not convinced we're on the same page. We're probably all imagining very different scenarios with regards to how these features would be used in the Mathesar codebase.

In the interest of reaching a decision quickly, I think we should avoid using dataclasses and type annotations for now. If there is a specific use case that anyone thinks that these features would help solve better than other solutions, then let's talk about that particular use case when it comes up.

[dmos62]

Our perspectives are different. Well, to be fair, there seem to be only two camps, and I'm alone in mine, haha. So I guess my perspective is different.

In my very personal experience, technological solutions are where I can find the lowest hanging fruit. The tech (a.k.a. tool) might not be trivial to use or to switch to, but if it can improve the consistency with which a problem is solved, it's usually worth the effort in my book. In a situation like that, I'm exchanging the effort of dealing with inconsistency with the effort of upgrading my toolkit.

I guess it's a question of when the exchange makes sense. I have a low tolerance for inconsistency and a high tolerance for novelty. But, I guess I should appreciate that others might look for setups that favor other things.

At the same time, I feel like we're sometimes prioritizing technological conservacy. I acknowledge the point about that being good for the future community around Mathesar, but I'm just not sure if that's true.

---

Needing to review and scrub annotations will give us more ongoing maintenance work to deal with.

See, in my experience it's the opposite. It's the ambiguity that requires ongoing maintenance, and annotations address that.

We could use the same argument for code comments and docstrings. They put constraints on your thinking and interpretation; they can be done wrong; when you change a seemingly small thing you might have to go through the comments making changes; writing up code might make you think about nuances that you wouldn't have to otherwise think about (again, slowing you down and constraining you).

What I was saying is that we shouldn't approach this tool as something that we (or the community) wouldn't have the skill to use.

I think we should avoid using dataclasses and type annotations for now. If there is a specific use case that anyone thinks that these features would help solve better than other solutions, then let's talk about that particular use case when it comes up.

I don't expect type annotation related discussions to come up. Each moment it might be useful is such a small nuisance on its own that the effort to renew a discussion will not feel warranted.

I'll see how data classes work out in the filtering refactor. We might be able to experiment with alternatives there.