Simplifying ideas

[blythed]

Removing blocks of dead code

• superduperdb/data
• superduperdb/misc/dataclasses.py
• superduperdb/misc/run.py
• superduperdb/vector_search/faiss_index.py
• superduperdb/db/mongodb/loading.py

Things we could do in a simpler way

• Encoders: we could hard code all encoders, instead of adding them as components.

Things which are nice to have but which may not be necessary to interest early adopters

• Training
• Server

Potential for significantly less code

• superduperdb/db/mongodb/query.py
• superduperdb/vector_search/*

[rec]

This seems pretty low priority overall.

I don't think anyone reads through codebases and says, "This class isn't used, so the whole thing is useless."

We had a little unittest framework for servers, and instead of writing tests for the new server, we deleted the test framework, which we'll have to bring back again at some point.

Cutting down mongodb.py/query.py seems the highest priority by far, as everyone who reads anything will read that file and it's long.

s.misc.dataclasses was a mistake when I was struggling with Pydantic 1 and screws with our typing, it should go.

We already use s.data.tree a great deal, s.data.cache will be used in the typed server. We will use subprocesses again, if only to test our existing code, and misc.run is tiny.

I have twice I had to deal with the faiss stuff to prevent it from spamming us on startup, would love to see it go.

---

Some of these can be done instantly, so I'll just do the easy ones right now.

[rec]

See #649

[nenb]

I have a similar feeling to @rec here.

My vote would be to build out a couple of use-cases, and then let these use-cases and user feedback inform us about what needs to be improved/reduced/deleted.