Gymnasium Compatibility

[ernestum]

Description

Make imitation compatible with gymnasium because the old gym is outdated.

Main changes:

• import gym -> import gymnasium as gym
• Adoption of the new gymnasium API (.seed(x) -> .reset(seed=x), done -> terminated/truncated, .reset() not returns a tuple)
• Added some extra checks because the new API does not guarantee that spaces have a shape
• We don't import seals any more and rather use the seals: prefix on environment names.
• make_vec_env does not mess with TimeLimit wrappers any more (gymnasium does that for us)

To be changed before merging

• Update the models on HuggingFace model hub (they are already trained on rnn, I just need to push them)
• Stop installing gym as part of the pipeline (that was just so we can load the current models)
• Depend on upstream seals and not on a branch of it.

Notes on the failing pipeline

• Windows pipeline is addressed in Add a test timeout #779

Testing

No new tests have been added but all tests pass now.

---

Thanks to all the people that helped coding and reviewing!
@AdamGleave, @araffin, @EdoardoPona, @Rocamonde, @simoninithomas,

This PR has been possible due to lots of work in our dependencies:

• Gymnasium Support (v2.3) huggingface/huggingface_sb3#35
• Fix typing issue in python 3.9 seals#77
• Migrate to gymnasium maintaining python 3.8 compatibility  seals#73
• Remove gym dependency DLR-RM/rl-baselines3-zoo#403

[Rocamonde]

Linking HumanCompatibleAI/seals#72 to this

[Rocamonde]

I am getting some weird pytype error that it cannot import packages. I remember facing this at some point in the past. @ernestum do you know what the cause of this is?

$ pytype -j auto
Computing dependencies
Analyzing 107 sources with 0 local dependencies
ninja: Entering directory `.pytype'
[1/64] check test_benchmarking
FAILED: /home/rocamonde/HumanCompatibleAI/imitation/.pytype/pyi/test_benchmarking.pyi 
/home/rocamonde/HumanCompatibleAI/imitation/venv/bin/python -m pytype.single --imports_info /home/rocamonde/HumanCompatibleAI/imitation/.pytype/imports/test_benchmarking.imports --module-name test_benchmarking --platform linux -V 3.10 -o /home/rocamonde/HumanCompatibleAI/imitation/.pytype/pyi/test_benchmarking.pyi --analyze-annotated --nofail --quick /home/rocamonde/HumanCompatibleAI/imitation/tests/test_benchmarking.py
File "/home/rocamonde/HumanCompatibleAI/imitation/tests/test_benchmarking.py", line 6, in <module>: Can't find module 'imitation.scripts.train_adversarial'. [import-error]
File "/home/rocamonde/HumanCompatibleAI/imitation/tests/test_benchmarking.py", line 6, in <module>: Can't find module 'imitation.scripts.train_imitation'. [import-error]

[ernestum]

@ernestum do you know what the cause of this is?

No I have never seen this error before 🤷

[ernestum]

Note @ernestum : ensure to treat all open comments in #765 here.

[AdamGleave]

@dan-pandori could you review this since Juan is busy with an ICLR submission right now? Please do tag me in any areas of the code you're uncertain about.

[dan-pandori]

LGTM! Thanks for cleaning up types in several places and the detailed PR description.

I just added a few minor nits about comments etc. I don't deeply understand the behavior of resetting the seed vs using seed(), so you should ask Adam if you want a better opinion on that.