Better optimiser defaults

[cweniger]

I have added a StandardOptimizer class that can be simply used as (see feat/better_defaults branch)

class Module(swyft.StandardOptimizer, swyft.SwyftModule):
...

and that automatically configures Adam, EarlyStopping, and ReduceLROnPlateau with reasonable defaults. One can overwrite the defaults with, e.g., self.lr = 1e-4 in the module __init__. In the long term, this could replace the current way of configuring things. But for now it is just a very simple addition.

@NoemiAM @james-alvey-42 Please let me know if there are any thoughts. We certainly have to update notebooks for that at some point, but right now I'm mostly interested in whether you agree with the general structure.

One of the reasons to add this EarlyStopping, LR decay etc as default is that the quality of the results massively depends on using these features. If the default is just Adam with a fixed learning rate, results are not very convincing and there is some annoying boiler plate code that one has to add all the time.

master...feat/better_defaults

[NoemiAM]

@cweniger Agree this feature is highly needed to be implemented as default without having to add external code every time. swyft.StandardOptimizer structure looks good. However, I would prefer swyft.SwyftModule to inherit directly swyft.StandardOptimizer. In this way a class Module will inherit only from swyft.SwyftModule as in the original version and in my opinion it's cleaner.

[james-alvey-42]

@cweniger @NoemiAM I agree also that this is very helpful to have. I agree with Noemi especially for the reason that the point of having this implemented is so people don't have to worry about implementing themselves. Of course, we can make it clear how to play around with it for more "advanced" users, but the less objects people need to worry about the better probably?

[cweniger]

One reason to not have this directly implemented as part of SwyftModule (effectively what @NoemiAM suggests) is that this would reduce flexibility. For instance, if somebody wants to use SGD, or another scheduler, or some other monitoring, not only configure_optimisers and configure_callbacks have to be overwritten, but also some of the data variables of SwyftModule (like lrs_factor) might become obsolete and just clutter namespace. Not sure how to deal with that.

[NoemiAM]

I see your point. How about having a flag that turns it on and off?

[cweniger]

I pushed another version. Now the defaults are all inside the configure_* functions. My suggestion would be to use exactly these configure_* functions just as the default methods in SwyftModule. If users want to tune the learning rate, or use automatic lr tuning, they can use the defaults and just set self.lr = 1e-4 or whatever in the init method of their module. If they want more complex behaviour or a different optimiser or scheduler or whatever (which ideally means they know what they are doing), they have to overwrite the configure functions.

[cweniger]

Ideally we should determine the defaults at some point by running a hyper-parameter scan.

[NoemiAM]

Re automatic lr tuning, it's worth to keep in mind that since swyft loss is now taking negative values, the standard PyTorch lr tuner doesn't work out of the box.

[cweniger]

Can you give me some details / a link? I noticed that the tuner sometimes crashes, but didn't understand why.

[cweniger]

Negative losses can be removed again, and we could just change how losses are printed in order to get the same effect. Maybe negative losses are very uncommon in ML, not sure.

[NoemiAM]

I don't have any links. I just remember when trying to run the lr tuner that it always crashed around where the loss started to assume negative values.

[cweniger]

@NoemiAM @james-alvey-42 I pushed a new version of updates.

• SwyftModule is now modular, I have separated the parts defining optimisers from the loss calculations
• The default when using SwyftModule is now AdamW optimiser with early stopping. Apparently ReduceLROnPlateau is "very 2015" (and typically in combination with SGD rather than AdamW). Modern recommendation include AdamW with learning rate tuning (which can now be easily switched on if needed), or AdamW with OneCycleLR policy. https://www.fast.ai/posts/2018-07-02-adam-weight-decay.html
• I added a callback such that the best model is automatically loaded at the end of training. That makes it obsolete to reload the model later on for testing or inferring (which is easy to forget when using early stopping, which is now the default, so that is important). Furthermore, the path to the best model is at the end of the fit stored in "model.best_model_path", so that it can be easily saved for later reloading etc.
• AdamW_OneCycleLR or AdamW_ReduceLROnPlateau are also there, just in case.

Let me know what you think. For now I only updated the 0A-* notebook to reflect the changes (in the feat/better_defaults branch). I guess one thing I'm interested in is whether these updates are consistent with your use-cases, or if there are collisions.