You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For very large emulators, it can sometimes still take many hours (or days, if they get extremely large) to analyze an iteration.
Losing progress in this analysis process due to an interruption (like, timeout on a cluster) could cost a lot of time.
This should be avoided.
I could modify the _evaluate_sam_set method to automatically make a backup of all evaluation progress whenever the entire plausible sample set has been evaluated in a single iteration.
I do not think it can be done any earlier due to that requiring many MPI calls back-and-forth, which heavily slows down the process.
If this modification could be made to specifically work for analyze, the method can check if an analysis process was in progress and continue if so.
I will have to think about what a KeyboardInterrupt does (do I see that as the user not wanting the process to finish and thus PRISM should remove the progress backup?).
The text was updated successfully, but these errors were encountered:
For very large emulators, it can sometimes still take many hours (or days, if they get extremely large) to analyze an iteration.
Losing progress in this analysis process due to an interruption (like, timeout on a cluster) could cost a lot of time.
This should be avoided.
I could modify the
_evaluate_sam_set
method to automatically make a backup of all evaluation progress whenever the entire plausible sample set has been evaluated in a single iteration.I do not think it can be done any earlier due to that requiring many MPI calls back-and-forth, which heavily slows down the process.
If this modification could be made to specifically work for
analyze
, the method can check if an analysis process was in progress and continue if so.I will have to think about what a
KeyboardInterrupt
does (do I see that as the user not wanting the process to finish and thus PRISM should remove the progress backup?).The text was updated successfully, but these errors were encountered: