Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bot shutdown hang on connection error (or other unhandled exceptions) #6261

Open
wants to merge 5 commits into
base: V3/develop
Choose a base branch
from

Conversation

CyberKiller
Copy link

@CyberKiller CyberKiller commented Oct 16, 2023

Description of the changes

Change red_exception_handler() from calling shutdown_handler() to sys.exit()
fixes #5780

Have the changes in this PR been tested?

Yes

Added a catch all exception to the try block where red.start() is called to ensure bot shuts down correctly.
@github-actions github-actions bot added the Category: Core - Command-line Interfaces This is related to Red's CLIs (redbot, redbot-launcher, redbot-setup). label Oct 16, 2023
@CyberKiller
Copy link
Author

CyberKiller commented Oct 19, 2023

On further thought, I realise that a catch all exception will probably not be acceptable, as it will catch exceptions during the running of the bot and not just at start up so I am going to convert this pull request to a draft and look for a better solution to this.

@CyberKiller CyberKiller marked this pull request as draft October 19, 2023 09:52
@Jackenmen
Copy link
Member

Errors while the bot is running don't generally propagate this far and are handled earlier by things such as loop's exception handler. If they do propagate here, something more serious happened and we do probably want to shutdown after. If connection errors propagate here then the placement is likely fine.

@CyberKiller
Copy link
Author

Errors while the bot is running don't generally propagate this far and are handled earlier by things such as loop's exception handler. If they do propagate here, something more serious happened and we do probably want to shutdown after. If connection errors propagate here then the placement is likely fine.

So do you think a catch all exception is appropriate or would catching more specific exceptions be better?

@Jackenmen
Copy link
Member

It may be fine, can't really comment on that until I have a chance to test/play with it.

@CyberKiller CyberKiller force-pushed the startup-shutdown-hang-fix branch from ed040bb to c95a210 Compare October 19, 2023 11:57
@CyberKiller
Copy link
Author

CyberKiller commented Oct 19, 2023

I suspect if any exception doesn't get caught by run_bot() while the bot is starting up then the hang will occur. The question is does having a catch all exception that calls sys.exit() prevent shutdown_handler() from doing what it needs to do?

This reverts commit c95a210.

Revert "Fix Indent"

This reverts commit 7605ce3.

Revert "Fix bot shutdown hang on connection error"

This reverts commit 434e747.
red_exception_handler() hangs if it tries to call shutdown_handler() it's self so just call sys.exit() instead
@CyberKiller
Copy link
Author

CyberKiller commented Oct 19, 2023

I had a little more of a poke around and I think I found out the real cause of the hang: if red_exception_handler() catches an exception and calls shutdown_handler(), then the code never falls through back into main().
Replacing the call to shutdown_handler() in red_exception_handler() with sys.exit() allows the code to fall back though to main() and allows it to call the shutdown_handler() and run the finally block at the end of main() therefore stopping the loop correctly without hanging.
As for why it was behaving in this way I have no idea.

@CyberKiller CyberKiller marked this pull request as ready for review October 19, 2023 13:38
@CyberKiller CyberKiller changed the title Fix bot shutdown hang on connection error Fix bot shutdown hang on connection error (or other unhandled exceptions) Oct 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category: Core - Command-line Interfaces This is related to Red's CLIs (redbot, redbot-launcher, redbot-setup).
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Investigate shutdown handling from unhandled exceptions in main bot task
2 participants