How to use rich progress bar with RichHandler and python's logging API?

[LarsHill]

Hi :)

In our multiprocessing framework we use a logging queue and a QueueHandler from python's logging API to send all logs from child processes to a dedicated logging listener thread in the main process. In the main process we then configure a logger using the RichHandler from your library and an additional message formatter, which handles all the received logs from the listener thread.

Logging like this works fine. However, so far we could not figure out a way to also log for example rich's progress bar in a child process. Because right now we rely on the setup of calling e.g. logger.info('Some message.') so that the QueueHandler can forward the log message to the listener thread, where it gets handled. So somehow we're looking for a way to use the logger object in the child process to also send progress bar updates to the main process.

Ideally, in case of specifying a stream handler like your RichHandler the progress bar would update itself always at the bottom of the console and intermediate logs would pop up above the progress bar. In case of specifying an additional FileHandler to the main process logger, it would be still great to see the progress bar in the file log, but of course then line by line.

Is there a way of accomplishing this? :)

Thanks in advance for your support!

Edit 1: Not equally important, but besides logging progress bars, is there maybe also a solution to log other rich specific features such as trees, tables, etc. via the RichHandler and python's logging API?

Edit 2: For reference, our log forwarding approach described above follows the idea of the second example (using a listener thread instead of process) documented in the logging coocbook.

[willmcgugan]

Hi @LarsHill

You will need a mechanism to pass information from your workers to your main process. Either a pipe or a queue. Essentially what the QueueHandler is doing but for progress information.

Logging wasn't really designed to work with anything other than text. Although you can format anything Rich can render in log format with the log method.

Hope that helps.

Will

[LarsHill]

Hi Will,

Thanks for the reply. So, essentially we would create an additional Queue, e.g. rich_queue = multiprocessing.Queue() and in the same listener thread on the main process we would handle (print to stdout) the received rich information (e.g. progress bar updates, tables, trees, etc.).

Now, what I'm unsure about is how to implement the mechanism you're talking about (analogous to QueueHandler). Because the user currently can write logger.info('Some message.') and since the child process has a root logger configured with the QueueHandler, which holds the logging_queue, it get's passed to the main process. However, it seems to me this functionality is bound to python's logging API (global root logger and specific QueueHandler).

So, you're basically saying that we would need to rebuild this logic, but without relying on the logging API? Do you have any further pointers on that?

By the way, for tqdm it seems to be possible to couple it with the logging API. See here: tqdm/tqdm#724

Do you think something like this would also be possible for rich's progress bar? Of course it would be more convenient to just utilize logging's message passing implementation between processes, than implementing our own, taylored to rich.

Thanks again for your support :)

Edit: Now that I'm thinking about it, we probably need a general way to forward all information send to stdout and stderr in the child process to the main process, so that it gets handled by the stdout and stderr configuration of the main process. I'll look inte options described here: multiprocessing-how-can-i-reliably-redirect-stdout-from-a-child-process.
Solving this would also implicitely solve all rich related outputs but also normal prints, etc. which we also need to cover.

[willmcgugan]

So, you're basically saying that we would need to rebuild this logic, but without relying on the logging API? Do you have any further pointers on that?

Pretty much.

If you are using a queue, your workers would call queue.put() when they have progress to update. Your main process would receive these updates with queue.get() in a loop, and update the Progress bar accordingly.

By the way, for tqdm it seems to be possible to couple it with the logging

You can convert Rich objects to text and log that, which is what tqdm is doing there.

Edit: Now that I'm thinking about it, we probably need a general way to forward all information send to stdout and stderr in the child process to the main process

That would help, but you would still have the issue of managing Progress bars, as there can be only one active at a time.

[LarsHill]

That would help, but you would still have the issue of managing Progress bars, as there can be only one active at a time.

Ah ok, I am quite unfamiliar with the Progress bar API, but I will check it out. So, basically when several child processes create a Progress bar instance and these bars get forwarded to the main processes, I will run into problems since only one instance per process is allowed? I guess we will just start playing around with different options and see what in the end works.

When we come up with a general, good solution, I would add it here, since it might be helpful to others as well :)

Thanks again for your time and helpful comments!

[willmcgugan]

I will run into problems since only one instance per process is allowed?

It's more that there is only one stdout. In order to get the progress bar updating at the bottom, Rich has to move the cursor around. If another process or other progress bar is also moving the cursor the visuals become garbled.

[LarsHill]

That makes sense. We will see if we can come up with a workaround in this situation. So, it wouldn't help if we again make use of Queue for the general message passing to the main process? And then only a single progress update message from the queue will be send to stdout one at a time?

We will just start tackling this issue and see how we can get our desired behaviour :)

[willmcgugan]

Don't quite follow, but you should aim for a single Progress instance in the main process, updated by information sent by the workers.

Good luck!