Send inputs to already running remote process

[rinkurajole]

Hello Everyone,
I've written below script to run remote python script. It's runnning good and read/write input/output is also working.

import asyncio, asyncssh, sys

class MySSHClient(asyncssh.SSHClient):
    def connection_made(self, conn: asyncssh.SSHClientConnection) -> None:
        print('Connection made to %s.' % conn.get_extra_info('peername')[0])

    def auth_completed(self) -> None:
        print('Authentication successful.')

async def run_client() -> None:
    conn, client = await asyncssh.create_connection(MySSHClient, '<remote_host>', username='<user>', client_keys='<ssh_key>')

    process = await conn.create_process('/usr/bin/python3 /tmp/foreverrun.py')
    for op in ['2+2', '1*2*3*4', '2^32']:
        process.stdin.write(op + '\n')
        result = await process.stdout.readline()
        print(op, '=', result, end='')

try:
    asyncio.get_event_loop().run_until_complete(run_client())
except (OSError, asyncssh.Error) as exc:
    print(exc.__traceback__)
    sys.exit('SSH connection failed: ' + str(exc))

With above code I'm able to run foreverrun.py and send input and read output. As foreverrun is running while loop so it's keep running even after I exit the above script. Now my question is, Is it possible to reconnect to already running foreverrun process again and send more messages to it with asyncssh?

foreverrun.py

import os, sys

outfd = os.dup(sys.stdout.fileno())
infd = os.dup(sys.stdin.fileno())
outfile = os.fdopen(outfd, "w")
infile = os.fdopen(infd, "r")

while True:
    data = infile.readline()
    if 'break' in data:
         break
    outfile.write(data)
    outfile.flush()

Thank You.

[ronf]

As written here, forever run.py may appear that it keeps on running, but I think you'll find it's actually in an infinite loop reading from a closed "infile", chewing up lots of CPU time with readline() repeatedly returning an empty string. That's probably not what you want, but stdin & stdout are going to be closed as soon as the top script closes the connection (or the process it starts on that connection).

If you are looking to have a long-running script on the remote system that you can attach to from different SSH connections over time, you'd probably need to use something like 'screen' or 'tmux' for that. That would be independent of anything happening at the SSH level, but you could have the SSH clients run something like "screen -r" to reattach to a previously created screen session. There are other variants of the "screen" command you might want to use if you want it to automatically create a new screen session when one doesn't exist yet, or if you want to detach other connected clients each time a new client is started. See its man page for details.

If you just want to do more after breaking out of the loop in foreverrun.py, that should be possible after the client sends "break", as long as you don't exit the client process.

If you want to support running foreverrun.py multiple times over the same SSH client connection, you should be able to call create_process() multiple times using the same SSHClientConnection (conn). Each of these will run its own separate copy of foreverrun.py on the server, though.

[rinkurajole]

Thank you very much Ron for quick response.
The scritps in question are just sample scripts I've added. In real scenario I've continuos running client(python processes). The existing flow is, I've N number of steps in a job and every step do some stuff on remote machine. Whenever some job comes in, To execute each step I've to do follow:

1. Establish ssh remote connection from client side to remote
2. Launch process
3. Set context to do the job(Messages get sent to process over stdin from client)
4. Execute the step and reply(Outcume of the step get sent back to client over stdout of process)
5. Do cleanup and kill the process
6. Close the ssh connection

Now, I want to execute all steps from a job with initially launched process(want to avoid process launch, set context, cleanup again and again).
This can be done by preserving/caching ssh connection and create_process objects but will it be a good approach as I may have 1000 or more remote hosts and caching that much objects may lead to memory issue?

Is the any good way to do this ?
Also, Is it possible to open multiple sessions around same process?

Thank You.

[ronf]

In SSH terminology, there are connections and channels, where you can open multiple channels over the lifetime of a connection (including having a small number open simultaneously), allowing you to avoid having to open the SSH TCP connection and perform the initial SSH handshake and authentication every time. A channel can be used to run an interactive shell, run a command, talk to some other subsystem like an SFTP server, or perform various kinds of "port forwarding" between the SSH client and server.

Unfortunately, SSH doesn't let you open multiple channels tied back to the same remote process. Each SSH channel has an associated session which is started fresh when the channel opens. So, if you are connecting to a standard SSH server like OpenSSH, it will always create a new process on the remote system every time. You'd have to use a program like "screen" to allow that new remote process to attach to some other existing remote process. There should be no problem running "screen" over an SSH connection to do this, but "screen" is more designed for terminal sessions, and so it's going to do things like attempt to redraw the terminal's "screen" every time you reconnect to a process. This may make it difficult to use in the way you're describing here.

Assuming you go with a new process every time, you can still eliminate the need to do steps 1 & 6 in your list multiple times, but there are some limits on the number of simultaneous sessions allowed on a connection, and also a limit on how quickly you can start new SSH connections if you want to run a really large number of sessions to the same host in parallel. You can use things like semaphores to limit the amount of parallelism there, though, staying within the limits of the remote servers. If you own those servers, you can also increase the limits if you want to. The default is usually something like a max of 10 simultaneous sessions per connection, and a similar small limit on how many SSH connections are allowed to start in parallel (before authentication completes).

As for supporting a very large number of remote servers at once, that should be less of a problem, though you may find you need to increase the number of open file descriptors on your Python process running the SSH clients. It takes one file descriptor per SSH connection, and some systems can max that out at something like 256 or 1024 open FDs. You can raise this with "ulimit".

In theory, if you design your remote scripts right, you COULD keep around and reuse not only the SSH connection objects but also the SSH process objects (and their associated SSH channels). However, you'd also need to keep around the remote process, making sure it didn't exit and it didn't close its stdin/stdout/stderr. In that case, you could give the existing client-side SSHProcess more data to write to stdin and it could read back the results from stdout before putting the process objects back in a pool. The key point is that whatever you start up as the command to execute when you first create the process needs to keep running the entire time the channel remains open.

Assuming your remote scripts are Python, the command you tell SSH to run could just be a Python interpreter on the remote system, and that could very well stay around across jobs. You'd just need to have a way through stdin to tell that interpreter which new job to run each time the previous one finishes, and only exit the remote Python interpreter when you were ready to close the channel.

[rinkurajole]

Thank you very much Ron for the detailed answers to my questions. I'll try the methods you suggested and will post the updates and questions if any.