MFA/2-factor/OpenSC Authentication Issues

[michael-sandoval]

Hi -

Some staff here at ORNL/OLCF are required to login to OLCF systems with a yubikey, which utilizes OpenSC to use their smartcard (yubikey) pin instead of a traditional RSA passcode. The way this is handled with SSH is by having this in a ~/.ssh/config file:

host *.ccs.ornl.gov *.olcf.ornl.gov *.ornl.gov pivot pivot2
  Match user <insert_username_here>
  PKCS11Provider /usr/local/lib/opensc-pkcs11.so

The above works for traditional SSH logins via terminal, but VisIt hangs when connecting to the remote host (in this case, the Andes OLCF cluster) when using a username that requires this special type of login. No password/passcode prompt is ever displayed and VisIt just hangs until you have to manually kill the process. This happens on both Mac and Windows, and "passwordless SSH" does not solve the issue, as authentication is still smartcard/yubikey based rather than trusted SSH keys.

This workaround is known to work for some: https://puppet.com/blog/speed-up-ssh-by-reusing-connections/

(i.e., connecting to Andes via SSH on terminal, then piggybacking off of that connection so that VisIt doesn't have to prompt for a yubikey pin)

However, we are looking to find a more permanent / less-workaroundy solution, as the above didn't end up working for everybody. Any ideas? As a comparison, this type of issue doesn't happen with ParaView, but I'm guessing that has something to do with that application explicitly using something like xterm/xquartz when trying to authenticate.

This is using VisIt 3.2.2, but we did not see success in the past with older version either.

Michael

[markcmiller86]

@visit-dav/visit-developers anyone have any ideas for this inquiry?

[markcmiller86]

Hi -

Some staff here at ORNL/OLCF are required to login to OLCF systems with a yubikey, which utilizes OpenSC to use their smartcard (yubikey) pin instead of a traditional RSA passcode. The way this is handled with SSH is by having this in a ~/.ssh/config file:

host *.ccs.ornl.gov *.olcf.ornl.gov *.ornl.gov pivot pivot2
  Match user <insert_username_here>
  PKCS11Provider /usr/local/lib/opensc-pkcs11.so

The above works for traditional SSH logins via terminal, but VisIt hangs when connecting to the remote host (in this case, the Andes OLCF cluster) when using a username that requires this special type of login. No password/passcode prompt is ever displayed and VisIt just hangs until you have to manually kill the process. This happens on both Mac and Windows, and "passwordless SSH" does not solve the issue, as authentication is still smartcard/yubikey based rather than trusted SSH keys.

This workaround is known to work for some: https://puppet.com/blog/speed-up-ssh-by-reusing-connections/

The above article appears to have been removed or the path has changed. That said, I am thinking it is about using SSH in Control Master mode (aka Multiplexing). That allows a new ssh connection to ride-along on an existing connection. Its kinda sorta like passwordless but it isn't. You do have to authenticate the initial connection. But, its a great feature to deal with situations like this.

(i.e., connecting to Andes via SSH on terminal, then piggybacking off of that connection so that VisIt doesn't have to prompt for a yubikey pin)

However, we are looking to find a more permanent / less-workaroundy solution, as the above didn't end up working for everybody. Any ideas? As a comparison, this type of issue doesn't happen with ParaView, but I'm guessing that has something to do with that application explicitly using something like xterm/xquartz when trying to authenticate.

Hmm...not enirely sure what you are saying here regarding ParaView method of authenticating. But, I wonder if you mean that ParaView is essentially turning around and creating control-mastered connection automagically by first having the user authenticate via a basic terminal connection? The way you could tell for sure is to look in ~/.ssh/<some-folder> (where <some-folder> is often named something like cm_socket and see if you see a lock file getting created (it isn't there before the connection with ParaView and is there after). Then delete that file for laughs and see what happens to ParaView connection...it'll likely fail. Alternatively, ParaView could be doing all of this without fiddling with your ~/.ssh and instead doing it in some temporary location (maybe using an ssh that comes with ParaView or maybe using your ssh or another local ssh).

What if we gave you a check-box in our host-profiles that basically did what I describe above...It launches a local terminal window (which you would likely have to go hunt for in the window system on the client because it may not appear in front of VisIt's windows) where you enter username and password (or the yubi-key thing) and then control-mastered Visit's connection on top of that automagically?

I need to do a little more research about yubi-keys and ssh authentication.

@biagas and @brugger1 this may be another example of a case where instead of trying to integration the ssh connection logic into VisIt's Qt GUI, it might be better to simply delegate to whatever the local ssh tooling is. Thoughts?

[markcmiller86]

With how our systems are setup, the below snippet in .ssh/config works if you initiate a connection to one our systems (e.g., Andes) and then explicitly exit that terminal. VisIt will then be happy with using that existing connection to connect to Andes fine (as long as you're within the ControlPersist window), but will hang indefinitely if you did those steps and forgot to exit the initial terminal (or if you didn't have your .ssh/config right).

Can you add -vvv to the initial ssh connection you create and then keep that terminal active instead of exiting? Then, attempt to connect with VisIt. The terminal window will spew ssh debug stuff. Sanitize and paste here or if you prefer, email miller86 at llnl dot gov.

[michael-sandoval]

I'll email it, I have a few debug output files that may be helpful

[markcmiller86]

@michael-sandoval I haven't finished looking over the debug logs you sent. But, I wanted to suggest, can you try adding -nopty to the command-line you use to launch VisIt while you keep the other, initial ssh connection terminal open?

[michael-sandoval]

From the initial terminal's end of things, similar failure output of getting rejecting for trying to multiplex. In the terminal used to launch VisIt from the command line with -nopty, this is what it spits out while getting rejected (very rapidly without actually entering passwords/pins):

Enter PIN for 'Sandoval, Michael (9b8)': 
login failed
pkcs11_get_key failed
sign_and_send_pubkey: signing failed for RSA "PIV AUTH pubkey": error in libcrypto
(sando@andes.olcf.ornl.gov) Enter PASSCODE:
(sando@andes.olcf.ornl.gov) Enter PASSCODE:
(sando@andes.olcf.ornl.gov) Enter PASSCODE:
sando@andes.olcf.ornl.gov: Permission denied (publickey,keyboard-interactive,hostbased).

[markcmiller86]

Ok, thanks for trying that. It was worth a shot but based on a total swag on my part. I'll keep digging.

[biagas]

On Windows, you can use an environment variable to override use of VisIt's internal ssh. This will bypass use of the password window, and the ssh's terminal window will open instead.

I wonder if that would help the situation.

You would add VISITSSH environment variable with the path to ssh (OpenSSH in system location for newer OS versions) or plink (putty), or other form of ssh. If extra args are needed for ssh they can be specified via VISITSSHARGS environment variable. Without the password window/change username option you will have to ensure the username is correct in the host profile before attempting to connect to the remote machine.

[markcmiller86]

@biagas do you happen to know if the VISITSSH functionality to bypass the password Qt widget works also on other platforms? Or, is currently implemented to work only on Windows?

[biagas]

Based on logic in RemoteProcess, it should work on all platforms. I don't have any experience to know if the behavior is the same (eg no PasswordWindow).

[markcmiller86]

Ok, I tried a few things. First, just because qtssh in VisIt itself is there, I tried setting VISITSSH to point to that ssh tool and then I tried to connect to an LC resource. But, I tried just trying to connect running that bare app first. It failed with

qt.qpa.plugin: could not find the qt platform plugin "cocoa" in "".

So, I needed to set

QT_QPA_PLATFORM_PLUGIN_PATH=/Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64/bin/gui.app/Contents/MacOS/platforms/

to work-around that. That started to work but then I got

FATAL ERROR: Couldn't agree a key exchange algorithm (available: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521)

because the version of PuTTY it is based on is too old.

So, I downloaded PuTTY and built from sources. That produced a plink executable which I then set VISITSSH to point to. That started to work but produced this output on terminal from which I launched VisIt (I was launching from terminal instead of macOS Finder to allow me to easily set env. variables)...

The host key is not cached for this server:
  quartz.llnl.gov (port 22)
You have no guarantee that the server is the computer you
think it is.
The server's ssh-ed25519 key fingerprint is:
  ssh-ed25519 255 SHA256:LKUkgoAINmEYUSo6yaWWPAL0vCfQMeaQL5W4f6e9ft0
If you trust this host, enter "y" to add the key to Plink's
cache and carry on connecting.
If you want to carry on connecting just once, without adding
the key to the cache, enter "n".
If you do not trust this host, press Return to abandon the
connection.
Store key in cache? (y/n, Return cancels connection, i for more info) y
y: Command not found.

This did NOT work. It did not see my y that I typed at the terminal. So, then I tried ./plink as a bare app to connect. I then connected to quartz fine and accepted the host key. So, I re-tried VisIt again thinking that would get me passed the host key issue shown above. It did. But, PuTTY asks for another key press...

-- Keyboard-interactive authentication prompts from server: ------------------
| Password: 
-- End of keyboard-interactive prompts from server ---------------------------
Access granted. Press Return to begin session. 

To get it to stop asking for that, I needed to add -no-antispoof.

So, after all of that, I get this to work...

env VISITSSH=`pwd`/plink VISITSSHARGS=-no-antispoof /Applications/VisIt.app/Contents/MacOS/VisIt

And that connected using the PuTTY I just built.

However, VisIt still produces the Qt window to ask for the password entry. It does not delegate that to the terminal.

[biagas]

You ran into the same issues I discovered when I updated putty last week for our 3.3.2 release.
The password window is wired up differently on Windows than the other platforms, so I'm not surprised that the password window still popped up on your Mac. Without modifications to VisIt source at this point I am out of suggestions for now. Sorry about that.

[markcmiller86]

No worries. I just wanted to dig a little deeper to see what we see. Now that I know I can force VisIt to use some other ssh implementation and tooling, I am wanna see what else might be possible.

[michael-sandoval]

For the record, this has been fixed as of VisIt 3.4.1 :)