> The CLI Guy 

Please contact your system administrator...

SSH Protocol - RSA Signature Algorithms

14 Oct 2022  ·  Comments:  ·  Tags: SSH, OpenSSH, security, ssh-audit

Summary

I’ve been working on a project to phase-out the use of the ssh-rsa signature algorithm in an environment where hundreds of automated file transfers are performed using SFTP, the standard file transfer protocol for use with SSH. The motivation for doing so being that ssh-rsa (as a signature algorithm) uses sha-1 hashing and has therefore been deemed deficient according to RFC 8332 and its usage was deprecated in OpenSSH 8.8.

The objective was that all SFTP components (client and server) should continue to use existing RSA keys but signing should be performed with rsa-sha2-256 or rsa-sha2-512 (instead of ssh-rsa) which as the names imply use sha-256 and sha-512 respectively from the sha-2 family of cryptographic hash functions.

What follows is not a step-by-step guide on how to rid your SSH environment of the ssh-rsa signature algorithm but rather a series of notes presented in a logical order that provide relevant information, observations and guidance.

Key Pair Uses

An RSA key pair is comprised of a public and a private key.

Encryption - A public key can be used to encrypt data and the corresponding private key used to decrypt it. Public key cryptography (PKC) isn’t intended to encrypt large volumes of data, therefore it’s common to use symmetric cryptography (such as AES) to encrypt the actual payload and use RSA to encrypt the symmetric key (SSH is no exception to this way of working).

Signing - A digital signature can be produced to accompany data to be exchanged by hashing the data and then signing the resultant hash (aka “digest”) with a private key. The recipient uses the same hashing algorithm to independently generate a hash of the data. The recipient then verifies the signature using the sender’s public key to obtain the hash that was generated by the sender and compares it against their own hash of the data for equality. If verification succeeds then it proves that the sender is in possession of the private key that corresponds with the public key and that the data has not been tampered with since production by the sender. I won’t be going into any specific details about what the signed block of data is comprised of in SSH.

Of the two uses listed above, signing is the only one of relevance in relation to the subject of signature algorithms.

Terminology

The algorithms ssh-rsa, rsa-sha2-256 and rsa-sha2-512 are examples of public key algorithms. I was initially confused by the term “public key algorithm” because it caused me to think it was perhaps something that’s concerned exclusively with the public component of a key pair, but I’ve since concluded that the use of “public key” in this context derives from public-key cryptography, which is just a synonym for asymmetric cryptography.

Here’s where I think the terminology becomes a tad ambiguous… In SSH, when a public key algorithm is used with a host key pair for server authentication (where a client verifies the authenticity of a server it’s connecting to), it tends to get referred to as a “host key algorithm”. Whereas, there’s no contextually sensitive equivalent term that’s applied when a public key algorithm is used with an authorized user key pair for client authentication (where a server verifies the authenticity of a user that wants to connect), it’s still just known as a “public key algorithm”. To illustrate this point, OpenSSH (client and server) uses the option names HostKeyAlgorithms and PubkeyAcceptedAlgorithms (or PubkeyAcceptedKeyTypes prior to OpenSSH 8.5).

Since the specific purpose served by these public key algorithms in SSH is the production and verification of digital signatures, they may also be referred to as “signature algorithms”, which is a term that I’ve used throughout this post.

Discovery and Negotiation

The signature algorithms ssh-rsa, rsa-sha2-256 and rsa-sha2-512 can be used for the production and verification of digital signatures with a host key pair during server authentication and an authorized user key pair during client authentication. The process of discovery and negotiation of signature algorithms in SSH differs for server authentication and client authentication.

Server Authentication

Since the server holds the private key and client holds the corresponding public key, the server signs and the client verifies.

The client and server both send a SSH_MSG_KEXINIT message to each other, containing lists of acceptable algorithms. The first preferred algorithm in each of the client’s lists (where the order signifies most to least preferred) that’s supported by the server is selected. One of those lists is server_host_key_algorithms (see page 17 of RFC 4253).

With server authentication, it can be argued that the burden of risk associated with using the sha-1 hashing employed by ssh-rsa falls on the client and not the server, since the client is trying to establish the legitimacy of the server.

Client Authentication

Since the client holds the private key and server holds the corresponding public key, the client signs and the server verifies.

Prior to the introduction of rsa-sha2-*, it was possible to determine the appropriate choice of signature algorithm from the key type in use, since key types and signature algorithms corresponded one-to-one.

RFC 8308 defines an extension mechanism for SSH, enabling peers to signal support for and negotiate certain capabilities. One of the extensions to use this mechanism is server-sig-algs, where a server provides a list of signature algorithms that it supports to a client. In this particular interaction, the client begins by including ext-info-c in the kex_algorithms field of a SSH_MSG_KEXINIT message that’s sent to a server and prepares but does not require the server to respond with an SSH_MSG_EXT_INFO message containing server-sig-algs. I say “does not require” because of course at this stage the client has no idea if the server supports the extension.

If a server fails to respond with “server-sig-algs” then my understanding is that how a client decides to proceed is a matter of discretion for the developer. One possible approach is for the client to implicitly use ssh-rsa because it assumes that the absence of server-sig-algs indicates that a server will not support rsa-sha2-256 and rsa-sha2-512. I suppose this is a reasonable approach because if a server uses RSA authorized user keys for client authentication then it must support at least one RSA signature algorithm and it’s perhaps unlikely that a developer would go to the trouble of implementing rsa-sha2-* without also implementing “server-sig-algs” to signal its presence.

With (key-based) client authentication, it can be argued that the burden of risk associated with using the sha-1 hashing employed by ssh-rsa falls on the server and not the client, since it’s the server that’s trying to establish the identity of the client.

Auditing

When working with SSH client or server software, it’s sometimes difficult to establish whether configuration changes have had the desired effect and furthermore, some products lack transparency, using hardcoded and/or undocumented settings. Also, you may find yourself needing to verify settings on a server to which you have no administrative access. It’s therefore useful to have some means of auditing a client and/or server to determine which signature algorithms are being advertised as supported.

Server Authentication

ssh-audit can be used to scan both clients and servers to determine which host key algorithms are supported.

Also, the OpenSSH client can be used to obtain the host key algorithms a server supports, EG:

# Obtain a server's host key algorithms.
$ ssh TEST@192.168.0.1 -vv 2>&1 | grep "host key algorithms" | tail -n1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-ed25519

Client Authentication

The OpenSSH client can be used to obtain the algorithms advertised by a server in server-sig-algs:

# Obtain the signature algorithms a server claims to support for client authentication.
$ ssh TEST@192.168.0.20 -v 2>&1 | grep "server-sig-algs"
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>

NB: Frustratingly, the values contained within server-sig-algs can’t always be trusted to truly reflect what a target server supports, continue reading to the end of this post for further details.

As for auditing client software, the way ssh-audit performs client audits is to operate as a faux SSH server, have the client connect to it and then extract the various algorithms from the SSH_MSG_KEXINIT that took place. If ssh-audit shows that ext-info-c is absent from the key exchange algorithms then it’s reasonable to assume that the client is limited to supporting ssh-rsa only. If however ext-info-c is present then it is reasonable to assume that the client is technically capable of supporting rsa-sha2-256 and/or rsa-sha2-512 but we can’t determine if its one, or the other, or both. It is also a possibility that even though a client is technically capable of supporting rsa-sha2-256 and/or rsa-sha2-512, usage could have been disabled (inadvertently or intentionally) in the client’s settings.

So, if ext-info-c is present, how can we really be sure that the client is configured to use rsa-sha2-256 or rsa-sha2-512? One approach (requiring access to a server) would be to configure a server to use only rsa-sha2-256, have the client connect to it and then do the same again for rsa-sha2-512. The client will raise an error and disconnect if it doesn’t support the algorithm on the server. Exactly how that failure manifests itself depends on decisions taken by the developer of the client software. A client might disconnect on discovering that there are no mutually agreeable algorithms present in server-sig-algs, or it might proceed a bit further before disconnecting.

Additionally, you can inspect the resultant logs produced by the client and/or the server, one or both of which may reveal the choice of algorithm. Using OpenSSH as an example… The OpenSSH client will show its choice of algorithm in the log when run with maximum verbosity (-vvv), EG: debug3: sign_and_send_pubkey: signing using rsa-sha2-256. The OpenSSH server (which sends logging information to the system logs [/var/log/auth.log on Debian and Ubuntu]) records the client’s choice of algorithm, whether it succeeds (requires a minimum debug level of 2) or fails, EG:

thecliguy@SANDBOX:~$ sudo tail -f /var/log/auth.log | grep userauth_pubkey

Sep 30 16:31:33 SANDBOX01 sshd[620769]: debug2: userauth_pubkey: valid user fred attempting public key rsa-sha2-512 <REDACTED> [preauth]
Sep 30 16:31:33 SANDBOX01 sshd[620769]: debug3: userauth_pubkey: have rsa-sha2-512 signature for RSA SHA256:<REDACTED> [preauth]
Sep 30 16:31:33 SANDBOX01 sshd[620769]: debug2: userauth_pubkey: authenticated 1 pkalg rsa-sha2-512 [preauth]
thecliguy@SANDBOX:~$ sudo tail -f /var/log/auth.log | grep userauth_pubkey

Sep 30 16:35:08 SANDBOX01 sshd[620775]: debug2: userauth_pubkey: valid user fred attempting public key ssh-rsa <REDACTED> [preauth]
Sep 30 16:35:08 SANDBOX01 sshd[620775]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedKeyTypes [preauth]
Sep 30 16:35:08 SANDBOX01 sshd[620775]: debug2: userauth_pubkey: authenticated 0 pkalg ssh-rsa [preauth]

OpenSSH Server - Misleading Values in server-sig-algs

I configured OpenSSH server on an Ubuntu 20.04.4 LTS server (where the version of OpenSSH server was OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020) to use only rsa-sha2-512 as the permitted signature algorithm for key-based client authentication:

$ grep PubkeyAcceptedKeyTypes /etc/ssh/sshd_config
PubkeyAcceptedKeyTypes rsa-sha2-512

I then used the OpenSSH client to check which algorithms the server was returning in server-sig-algs. To be thorough, I did so from an Ubuntu 20.04.2 LTS server with OpenSSH client version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020 and also from a Windows 10 machine using the version of OpenSSH that ships with Windows 10 21H2 which is OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2, a downloaded beta version OpenSSH_for_Windows_8.9p1, LibreSSL 3.4.3 and OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f 31 Mar 2020 on an Ubuntu 20.04.4 LTS instance of Windows Subsystem for Linux. In all cases, the result was exactly the same, which to my surprise was that the server was advertising many more algorithms that just rsa-sha2-512:

For the sake of brevity, since there was no significant difference in behaviour between all the different versions of OpenSSH client, I’ll only show the output returned by the version that ships with Windows 10 (21H2).

C:\> ssh TEST@192.168.0.1 -v 2>&1 | findstr "server-sig-algs"
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com>

The OpenSSH client allows the user to be explicit in which algorithms to use, where the signature algorithm for key-based client authentication is specified using PubkeyAcceptedAlgorithms (or PubkeyAcceptedKeyTypes prior to OpenSSH 8.5). I tried to connect to the server using rsa-sha2-256 which I hadn’t assigned to PubkeyAcceptedKeyTypes in /etc/ssh/sshd_config but it was being advertised in server-sig-algs and this resulted in failure:

C:\> ssh TEST@192.168.0.1 -vvv -i "C:\Bitbucket\TEST.priv" -o PubkeyAcceptedKeyTypes=rsa-sha2-256 exit
...
debug3: sign_and_send_pubkey: signing using rsa-sha2-256
...
debug1: Authentications that can continue: publickey
...
debug1: No more authentication methods to try.
TEST@192.168.0.1: Permission denied (publickey).

Next, I tried to connect again but this time using rsa-sha2-512 which was present as a PubkeyAcceptedKeyTypes value in /etc/ssh/sshd_config and also present in server-sig-algs, this worked:

C:\> ssh TEST@192.168.0.1 -vvv -i "C:\Bitbucket\TEST.priv" -o PubkeyAcceptedKeyTypes=rsa-sha2-512 exit
...
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
...
debug1: Authentication succeeded (publickey).
Authenticated to 192.168.0.1 ([192.168.0.1]:22).

I then proceeded to enable both rsa-sha2-256 and rsa-sha2-512 as PubkeyAcceptedKeyTypes in /etc/ssh/sshd_config on the server:

$ grep PubkeyAcceptedKeyTypes /etc/ssh/sshd_config
PubkeyAcceptedKeyTypes rsa-sha2-256,rsa-sha2-512

Finally, I tried to connect again, this time using rsa-sha2-256 and it worked:

C:\> ssh TEST@192.168.0.1 -vvv -i "C:\Bitbucket\TEST.priv" -o PubkeyAcceptedKeyTypes=rsa-sha2-256 exit
...
debug3: sign_and_send_pubkey: signing using rsa-sha2-256
...
debug1: Authentication succeeded (publickey).
Authenticated to 192.168.0.1 ([192.168.0.1]:22).

The behaviour exhibited showed that PubkeyAcceptedKeyTypes in /etc/ssh/sshd_config was ultimately adhered to but it had no influence on what was being advertised in server-sig-algs. I managed to find several related bug reports, see Debian Bug report logs - #998619 : openssh-server: server-sig-algs, sshd_config: PubkeyAcceptedKeyTypes option does not seem to have any effect and Bug 3348 - Not possible to disable rsa-sha2-512 in sshd . All bug reports make reference to a comment that accompanies the code responsible for populating server-sig-algs, the significance of which became apparent to me after consulting the Tags section of the Wikipedia article on programming language comments, which explained that the use of three contiguous X’s serves to “warn other programmers of problematic or misguiding code”.

This is rather frustrating from an auditing perspective because we can’t necessarily trust the values in server-sig-algs that’s supplied by an OpenSSH server. I can’t think of a workaround that doesn’t require having an account that uses an RSA authorized user key on a target server. If you did have such an account on a target server then you could adopt a trial and error approach, where a client makes three connections to the server and on each occasion switches between using ssh-rsa, rsa-sha2-256 and rsa-sha2-512 and then note down which connections succeed and which fail.

Pace of Adoption

According to the IETF Datatracker, RFC 8308 and RFC 8332, were submitted as Internet Drafts in November 2015 and on 2018-03-19 they were both published as RFCs. Since RFC 8332 introduced two new algorithms (rsa-sha2-256 and rsa-sha2-512) these needed to be recorded in the register of Secure Shell (SSH) Protocol Parameters that’s maintained by the IANA (Internet Assigned Numbers Authority). The register lacks a changelog but it does have a “last updated” date. Some sleuthing on the Wayback Machine suggests that the first appearance of the new signature algorithms in the register was 2018-01-03 but referencing the RFC in a draft state and then the next updated appearance with a reference to the ratified RFC was on 2018-03-22 (three days after the RFC was published).

It was no surprise to see that Bitvise Limited were early adopters given that the RFCs were written by an employee of Bitvise, see the version history for client and server 7.12, released on 2016-06-25. OpenSSH were also early adopters, in version 7.2/7.2p1 on 2016-02-29 (see also Release Notes). I thought it nice that OpenSSH made reference to the specific draft versions of the RFCs they were working from, draft-rsa-dsa-sha2-256-03.txt and draft-ssh-ext-info-04.txt. There will no doubt have been other early adopters too.

In contrast to Bitvise and OpenSSH, I’ve encountered various SSH/SFTP client and server products that hadn’t implemented these RFCs into a stable release until this year (2022) and I imagine there may be others that have yet to take action. This is not intended as a criticism, I’m simply making the point that different projects move at different paces. It’s something that must be taken into consideration by anyone that’s attempting to eradicate use of the ssh-rsa signature algorithm from their environment. Let’s say for example you’re the administrator of an SFTP server and like a good admin you’ve successfully upgraded to a version of the software that adds support for rsa-sha2-* signature algorithms (for server and client authentication). You may not actually be able to switch off support for ssh-rsa until you’ve confirmed that all the clients that connect to your server are capable of using rsa-sha2-* (or some suitable alternative to RSA if your server offers such a thing).

Relevant RFCs

In addition to the RFCs, I also found the following to be a useful source of information:

Comments

Leaving comments has been disabled for this post.

Copyright © 2018 - 2022 thecliguy.co.uk
For details, see Licences and Copyright