As we know, SSL/TLS services two primary roles: encryption and authentication. Using both symmetric and asymmetric cryptography, we are able to send a secure message over an insecure channel. Through certificate validation, not only are we able to send a secure message across an insecure channel, but we also know that the party on the other end is who they claim to be. But what mechanism allows for this assurance? Certificate authorities are a trusted third party who issue digitally signed certificates after performing their due diligence. For you to see a green lock icon instead of being presented with an error, the public node certificate served by a website must be able to be “chained” back through digital signatures to a root certificates which the client is already aware of. Operating systems and browsers ship with a number of these root certificates, and endpoint administrators can also add their own.
If the node certificate is directly signed by a root CA using the corresponding private key to one of the public keys the client is directly aware of, then the path validation logic is complete. What if however, (as if often the case), one of those root CAs has delegated their authority to a DIFFERENT root certificate? This is perhaps the most important thing for us to understand about intermediate certificates: intermediate certificates ARE root certificates, but in a specific path from client to root CA, they perform a specific role. Examine the screenshot below.
“GTS CA 1O1” is in fact a root certificate in its own right. All intermediate certificates are. If I were to remove “Google Trust Services – GlobalSign Root CA-R2” from my endpoint’s root certificate store and add “GTS CA 1O1”, the path would be equally valid, but contain only two certificates – “GTS CA 1O1” and the www.google.com leaf certificate.
Can we have multiple intermediate certificates between server and client? Absolutely. 4 or even 5 certificates in a certificate chain is an extremely common occurrence.
Can we have multiple valid paths between a root certificate and a node certificate? Again, yes. Trivially, if I were to explicitly add “GTS CA 1O1” to my root certificate store without removing the existing trusted certificate, both of those paths would be valid, and any software could rightfully stop attempting to construct a valid path and return success. What happens though if we have multiple valid paths and a certificate expires from one path but not the other, leaving a single valid path? I have seen this edge case break many a server over the years. Software which properly handles this corner case will continue attempting to construct a valid path even after it runs across a previously valid path, but frequently (especially with legacy software), it returns a failure. The behavior above is often relied upon when certificate authorities strategically make a series of chess moves to phase out older CAs without unintentionally affecting other valid paths. (See Understanding Certificate Cross-Signing).
You might be asking yourself at this point whether it is the server’s responsibility to send intermediate certificates or the client’s responsibility to be aware of them via its root trust store? It is a mix of both. Often if an administrator does not configure the certificate served in the server software to include an intermediate, the client will already be aware of the intermediate and trust the site regardless. However, this should not be relied upon due to the uncontrollable nature of unmanaged endpoints. It is a best practice to serve all intermediate certificates for the path you most desire clients to take in order to trust your server. Note that it is a common misconfiguration to also include the root certificate in the list of certificates served. While it is a mostly benign error, it does increase the amount of traffic in every request. There is never a good reason to serve the root certificate.
Troubleshooting certificate errors can be difficult. Error messages don’t always make intuitive sense, especially when dealing with older software. One of the best ways for troubleshooting is to first figure out what the client is using for its root certificate store, and then to emulate connecting to the endpoint via CLI. For windows systems running .NET applications as the client, this simply means running OpenSSL’s SClient from the machine and reviewing the results.
It’s worth noting that two variants of this command exist. Newer versions of OpenSSL support SNI which allows for a request to include a host header specifying a desired site for the other end to load. This is useful so that many sites can share the same web server and port. (It serves as a selector).
Without SNI:
shell
openssl s_client -connect www.google.com:443
With SNI:
shell
openssl s_client -connect www.google.com:443 -servername google.com
Other client software can be trickier. For example, let’s say the client in the transaction is a program running java. Java does not provide an easy command line tool for this testing. You can write and compile a small java program if you’re so inclined, or you can take the CACerts file from a live java application, and use keytool to export all of the certificates to your file system in PEM format. From there, openssl’s s_client supports the -cafile argument for explicitly providing trusted certificates to the utility.
For example:
shell
keytool.exe" -list -rfc -keystore "C:\Scratch\ca\cacerts" >> "C:\Scratch\ca\MyFile.pem"
(if prompted for a password, the default cacerts keystore password is “changeit” without the quotes)
We need to do some post processing in order to remove content from our file other than the PEM formatted certificates output from the keystore.
You could go through manually and remove everything that’s not between the Begin Certificate and End Certificate line, but I’ve also included some handy powershell at the end of this article to perform this tedious work for us.
Note the difference visually between a successful connection with a valid certificate path:
And a failed certificate chain verification:
text
## Input is the output of the keytool command keytool.exe" -list -rfc -keystore "C:\Scratch\ca\cacerts"
$var = gc "C:\Scratch\ca\MyFile.pem"
## Output will become the input to the OpenSSL S_Client’s CAfile parameter
$outputFile = "C:\Scratch\ca\PemCerts.pem"
$contentArray = @()
$contentArray = $var.split("`r`n")
$capture = $false
foreach ($line in $contentArray) {
if ($line -eq '-----BEGIN CERTIFICATE-----') {
#$certContent += '-----BEGIN CERTIFICATE-----'
$capture = $true
}
if ($line -eq '-----END CERTIFICATE-----') {
$capture = $false
$certContent += '-----END CERTIFICATE-----'
$certContent >> $outputFile
$certContent = ""
}
if ($capture -eq $true) {
$certContent += $line
$certContent += "
"
}
}
Discussions and Comments
Click here to view and join in on any discussions and comments on this article.
