Beware of Unverified TLS Certificates in PHP & Python

Web developers today rely on various third-party APIs. For example, these APIs allow you to accept credit card payments, integrate a social network with your website, or clear your CDN’s cache. The HTTPS protocol is used to secure the connection with the API server. However, if your web app doesn’t verify the TLS certificate, a malicious person can steal your passwords or your customers’ credit card numbers.

When implemented correctly, the TLS protocol provides both encryption and authentication. The connection between your server and the API server is encrypted using a symmetric cipher (typically AES) so an eavesdropper cannot read your data. The server also confirms its identity (authenticates itself) by sending an X.509 certificate to the client. The client must verify the certificate’s signature against the list of known root certificates, but this step is often neglected. As a result, a man-in-the-middle attack becomes possible.

If you don’t verify the certificate, the attacker can masquerade as the API server, intercept data sent in both directions, or even return false messages that the API server never sent to you. This attack was previously discussed in the paper The Most Dangerous Code in the World: Validating SSL Certificates in Non-Browser Software by Martin Georgiev and others. The authors found that several API client libraries written in Java and PHP don’t verify the certificates correctly, so they are vulnerable to the attack. The authors tested these client libraries with a self-signed certificate and a valid certificate belonging to another domain name. They consider counter-intuitive SSL API (for example, CURLOPT_SSL_VERIFYHOST in cURL) and insecure SSL libraries (the fsockopen function in PHP) to be the root of the problem.

If you transmit any personal or financial data via HTTPS, please make sure that the TLS certificate is verified correctly. Two years ago, IOActive tested 40 mobile banking apps and found that 40% of them are vulnerable to the MITM attack. Another group of researchers from Leibniz University of Hanover and Philipps University of Marburg found that 8% of popular Android apps fail to verify certificates. A passive MITM attack against these mobile apps is very real when you use a public WiFi hotspot. The attack is also possible in case of a web server accessing a third-party API.

PHP 5.6 fixes some of the certificate verification problems. Python does the same in the versions 3.4.3 and 2.7.9. I tested the new versions to see what was fixed and what was not. I also tested revoked and expired certificates, and certificate pinning (not included in the research by Georgiev et al.)

Tests

Let’s use the test HTTPS servers (working at the time of writing) that trigger a security error in any modern browser:

• https://revoked.grc.com (revoked certificate, see GRC Revocation Awareness Test);
• https://tv.eurosport.com (the domain name does not match, see SSL Test Center by NetLock Ltd.);
• https://qvica1g3-e.quovadisglobal.com (expired certificate, see QuoVadis Test Certificates);
• https://self-signed.badssl.com (self-singed certificate, see BadSSL.com);
• https://rc4.badssl.com/ (outdated RC4 cipher);
• https://dh480.badssl.com/ (a weak Diffie-Hellman key).

The BadSSL.com site contains other test cases that may be useful for testing your web apps.

The test scripts connect to each server using a default TLS configuration. You can run these scripts on your installation to see if you have problems.

All programming language implementations fail to check if a certificate is revoked.

TLS implementation in PHP 5.5 and below is broken when you use stream functions (fsockopen, fopen, stream_socket_client, stream_socket_enable_crypto, or file_get_contents). You should use cURL functions instead and upgrade to PHP 5.6 whenever possible. Note that PHP 5.5 allows the outdated RC4 cipher even when using cURL.

For Python, the situation is even more complicated and badly documented. The best solution is upgrading to the versions 3.4.3 and 2.7.9 or higher and always using the context parameter.

Recommended TLS Configuration for PHP

If you can install a newer PHP version on your server, please upgrade to PHP 5.6 or higher. It will solve all verification problems except revoked certificates.

If you don’t control the server software (for example, you are running on a shared hosting, or your PHP application is used by thousands of people spread all over the globe), please use the cURL library with the following options:

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);

These options are already set to the correct values as of cURL 7.10, but please set these options explicitly for the older versions.

HTTPS streams in PHP 5.5 or below are insecure and should never be used. You can reject self-signed certificates by setting the ‘verify_peer’ context option to TRUE, but the attacker still can use a certificate issued for another domain. Some certificate authorities offer free DV certificates today, so it will not cost the attacker anything.

If your code is synchronous, it’s possible to replace the stream functions (fsockopen and others) with the cURL functions. If you are doing non-blocking (asynchronous) stream I/O using TLS or HTTPS transport, please upgrade to PHP 5.6 because it’s insecure in earlier versions.

TLS Certificate Verification in Python

Old Python versions (below 2.7.9 or 3.4.3) perform no certificate verification.

There is a third-party library, Requests, that improves the situation for some versions (e.g., 3.3.0 in the test). However, it does not solve all problems: weak DH keys are still allowed. When using Requests with Python below 2.7.9, you should install additional libraries (ndg-httpsclient and libffi); without these libraries, Requests fails to reject self-signed or expired certificates. A better solution is upgrading both Python and Requests to the latest versions.

The new Python versions are supposed to fix the TLS verification problems, but there are some caveats. If you use Python 2.7.9 or higher, the certificate will be verified by default:

f = urlopen(url)

However, this code mistakenly allows any certificate under Python 3.4.3 or higher. With Python 3.x, you must pass the context parameter to verify the certificate:

f = urlopen(url, context = ssl.create_default_context())

The same applies to the http.client.HTTPSConnection constructor. This fact is never mentioned in the documentation. The change log says that the certificate is verified “by default”; the library reference is silent, too.

Python TLS implementation also allows weak Diffie-Hellman keys and the outdated RC4 cipher in several Python versions (see the table above).

Certificate Pinning

It’s possible to strengthen the TLS security by allowing a limited number of certificates. For example, if an API server should always return the same certificate, you can hard-code it in your web application.

PHP 5.6 and higher provides the peer_fingerprint context option for this purpose (unfortunately, they use a weak SHA-1 hash). When using cURL, you can pass the CURLINFO_CERTINFO parameter to extract the certificate:

if (defined('CURLOPT_CERTINFO')) {
    curl_setopt($ch, CURLOPT_CERTINFO, TRUE);
}

curl_exec($ch);

if (defined('CURLINFO_CERTINFO')) {
    $certinfo = curl_getinfo($ch, CURLINFO_CERTINFO);
    echo $certinfo[0]['Cert'];
}

The certificate can then be hashed and compared (using hash_equals) with the known value. Note that this code requires cURL 7.19.1 or higher with OpenSSL (for example, it will not work under OS X). In Go, you can retrieve the certificate from the Response.TLS.VerifiedChains array. In Python, this task involves using the lower-level ssl module; urllib has no documented API for pinning or retrieving the certificate.

Twitter recommends another approach for their APIs: verify against the minimum number of root certificates. Twitter’s CDN uses several certificates that are signed by Symantec/Verisign or Digicert, so it’s not possible to pin a single certificate. In this case, you can build a custom root certificates file, then verify against these root certificates only. It’s possible with PHP (see the CURLOPT_CAINFO option), Python, and Go.

Revocation checks

PHP, Python, and Go perform no revocation checks by default, neither does the cURL library. If the certificate was compromised and revoked by the owner, you will never know about it.

Writing your own revocation check code is hardly realistic; you would need a skilled cryptographer for this task. PHP has the CURLOPT_CRLFILE option, but you would have to download the CRL file and verify its signature yourself. OCSP stapling is not supported in PHP. Go only returns a raw stapled OCSP response in Response.TLS.OCSPResponse. A lot of work would be needed to build a revocation check function from these primitives.

Conclusion

The Most Dangerous Code in the World paper was published in 2012. Since then, PHP fixed most of the TLS problems; Python still has a non-intuitive API and allows outdated ciphers.

If you want to avoid these problems in your code, you should upgrade to the latest PHP version and test your application against BadSSL.com or the test pages provided by CAs.