Hostnames with underscores fail SSLContext hostname verification with wildcard certificates #103490
Description
Bug report
There is a debate about whether DNS A records support underscores in the hostname portion. But, they appear to work and modern operating systems seem to support them.
While test_ssl.py has tests for ssl.match_hostname(), that function is never called (they were removed in newer Python versions) when doing hostname verification using SSLContext with check_hostname = True. This is hidden in most things using urllib3 as before the upcoming 2.0.0 version, they do their own hostname checking and do not hit this issue. Version 2.0.0 removes their self-checking and it too experiences the issue described herein.
Use of Google's *.a.run.app in the examples is for convenience only.
in a fresh python:3.11 container:
with aiohttp:
# python
Python 3.11.3 (main, Apr 12 2023, 14:31:14) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio, aiohttp
>>> async def test():
... async with aiohttp.ClientSession() as session:
... await session.get(
... "https://foo_bar.a.run.app/",
... )
...
>>> loop = asyncio.get_event_loop()
>>> loop.run_until_complete(test())
...
Traceback (most recent call last):
...
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host foo_bar.a.run.app:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'foo_bar.a.run.app'. (_ssl.c:1002)")]
with requests and the 2.0.0 branch of urllib3:
# pip install urllib3==2.0.0a3 requests==2.28.2 --no-deps
...
Successfully installed requests-2.28.2
...
Successfully installed urllib3-2.0.0a3
# python
Python 3.11.3 (main, Apr 12 2023, 14:31:14) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
/usr/local/lib/python3.11/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (2.0.0a3) or chardet (None)/charset_normalizer (3.1.0) doesn't match a supported version!
warnings.warn(
>>> r = requests.get('https://foo_bar.a.run.app')
Traceback (most recent call last):
...
requests.exceptions.SSLError: HTTPSConnectionPool(host='foo_bar.a.run.app', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'foo_bar.a.run.app'. (_ssl.c:1002)")))
comparison to curl (browsers appear to trust it as well):
# curl -I -X GET 'https://foo_bar.a.run.app'
HTTP/2 404
content-type: text/html; charset=UTF-8
referrer-policy: no-referrer
content-length: 1561
date: Wed, 12 Apr 2023 22:12:20 GMT
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
# curl -I -X GET 'https://foo_bar.bad.a.run.app'
curl: (60) SSL: no alternative certificate subject name matches target host name 'foo_bar.bad.a.run.app'
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
A DNS lookup:
00000000 a2 80 01 20 00 01 00 00 00 00 00 01 07 66 6f 6f ... .... .....foo
00000010 5f 62 61 72 01 61 03 72 75 6e 03 61 70 70 00 00 _bar.a.r un.app..
00000020 01 00 01 00 00 29 10 00 00 00 00 00 00 00 .....).. ......
00000000 a2 80 81 80 00 01 00 04 00 00 00 01 07 66 6f 6f ........ .....foo
00000010 5f 62 61 72 01 61 03 72 75 6e 03 61 70 70 00 00 _bar.a.r un.app..
00000020 01 00 01 c0 0c 00 01 00 01 00 00 4f b3 00 04 d8 ........ ...O....
00000030 ef 22 35 c0 0c 00 01 00 01 00 00 4f b3 00 04 d8 ."5..... ...O....
00000040 ef 26 35 c0 0c 00 01 00 01 00 00 4f b3 00 04 d8 .&5..... ...O....
00000050 ef 24 35 c0 0c 00 01 00 01 00 00 4f b3 00 04 d8 .$5..... ...O....
00000060 ef 20 35 00 00 29 10 00 00 00 00 00 00 00 . 5..).. ......
A less-secure workaround here appears to be to pass a custom SSLContext, but that seems to be bad as you have to disable hostname checking. Additionally, setting hostname_checks_common_name = True appears to not be a solution / has no effect even when the CN is the valid wildcard name.
With aiohttp that looks like this:
async def test():
context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
context.verify_mode = ssl.CERT_REQUIRED
context.check_hostname = False
context.load_default_certs()
async with aiohttp.ClientSession() as session:
await session.get(
"https://foo_bar.a.run.app/",
ssl=context,
)Your environment
I've tested this on multiple operating systems (Linux + MacOS) and from Python 3.7 to the latest 3.11.