More than 44 (default) groups crashing #23624
Description
This problem applies to all OpenSSL3.x versions incl. "master".
When registering more than 44 default groups, e.g., via SSL_CTX_set1_groups, openssl crashes unpredictably during handshaking (gdb trace below).
This was encountered when adding standardized PQ KEM algorithms to oqsprovider pushing the supported KEM algorithm count to 52.
The problem can be "resolved" by increasing the size of the constant GROUPLIST_INCREMENT, e.g., to 60, but this is not advisable as the stack trace (collected on x86_64 Linux) indicates some memory corruption:
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352440704) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=140737352440704) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=140737352440704, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff7442476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007ffff74287f3 in __GI_abort () at ./stdlib/abort.c:79
#5 0x00007ffff7489676 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff75dbb77 "%s\n")
at ../sysdeps/posix/libc_fatal.c:155
#6 0x00007ffff74a0cfc in malloc_printerr (str=str@entry=0x7ffff75de210 "munmap_chunk(): invalid pointer") at ./malloc/malloc.c:5664
#7 0x00007ffff74a0fdc in munmap_chunk (p=<optimized out>) at ./malloc/malloc.c:3060
#8 0x00007ffff74a549a in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3381
#9 0x00007ffff7a87f5e in CRYPTO_free (str=0x5555557035f0, file=0x7ffff7f76fc0 "ssl/ssl_sess.c", line=842) at crypto/mem.c:282
#10 0x00007ffff7a87fb2 in CRYPTO_clear_free (str=0x5555557035f0, num=928, file=0x7ffff7f76fc0 "ssl/ssl_sess.c", line=842)
at crypto/mem.c:291
#11 0x00007ffff7ed2dc2 in SSL_SESSION_free (ss=0x5555557035f0) at ssl/ssl_sess.c:842
#12 0x00007ffff7f6d2de in construct_stateless_ticket (s=0x555555727fc0, pkt=0x7fffffffd230, age_add=427351503,
tick_nonce=0x7fffffffd1a0 "") at ssl/statem/statem_srvr.c:4015
#13 0x00007ffff7f6e136 in tls_construct_new_session_ticket (s=0x555555727fc0, pkt=0x7fffffffd230) at ssl/statem/statem_srvr.c:4287
#14 0x00007ffff7f4c52e in write_state_machine (s=0x555555727fc0) at ssl/statem/statem.c:894
#15 0x00007ffff7f4b6b0 in state_machine (s=0x555555727fc0, server=1) at ssl/statem/statem.c:487
#16 0x00007ffff7f4b143 in ossl_statem_accept (s=0x555555727fc0) at ssl/statem/statem.c:307
#17 0x00007ffff7ec5800 in SSL_do_handshake (s=0x555555727fc0) at ssl/ssl_lib.c:4735
#18 0x00007ffff7ebf313 in SSL_accept (s=0x555555727fc0) at ssl/ssl_lib.c:2177
#19 0x00005555555f3e3e in init_ssl_connection (con=0x555555727fc0) at apps/s_server.c:2967
#20 0x00005555555f385c in sv_body (s=4, stype=1, prot=0, context=0x0) at apps/s_server.c:2822
#21 0x000055555562a43c in do_server (accept_sock=0x55555567f4c0 <accept_socket>, host=0x0, port=0x5555556a2ef0 "33333", family=0, type=1,
protocol=0, cb=0x5555555f2552 <sv_body>, context=0x0, naccept=-1, bio_s_out=0x5555556aeca0, tfo=0) at apps/lib/s_socket.c:423
#22 0x00005555555f1ea1 in s_server_main (argc=13, argv=0x7fffffffdc10) at apps/s_server.c:2314
#23 0x00005555555d0c6f in do_cmd (prog=0x5555556a1170, argc=13, argv=0x7fffffffdc10) at apps/openssl.c:426
#24 0x00005555555d07f9 in main (argc=13, argv=0x7fffffffdc10) at apps/openssl.c:307
Tagging @mattcaswell as author of the PR introducing this constant and @romen as enabler of pluggable groups (how many groups should be sensibly supported?).
If desired (?) I can try to create a reproducer of this problem using tls-provider but the same can already and easily be achieved using oqsprovider (after open-quantum-safe/oqs-provider#348) has landed -- or quite possibly before by reducing the size of "GROUPLIST_INCREMENT").