A couple of weeks ago, you mentioned that OpenSSL project could be interested in implementing a Pluggable Signatures mechanism. Such a mechanism potentially allows users to use custom signature schemes (e.g., post-quantum) in TLS without libssl modification. As we mentioned, we are willing to take part in this activity. In this issue, we would like to propose two possible Pluggable Signatures mechanism implementation approaches and to get some feedback from you.

Allowing custom sigalgs

In this implementation approach, we propose to use not only built-in sigalgs, but also to add a custom sigalgs list. Signatures from such a list should be treated as supported and allowed in TLS. We also assume a simple API to add/remove elements of this list here.

TLS extension

In this implementation approach, we propose to implement a TLS extension specification. Such an extension could keep an additional sigalgs or a list of sigalgs to be checked. For instance, we could use something similar to the Extension For Establishing An Additional Shared Secret. We propose to treat the aforementioned extension as built-in (its definition should be present in ext_defs (link)). However, we think it would be better to hide it behind the enable_custom_sigalgs runtime option. We propose to pass a list of sigalgs IDs to be checked via the new OpenSSL configuration option (in the configuration file).

In both proposals, we assume the development and integration of an API to map additional sigalgs NIDs to the corresponding TLS IDs (see SignatureScheme, link).

Among the benefits of the first approach, we would like to emphasize its simplicity and the absence of the necessity of TLS state machine modification. Also, it does not contradict the TLS standard. On the other hand, the second approach seems more flexible for us. In this case, we can check multiple sigalgs instead of one as in the previous method. To our point of view, this is the decisive advantage, even considering the contradiction to the TLS specification.

We would be grateful to hear your opinion on the described approaches.