We enabled SchUseStrongCrypto in the registry on a Windows Server 2008 R2 server. After the change, we noted that HTTP requests to another, third-party server made using .NET HttpClient no longer worked. The registry change altered the protocol used from TLS 1.0 to TLS 1.2. Schannel logged the following error to the system event log: "The following fatal alert was generated: 40. The internal error state is 252."
Some research with Wireshark revealed a situation very similar to the one outlined in this blog post: our server was sending a TCP RST after receiving the remote server certificate, presumably because the root certificate sent by the remote server uses MD5 as its hash algorithm and no signature algorithms using MD5 were listed in our ClientHello. By disabling TLS 1.2 in the registry (the signature_algorithms extension is specific to TLS 1.2), we were able to force the connection to be made over TLS 1.1 and eveything worked happily.
However, there are several aspects of the situation I don't understand. When running the same application on a Windows Server 2016 Technical Preview 2 server with SchUseStrongCrypto also enabled, the connection was made using TLS 1.2 just fine, despite the same signature algorithms being sent in the ClientHello, albeit in a different order. Both servers have .NET 4.6 installed and the application targets 4.6.
Why did the connection work on one server and not the other, despite both saying they wouldn't accept MD5 certs? Additionally, Wireshark revealed that on the 2008 R2 server Internet Explorer 11 (which is not governed by the .NET SchUseStrongCrypto flag but which uses Schannel) is also failing to make the initial TLS 1.2 connection, but rather than sending a RST it sends a FIN and then tries and successfully makes a TLS 1.0 connection (on the 2016 server IE11 uses TLS 1.2 successfully)--what governs whether/how HttpClient performs the same fallback? Finally, how are signature algorithms specified and ordered (i.e. like you can select and order cipher suites in gpedit.msc), and what should the client's behavior be when it receives a certificate that doesn't match any of the signature algorithms it said it was willing to use?
An SSL Labs report on the remote server revealed that everything except IE 6 (as expected) successfully negotiated with it, with Windows 8.1+ and OS X platforms using TLS 1.2. Another thing I noticed from the report is that the remote server's certificate chain actually includes two self-signed certs, the more proximal of which uses SHA1 rather than MD5, so there are two "paths". The first path is reported as "trusted" because the first self-signed cert is "in trust store"; the second is "not trusted" because the problematic MD5 cert, which is the ultimate root of the chain, is "not in trust store". This implies to me that other HTTP clients, like the browsers that SSL Labs tests, use the first path and ignore the MD5 cert.
Ultimately, this seems to be due to some fundamental difference between the two versions of Windows (maybe in Schannel), but is that difference documented or configurable anywhere?