WHIP from OBS v30.1.2 on Windows/Apple platforms fails to establishe DTLS session with "Bad certificate" reply from OBS

[ianzag]

Dear developers,

I have an issue with recently released OBS v30.1.2 and WHIP Streaming. I'm trying to broadcast media stream to WHIP compatible server. Problem is DTLS handshake fails with Windows/Apple OBS official builds but can be successfully established with Linux official build. Not much in OBS logs in both cases, but I got SDP exchange and DTLS network traffic traces. Looks like depending on OBS's platform (and used TLS engine?) it behaves slightly different. Please see attached traces. Will be glad to supply any additional info. Thanks!
Bad_DTLS.txt
Bad_SDP.txt
Good_DTLS.txt
Good_SDP.txt

[paullouisageneau]

It looks indeed like a different TLS library behaving differently.

@Sean-Der Is the TLS library in OBS builds for Linux different from the one for Windows and OSX? I guess Windows and OSX rely on Mbed TLS so it would be an issue specific to MbedTLS.

[Sean-Der]

@paullouisageneau exactly!

Sorry I missed this @ianzag is this still a problem? Could I test against a server with the issue?

[ianzag]

@Sean-Der pardon for long response had very busy days. I'll check if it's still applicable tomorrow. On server side there's an ordinary OpenSSL's DTLS state machine so I believe it can be wrapped into dedicated regression test. How FooSSL performs with BarSSL.

[Sean-Der]

@ianzag Is this still a problem? Maybe OBS upgrade fixed?

[ianzag]

Looks like we've found what's the problem. In short: if server's DTLS self-signed certificate does not have Common Name it is rejected by client when it runs mbedTLS. So when OBS broadcasting fails with "Bad Certificate" DTLS failure - first check, that server returns correct certificate with Common Name included. I'm not sure if CN's value makes some difference though. But it must be returned by server somehow. OpenSSL does not have such constraints and runs DTLS with certificates without CN field just fine.

I believe this issue can be closed. Just add a note into docs that CN is required.