Abstract:
Datagram Transport Layer Security (DTLS) is a cryptographic protocol that provides secure communication over unreliable connections such as UDP. DTLS is an adaption of Transport Layer Security (TLS), designed to run on top of reliable connections such as TCP. UDP is popular amongst streaming and Internet of Things technologies, whose popularity has been increasing. DTLS and TLS security has been challenged in recent decades, leading to the discovery of many attacks[35]. We also learned from the discovery of Heartbleed[14] that implementation flaws can cause major security issues. Reducing the risk of implementation flaws is achieved by testing. Protocol state fuzzing has been successful in testing TLS [34, 37] and DTLS[15, 13]. This testing technique uses model learning to approximate the state machine protocol implementations implicitly follow. These state machines are analyzed for deviations from protocol specifications.
In this thesis, we apply protocol state fuzzing to DTLS implementations and analyze the resulting state machines for security flaws or non-conformance issues. We conduct our research using TLS-StateVulnFinder, a protocol state fuzzing framework from the tls-attacker software family. We extend TLS-StateVulnFinder with DTLS support using TLS-Attacker’s implementation of DTLS and integrate TLS-Docker-Library into TLS-StateVulnFinder, simplifying server access. We obtain state machines for four DTLS implementations, OpenSSL, MbedTLS, LibreSSL, and WolfSSL. Our work reveals that applying protocol state fuzzing comes with several challenges. One of the main challenges is that state machines of DTLS implementations get quite large, making an automated analysis necessary. Our analysis reveals several non-conformance issues, but we do not identify any major security flaws. We find that MbedTLS shows flaws in validating the sequence numbers, while OpenSSL, WolfSSL and LibreSSL, we find non-conforming responses to certain messages.