Abstract:
A lot of protocols use TLS to encrypt and protect the integrity of their protocol data. Thereby, most support the STARTTLS mechanism to start TLS via a plain STARTTLS protocol command. This mechanism introduces new possibilities for an attacker to attack clients and servers. In this thesis, we analyze server implementations of POP3, IMAP, and FTP, all of them supporting STARTTLS, with an approach called protocol state fuzzing. With this appraoch, we extract the state machines out of those implementations to find vulnerabilities in its states and transitions. Our results show that none of the analyzed implementations behaves as defined by the protocol, and some even have serious vulnerabilities. In particular, we found mild, not by an attacker exploitable, vulnerabilities or misbehaviors in the POP3 and IMAP implementation of Dovecot and the FTP implementation of vsftpd. Furthermore, we discovered a serious vulnerability in the POP3 and IMAP implementation of Citadel and the FTP implementation of ProFTPd. The last serious vulnerability was thereby not found by previous research.