How OTR Messaging works in Detail

In this article, we will get technical and explain how OTR Messaging works. We do our best to give a correct description of the OTR protocol. The post contains a simple and a difficult version, choose which one you like. Please keep in mind that even the difficult version is a strong simplification, because of the advanced mathematics that a correct and complete description would require.

We all use messaging services, like SMS, WhatsApp, iMessage, Google Hangouts and Facebook Messenger at work or for private conversations. Unfortunately, popular services are highly compromised when it comes to privacy. There are much better options. In this post, we will explain the technical details of OTR Messaging in Pidgin.

How OTR Messaging works in Pidgin (non-technical)

The inventors of OTR really thought the concept through. It is almost completely fail-safe. The process is the same in other messengers that support OTR, but we will focus on Pidgin here. The process is called the OTR ratchet. Your Pidgin client does all the steps automatically, so do not worry about too many details. Programmers around the world have checked the open-source code and verified that it works like this, even if you cannot see it. We can therefore trust Pidgin and OTR.

On other websites and in textbooks, you will find similar descriptions, but they use Alice and Bob and do not call the chat participants sender and recipient directly. Alice and Bob are the placeholder names used to describe communication security, just like we use the names Mallory (for malicious attacker) or Trudy (for intruder).

1. Initialisation: The participants perform an ephemeral key exchange to start the session. That way, they make sure that they are talking to the right person. This step prevents an attacker from infiltrating your chat session and making you believe that the attacker is your contact.
2. Sending a message: Pidgin creates a package that contains the actual message, but also a Diffie-Hellman key. Your Pidgin client encrypts your message and you send it to the receiver.
3. Receiving a message: Your recipient gets the package and the message with it. He/she answers and his/her Pidgin client creates a package too. That contains an encrypted answer, the key of the recipient and also a statement that acknowledges the key, which the receiver got from you (the sender). Now you (the sender) get the answer. Your Pidgin client also knows that the receiver has acknowledged your key. Then, you can use the key your contact sent you for the next message and so on.

How OTR Messaging works in Pidgin (technical)

The inventors of OTR really thought the concept through. It is almost completely fail-safe. We will now cover the technical details. You do not have to understand them at all to use Pidgin, because you do not actually see the process happening. The procedure is the same in other messengers that support OTR, but we will focus on Pidgin here. Feel free to skip this technical explanation if you want to. Only read it if you are fascinated by the technology behind cryptography, like we are.

In the following, the first step is only done once at the start of the chat session. The steps 2 and 3 are repeated for every message. It all happens instantly and you do not need to worry about the details that run in the background. We tried to formulate it in simple terms, but in reality it is even more complicated than what we will describe here:

Initialisation

1. OTR makes sure that the person you are communicating with is actually the person you want to contact. This is done by a process called Authenticated Key Exchange (AKE). First, you choose a key pair. That contains your private and public key. The public key is then exchanged with you recipient. The Pidgin software calculates a common secret s. That is a key value calculated using the Diffie-Hellman key exchange protocol.
2. If all goes as planned, the secret is used to create a secure channel. Now, you two can use the channel to authenticate with your digital signature (from your key). It makes sure that your contact is really the person you think he/she is. This “handshake” is done with the Digital Signature Algorithm (DSA).
3. You have successfully established the connection. It is AES encrypted and Pidgin will continue to check for every message if the channel is still secure. This is done with SHA-256-HMAC, a Keyed-Hash Message Authentication Code. A hash function of the SHA-family (Secure Hash Algorithm) obfuscates an original input with a one-way algorithm. Unlike encryption/decryption, hashing cannot be reversed.

Sending a message

1. Pidgin continues to keep your Diffie-Hellman keys up-to-date. For every message, Pidgin verifies the most recently sent keys of the recipient, so that you (sender) can use them. The common secret s is updated and can be recalculated with the most recent key you get from the recipient.
2. To actually be able to send a message, the software will calculate an AES-key called KAES from the first 128 bits of the SHA-hash-value of the common secret. From the AES-keys, your client program calculates an MAC-key called KMAC as a new hash value of the 160-bit hash of the AES-key. Next, Pidgin creates a numerator called c and enters Counter Mode.
3. After all these safety precautions, you write a message and Pidgin encrypts it with AES in Counter Mode, using KAES and c. The result is the encrypted message N. Pidgin builds a package T that contains N and other info, like the version number of the protocol. From that package data, Pidgin calculates a Message Authentication Code called MACKMAC(T). The code MACKMAC(T) and the package T itself are then sent to the receiver.

Receiving a message

1. The recipient will calculate the updated version independently from the sender. The secrets both of you calculate have to match. He/She then calculates KAES and KMAC too. Following this step, he/she uses KMAC to calculate MACKMAC(T) as well. This value is then compared to the one received from the sender.
2. The final step is decrypting the message N, using the KAES and c values that he/she got in the package T from the sender.

Did you make it to the end? 😛

Further Reading

We hope that we could help you understand how OTR Messaging works, but there is much more to learn. We recommend the Wikipedia articles on Symmetric-key algorithms and the Diffie–Hellman key exchange. If you speak German, you should read the German Wikipedia article on Off-the-Record Messaging, the English version is honestly not very helpful.