Monday, 16 May 2022

Troubleshooting TURN

 

WebRTC applications use the ICE negotiation to discovery the best way to communicate with a remote party. It dynamically finds a pair of candidates (IP address, port and transport, also known as “transport address”) suitable for exchanging media and data.


The most important aspect of this is “dynamically”: a local and a remote transport address are found based on the network conditions at the time of establishing a session. For example, a WebRTC client that normally uses a server reflexive transport address to communicate with an SFU. when running inside the home office, may use a relay transport address over TCP when running inside an office network which limits remote UDP targets. The same configuration (defined as “iceServers” when creating an RTCPeerConnection will work in both cases, producing different outcomes.


This means that a certain portion of WebRTC sessions happen over TURN, i.e. they are relayed through a TURN service, when the choice is left to the client. ‘host’, ‘server reflexive’ and ‘relay’ candidates are left to compete with each other, and the best will win, with the caveat that ‘host’ candidates have the highest priority, and ‘relay’ the lowest. This prioritization originates from the logical assumption that a relayed connection may be less performant than a direct one.


There are cases though when using a TURN service is not optional, but mandatory; an RTCPeerConfiguration setting, ‘iceTransportPolicy’ allows this.


In any case, when TURN is used, it’s important to be able to troubleshoot the session establishment, and this article aims to provide some important guidelines.


These are the key points:

  • Acquiring the TURN settings

  • Confirming the reachability of the TURN server

  • Creating a relay allocation on the TURN server

  • Setting permissions for using the created allocations

  • Exchanging ICE connectivity checks over TURN

  • Exchanging media and/or data over TURN


Acquiring the TURN settings

While STUN servers are typically used without the need for authentication, it’s unlikely that a TURN service can. The resources involved in a TURN service are expensive, in particular in the case of highly scalable and distributed systems, and for this reason are only allowed for authenticated customers.


The required TURN settings are:

  • A URL (in the form ‘turn:<FQDN or IP address>:port)

  • An username

  • A password (called ‘credential’)


These are provided inside the ‘iceServers’ configuration structure passed to the RTCPeerConnection at the moment of creation.

Troubleshooting points

It’s important to verify that the TURN settings are correctly configured; in Chrome, open Developer Tools and check in the JavaScript code that the ‘iceServers’ structure contains valid values.


Check also the ‘iceTransportPolicy’ (which default value is ‘all’).

Confirming the reachability of the TURN server

When the ICE candidates gathering phase begins, the ICE client verifies that the TURN URL defines a reachable service by sending a STUN Binding Request towards the IP and port resolved from the ‘iceServer’ settings.


This request originates from the IP address and port that will be used to access the TURN service, and so it will check that it’s suitable for it.


If the STUN Binding Request is received by the TURN server, then it will respond with a STUN Binding Success, carrying an attribute (XOR-MAPPED-ADDRESS) that tells what source IP and port was seen by the server.


If the STUN Binding Success response is received by the client, then there’s proof that the TURN server is reachable.


For example:



Now it’s possible to negotiate a relay allocation.

Troubleshooting points

In the host running the WebRTC client, take a network trace and verify that the STUN Binding Request is addressed to the expected destination (in particular if the TURN URL required a DNS resolution and so there are multiple IP addresses that could be used).

Verify in the trace that the STUN Binding Success Response is received.

Creating a relay allocation on the TURN server

This is the key element: the client asks the TURN server to become a relay on its behalf.


Here the TURN protocol is used, and the client issues an Allocate Request towards the TURN server. This request must be authenticated, for the reasons discussed earlier, and so it’s challenged with a 401 Unauthenticated response, carrying a realm and a nonce.


The client will use the provided credentials (username and credential), together with the given realm and nonce, to compute a MESSAGE-INTEGRITY attribute and send again the Allocate Request with this attribute.


If the credentials are correct (and also the user is allowed to access the service), then the TURN service will reserve a transport address for that allocation: this is the relay transport address. An Allocate Success Response is transmitted to the client, with a XOR-RELAYED-ADDRESS attribute.


At this point the client has gained a ‘relay’ candidate and transmits it to the remote party through the signalling system in use (this is service-specific and not standardized).


Here’s an example of a successful allocation:



Note that a client may create more than one allocation for the same session; each one will be identified by a different source port, so it will be easily identifiable. You can filter them out with something like ‘stun and udp.port==PORT`, where PORT is the client source port for a transaction you’re interested in.

Troubleshooting points

In the host running the WebRTC client, take a network trace and confirm that there’s an Allocate Success Response.


Wrong credentials

In case of wrong credentials, instead of an Allocate Success Response you’ll see another 401 Unauthenticated response. In this case you must check that the credentials are correct, and the user is authorized to access the service.



Other errors for Allocate Request

Any other error for the Allocate Request will have a detailed error code (in a similar fashion as HTTP or SIP have), so take a note on that and search for its root cause.

Setting permissions for using the created allocations


For security reasons, before media or data is exchanged through the relay, the client must set specific permissions for the remote party.


Once the client has a valid relay allocation, every time it receives an ICE candidate from the remote it must set a permission for the remote IP address.


This is accomplished with a TURN CreatePermission Request. The allocation the permission refers to is implicit from the client source IP address and port. The TURN server will respond with a CreatePermission Success if the request is accepted; note that often a client receives ICE candidates with private or reserved IP addresses: in that case the TURN server will most probably reject the request with a 403 Forbidden response.


Example:



Troubleshooting points

In the host running the WebRTC client, take a network trace and confirm that there’s a CreatePermission Success for at least one of the remote candidates.


If no CreatePermission requests are sent, or none of them is successfully accepted, then no relaying will be possible.

Exchanging ICE connectivity checks over TURN

Once the TURN server is reached, a relay allocation reserved and a permission created, there are the conditions for exchanging ICE connectivity checks over TURN.


These are performed by sending STUN Binding Requests with short term credentials; the peculiarity with TURN is that these Binding Requests are encapsulated inside a TURN Send Indication, addressed to the remote peer.


Wireshark will nicely solve this encapsulation for you, and instead of showing a Send Indication will show you its content, the Binding Request.


The TURN server will relay the Binding Request to the remote peer, performing the relay for the first time. The expected outcome is that the remote entity will respond with a Binding Success, which the TURN server will encapsulate inside a Data Indication and deliver to the client.


If that happens, then the client has learned that the remote candidate is indeed reachable via TURN and that’s a suitable candidate pair for exchanging media and data.

Troubleshooting points

In the host running the WebRTC client, take a network trace and confirm that there are Binding Requests carried over TURN that receive a Binding Success.


If Binding Success responses are not received, then something is preventing it and the best way to investigate is to take network traces on the TURN server host, if possible. Those traces will tell you whether the Binding Requests are correctly leaving the TURN server towards the remote party and whether the Binding Success responses are being received or not.


It’s possible that the remote endpoint is simply unreachable from the TURN service, and in this case the ICE candidates pair will be marked as unusable.

Exchanging media and/or data over TURN


The last fundamental step is the actual exchange of packets through the relay. The typical type of packets is RTP.


Once the connectivity checks will be successful, if the client has elected the relay candidate as the one to be used, then RTP can start flowing. You’ll be able to see the RTP packets flowing in both directions, typically with video and audio multiplexed.


There are two ways for transmitting data:

  • Indications

  • Channels


A Send Indication carries the data (RTP) and destination from the client to the TURN server. The TURN server, granted the allocation exists and the permission allows it, will extract the data and send it to the destination from the allocated relay transport address.


When the data arrives from the remote peer to the relay transport address, then the TURN server, after performing the above checks, will encapsulate the data inside a Data Indication and send it to the client.


There is a more efficient way though to exchange data: the client can define a Channel (through the ChannelBind request), which associates a channel ID to a remote party. From that moment both the client and the TURN server can exchange data via ChannelData messages carrying just the channel ID and data, omitting the remote transport address. This reduces the network and computing overhead and it is typically chosen against the use of Indications.

Troubleshooting points

In the host running the WebRTC client, take a network trace and confirm that data is being sent from the client with Send Indications or ChannelData messages, and to the client with Data Indications and ChannelData messages.


In case of monodirectional media, it’s advisable to take network traces on the TURN server host to clarify whether the media is being exchanged or not on the relay side with the remote peer.

Encrypted TURN

It’s possible to use TURN over TLS, with all the data exchanged encrypted. In this case using Wireshark as described won’t allow you to see the details of the requests and responses, and troubleshooting is harder.


One possible approach is to first of all ensure that all the operations described previously happen correctly when using unencrypted TURN (over UDP or TCP). It’s very likely that the TURN service you are using is accessible over unencrypted UDP (default behavior): before moving to TLS ensure UDP works fine.


Wireshark will show you anyway the TLS connections established with the server, so that will confirm whether the connection was successful, the TLS session established, and some application data exchanged.

Useful tools

Wireshark

Wireshark is available for a variety of platforms; it’s a fundamental tool to understand what’s happening between the local WebRTC client and the remote server.


It comes with filters that detect the type of packets. You can use `stun` to filter out STUN and TURN packets, and even select specific TURN transactions, like `stun.type.method==0x0003` to show Allocate Request and Responses.


Saving a trace into a pcap file and making it available to others helps enormously the ability to troubleshoot.


Wireshark can be used for both capturing and just displaying captures.


There are cases where the dissectors, i.e. the interpreters of the packets, don’t recognize a TURN transaction. For example this happens when they happen over a non-default port (3478 for UDP and TCP, 5349 for TLS). To “help” Wireshark, right click on a packet, select “Decode As…” and set ‘STUN’ as protocol: it will correctly interpret all the packets using that non-default port.


The same applies for RTP: when signalling is not available to Wireshark, then UDP packets containing RTP may not be correctly interpreted. Use the same “Decode As…” method.

tcpdump

On the server side, any tool for packet capture would do, with tcpdump being a common solution.


Save the trace into a pcap file with the `-w` option, e.g. `tcpdump -n -v -w trace_1.pcap`, copy it to your machine and use Wireshark to display the packets.


WebRTC samples, Trickle ICE


This open source tool allows you to verify the browser can correctly gather `relay` candidates with the given TURN server details (URL, username, credential).


Before troubleshooting a client implementation, ensure that this tool can correctly access the TURN resources you’re referring to.


turnutils_uclient

The popular open source implementation of a TURN server, coturn, comes with a tool that simulates a client. A plethora of options are available, allowing you to test specific aspects of the TURN operations, e.g. using Send Indications or using Channels, etc.


Use `turnutils_uclient` to ensure the TURN service you want to use is accessible correctly with the given TURN settings, You’ll also get information about the round trip time and jitter.


Chrome webrtc-internals

When using Chrome, the best way to understand what’s happening is to open a tab on chrome://webrtc-internals/. It will show you all the information related to each RTCPeerConnection being managed by the browser at that point, including the list of ICE candidates, the details of the TURN server being used (except the credential for obvious reasons), including the iceTransportPolicy (‘all’ or ‘relay’), the chosen ICE candidates pair, statistics on media transfer, etc.


Search for `relay` candidates and verify the client is able to retrieve them from the TURN service, and whether they are selected as the candidate pair or not.


Conclusions

This article should provide a good checklist for troubleshooting the connection to a TURN service. There is much more to say, in particular for what concerns browsers different than Chrome and server-side investigations: I plan to write about it in the future.



Thursday, 11 March 2021

[off topic] Differences between running and cycling

 I'm a passionate runner, and always considered cycling as something fun, e.g. mountain-biking, but difficult to practice regularly. There's a lot of overhead in cycling, like the preparation, bike maintenance, dealing with city traffic, etc.

Anyway about eight months ago I bought a road bike and felt in love with it. Soon after that I discovered Zwift and that gave an additional dimension to the sport: practice whenever you want from home, with accurate power measurements and a way to socialise with distant people. That was a game changer.

In five months I cycled 1600 virtual Km and climbed almost 17 virtual Km. Meanwhile my running performance, instead of degrading, improved, and that surprised me.

Anyway what I wanted to write about is a great article I read, "Physiological Differences Between Cycling and Running". It's a review of articles published in that area. Some conclusions are very interesting.

In general it seems sports medicine is still inconclusive for many aspects, and coaches may still have an advantage by following empirical/heuristic approaches in comparison with research-driven indications.

But more specifically, some notes from the conclusions:

- For the same person, VO2max depends on the speciality (i.e. runners achieve higher values on treadmill than cycle ergometer)

- There seems to be more physiological transfer from running to cycling than the other way around

- Pedalling cadence impacts the metabolic response during cycling, but also during a following run (at least in the short term)

- The Lactate Threshold is lower for athletes when not practicing their speciality, i.e. the Lactate Threshold depends on the training method

- Both female and male are impacted in the same way when comparing VO2max for running and cycling

- Triathletes have similar max Heart Rate when running and cycling, again pointing to the importance of the actual speciality used in training

- The position when cycling makes it harder to breathe

and probably other important elements that I wasn't able to fully grasp.


Wednesday, 10 March 2021

Notes on STUN protocol

 Since I needed to see a few details in the handling of attributes in STUN responses, I thought of going through the whole STUN protocol RFC again and take notes on the most important parts.

I put my notes in some slides, and I'm sharing then in Slideshare in case somebody else may find them useful too:


Friday, 19 February 2021

Extracting RTP streams from network captures

I needed an efficient way to programmatically extract RTP streams from a network capture.

In addition I wanted to:

  • save each stream into a separate pcap file.
  • extract SRTP-negotiated keys if present and available in the trace, associating them to the related RTP (or SRTP if the negotiation succeeded) stream.


Some caveats:

  • In normal conditions the negotiation of SRTP sessions happens via a secure transport, typically SIP over TLS, so the exchanged crypto information may not be available from a simple network capture.
  • There are ways to extract RTP streams using Wireshark or tcpdump; it’s not necessary to do it programmatically.


All this said I wrote a small tool (https://github.com/giavac/pcap_tool) that parses a network capture and tries to interpret each packet as either RTP/SRTP or SIP, and does two main things:

  • save each detected RTP/SRTP stream into a dedicated pcap file, which name contains the related SSRC.
  • print a summary of the crypto information exchanged, if available.


With those two elements, it’s then possible to decrypt an SRTP stream, depending on the availability of the exchanged crypto information, and also decode it into audio, depending on the codec.


Decryption and decoding is not part of my tool, but can be achieved easily with other tools, like pjsip’s pcaputil.

I might integrate that part into pcap_tool in the future. Again not because it’s strictly necessary, but to start getting more control on the parsing and manipulation. This may reveal to be useful in the future.


pcap_tool is available here for anybody interested in using it and may perhaps wish to change or extend some parts.


You can just clone it and build it as described in the README.


An example output:


./pcap_tool -d ../../trace_20210218_1.pcap


[…]


Extracted 1092 RTP frames

Detected RTP Stream: 0x7a2179fa Source port:22248 - Destination port:4000 - Packets: 544 (./stream-0x7a2179fa.pcap)

Detected RTP Stream: 0x772dc5d7 Source port:4000 - Destination port:22248 - Packets: 548 (./stream-0x772dc5d7.pcap)



source port: 22248 - tag: 3 - suite: AES_CM_128_HMAC_SHA1_80 - key: /1TI6DJWHk7fBJY1yBp7L51uEz1JJ2n6CcQAAsJM

-----

source port: 4000 - tag: 4 - suite: AES_CM_128_HMAC_SHA1_32 - key: mPytX24bRmyNgMaqQSxP8dMMqdkkmQeHgC2Ttb3v

source port: 4000 - tag: 3 - suite: AES_CM_128_HMAC_SHA1_80 - key: J1YS1owJDKAFdq5cRF+JtektYDf6IiowCAeijeal

source port: 4000 - tag: 2 - suite: AES_256_CM_HMAC_SHA1_32 - key: 5A9R8O8MCzbuGvJ08WWNJcNHsPaEcEp1ZDp5DunknZ+bZ2JQaVpZ2qmqraTmgQ==

source port: 4000 - tag: 1 - suite: AES_256_CM_HMAC_SHA1_80 - key: ZcZn1IY++2xsSIk/U1GsHSGp+OI/BYIocv/40ldJB28bcNeMmYzs4z4ozrNQ5Q==

-----


That network capture contained 2 SRTP streams, which have been saved separately into stream-0x7a2179fa.pcap and stream-0x772dc5d7.pcap files respectively.


For the negotiation it’s visible what the sender from port 22248 (owner of the 0x7a2179fa stream) used as crypto information, and looking at the same tag (3 in this case) it’s possible to see what crypto information was used by the sender of 0x772dc5d7 stream from port 4000.


With this it’s possible to decrypt (and decode since G.711 was used) with pjsip’s pcaputil with something like:


pcaputil -c AES_CM_128_HMAC_SHA1_80 -k /1TI6DJWHk7fBJY1yBp7L51uEz1JJ2n6CcQAAsJM stream-0x7a2179fa.pcap stream-0x7a2179fa.wav


and have the audio from that stream into a WAV file.


How to build pcaputil (in fact all pjsip’s applications) is widely documented but I also described it in the appendix of https://www.giacomovacca.com/2020/11/testing-sip-platforms-and-pjsip.html 


The call in the example was generated in fact with pjsua.





Friday, 4 December 2020

SIP - Connection reuse vs Persistent connection

It goes without saying that SIP solutions are impacted by NAT. So much that some scenarios required integration to RFC 3261, e.g. with RFC 3581, which defined the 'rport' attribute to be added in the Via header (integrating the 'received' attribute): with that information, responses could be routed to the source port of the related request, and not on the advertised port in the original Via.

That was called Symmetric Response, and applied to connection-less transports (UDP), while, as mentioned in RFC 6314, it's not necessary when using reliable transports (TCP in most cases): SIP responses can be sent back on the same connection on which the request arrived.

Also from RFC 3261, chapter 18, Transport layer, client behaviour:

"For reliable transports, the response is normally sent on the connection on which the request was received."

But the client needs to be prepared to receive the response on a new connection:

"[...] the transport layer MUST also be prepared to receive an incoming connection on the source IP address from which the request was sent and port number in the "sent-by" field."

That obviously would require the ability to create such connection from the server to the client.

Anyway when a reliable connection between two SIP entities is up, after a transaction is already concluded, there are two interesting opportunities:

- Use that same connection for more requests from the client
- Use that same connection for more requests from the server

where for "client" I refer to the entity that created the connection and sent the initial request, and "server" is the entity that accepted the connection and delivered the response(s).

The first case is mentioned in the same chapter on Transport layer:

"If a request is destined to an IP address, port, and transport to which an existing connection is open, it is RECOMMENDED that this connection be used to send the request, but another connection MAY be opened and used."

This is what's referred to as "Persistent connection", as mentioned in RFC 5923:

"The SIP protocol includes the notion of a persistent connection
   [...], which is a mechanisms to insure that
   responses to a request reuse the existing connection that is
   typically still available, as well as reusing the existing
   connections for other requests sent by the originator of the
   connection."

The second case (using the same connection from future requests from the server) is instead the subject of RFC 5923, and it is defined as "Connection reuse".

Once the connection is up, it seems a good opportunistic approach to reuse it, but an important limitation is mandated:

"Unlike TCP, TLS connections can be reused to send requests in the backwards direction since each end can be authenticated when the connection is initially set up."

In other words, only TLS connections formed by exchanging certificates can be reused, because the identities have been mutually verified.

The way the client can tell the server that connection reuse is desired is with a new parameter to be added in the Via header: 'alias'.

In general, RFC 5923 at chapter 5 clarifies:

"The act of reusing a connection needs
   the desired property that requests get delivered in the backwards
   direction only if they would have been delivered to the same
   destination had connection reuse not been employed."


Last important bit, not to be left implicit: persistent connections don't imply connection reuse, as RFC 5923 clarifies:

"[...] Persistent connections do not
      imply connection reuse."

So this post is basically sharing my own notes on this topic, which maybe somebody else (including me) can find useful in the future.



Tuesday, 24 November 2020

Dissecting traces with DTMF tones

I'm sure I belong to the large group of people who love to analyse network traces with tools like Wireshark. Being able to see the details of a packet or datagram down to the level of the bits is not only extremely useful, but also fascinating.


Time ago I wrote a dissector for Wireshark, using the Lua interface, and that was fun (I see it's still available here). The official recommendation is to use Lua only for prototyping and testing, but when performances are not key and there isn't the intent to add the dissector to the official distribution, it's fast and effective.

In order to parse network traces with audio and extract it into the payload first, and then decode it into a WAV file, C is a viable solution. I wrote about a program that does that here and since it attracted some attention and feedback I wrote an updated version later.

More recently I wanted to identify programmatically the presence (and value) of DTMF tones - as RTP Events, RFC 2833 - in network traces. This time rather than using C, I wanted to integrate it with python, and scapy seemed a good choice.

scapy is quite complete, but interestingly it doesn't have a parser for the RTP Event extension. So I thought of mapping the raw content in the RTP payload to a structure, with the help of C types.

This it the core of the program:

def process_pcap(file_name, sut_ip):

  for (pkt_data, pkt_metadata,) in RawPcapReader(file_name):

    ether_pkt = Ether(pkt_data)

    # A little housekeeping to filter IPv4 UDP packets goes here
    ...

    if ether_pkt.haslayer(UDP):
      udp_pkt = ether_pkt[UDP]

    # Get the raw UDP packet into an RTP structure
    rtp_pkt = RTP(udp_pkt["Raw"].load)

    ptype = rtp_pkt.payload_type

    # Assume payload type 96 or 101 are used for RTP events
    if (ptype == 96 or ptype == 101):
      rtpevent_content = rtp_pkt.payload

    # map the payload into an RTPEvent object
    rtpevent_struct = RTPEvent.from_buffer_copy(rtpevent_content.load)

The RTPEvent class looks like this:

class RTPEvent(ctypes.BigEndianStructure):
  _fields_ = [
    ('event_id', ctypes.c_uint8),
    ('end_of_event', ctypes.c_uint8, 1),
    ('reserved', ctypes.c_uint8, 1),
    ('volume', ctypes.c_uint8, 6),
    ('duration', ctypes.c_uint16)
  ]

so once mapped, the rtpevent_struct object will have its DTMF-specific details, in particular with the digit contained in rtpevent_struct.event_id, and the indication whether it's the marker of end of the event in the end_of_event bit.

All the other information (source/destination IP address and port, timestamp, SSRC) is obviously available in the UDP and RTP portion, so it's easy to adapt to your needs and filter out the DTMF tones for the streams you're interesting in.


Troubleshooting TURN

  WebRTC applications use the ICE negotiation to discovery the best way to communicate with a remote party. I t dynamically finds a pair of...