tag:blogger.com,1999:blog-90319227732240561332024-03-19T05:51:13.433+01:00Giacomo VaccaGiacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.comBlogger119125tag:blogger.com,1999:blog-9031922773224056133.post-66765213698804592312022-10-27T13:32:00.004+02:002022-10-27T13:32:48.087+02:00About ICE negotiation<p><i>Disclaimer: I wrote this article on March 2022 while working with Subspace, and the original link is here: <a href="https://subspace.com/resources/ice-negotiation">https://subspace.com/resources/ice-negotiation</a> . This post in my personal blog is a way to ensure it doesn't get lost. There is nothing service-specific in it, I've made only minor edits and I hope it can be a good technical reference on the topic.</i></p><p><br /></p><p><br /></p><p><a href="https://webrtc.org/" rel="nofollow" target="_blank">WebRTC</a> is a set of protocols that allow applications, typically running on Web browsers, to exchange media (audio, video, data) with other entities.</p><div>Before media can flow, however, the WebRTC entities need to discover what type of connection is possible, and among the possible connections, what’s the best to be used. This needs to happen as fast as possible, so that users can perceive the service as instantaneous as possible.</div><div><br /></div><div>WebRTC includes protocols like <a href="https://www.rfc-editor.org/rfc/rfc8489.html" rel="nofollow" target="_blank">STUN</a> and <a href="https://www.rfc-editor.org/rfc/rfc8656.html" rel="nofollow" target="_blank">TURN</a> that are designed to facilitate the establishment of connections when a direct connection is not possible. The typical case is a computer inside a home or office network, with a private IP address, and able to reach the public Internet only through an address translation (NAT).</div><div><br /></div><div>STUN helps in discovering the IP address and port from where a computer enters the Internet, and in some circumstances that IP address and port can be used by other entities to reach that computer. STUN is also used for keeping such bindings alive.</div><div><br /></div><div>TURN provides a way for two entities to communicate when they are behind two different symmetric NATs, or when one is behind a firewall that restricts outbound traffic to only some UDP or TCP ports. TURN uses STUN as the underlying protocol, adding requests, responses and indications to accomplish media relay.</div><div><br /></div><div>STUN and TURN play a role in the ICE negotiation process. <a href="https://www.rfc-editor.org/rfc/rfc8445.html" rel="nofollow" target="_blank">ICE</a>, Interactive Connectivity Establishment, is a protocol that allows the dynamic discovery of the best way to establish a connection for entities that may be behind NAT.</div><div><br /></div><div>All WebRTC clients use ICE before media can flow.</div><div><br /></div><div>There are three main phases: the gathering of candidates, the connectivity checks, and the nomination of the candidate pairs to be used.</div><div><br /></div><div>The ICE candidates are simply transport addresses (IP address, port and transport type) that can potentially be used to communicate (send and receive media) and that the ICE client collects and shares to the other party through some form of signalling.</div><div><br /></div><div>There are three main types of ICE candidates: 'host', 'server reflexive' and 'relay'.</div><div><br /></div><div>'host' candidates refer to transport addresses that are directly visible by the client, where the client can start listening for incoming connections or packets. Computers behind NAT may only have private IP addresses as 'host' candidates, but they are potentially usable if the other party belongs to the same network, or depending on the type of NAT.</div><div><br /></div><div>'server reflexive' candidates are the ones discovered through the interaction with a STUN server. The client sends a Binding Request to the STUN server and receives a Binding Success Response with a MAPPED-ADDRESS containing the source IP address and port from where the request was received. There may be more than one level of NAT, and that transport address represents the outmost one.</div><div><br /></div><div>'relay' candidates refer to allocations reserved on a TURN server. The ICE client requests an Allocation of a relay, and after successful authentication the TURN server provides a RELAYED-ADDRESS containing the transport address allocated on that server for the client.</div><div><br /></div><div>These types of ICE candidates have different priorities, 'host' being at highest priority and 'relay' at lowest priority; this is a way to privilege direct interconnection when possible (but that not necessarily represents the best solution in terms of connection quality and in general of Quality Of Experience).</div><div><br /></div><div>This diagrams shows a typical process where ICE candidates are gathered and sent to the other party:</div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEggwl-Sj6Y1GTc5HrjnW1SGdPynvBM8Vu5r6pLZY9fUIuiaD3tHbzyrAA-RZx0hKgyzFhxa11FLPRF9Mn8FLpXB4yXH3uO1100qou2o4hXg434BSjTchT10sXpQIDuNpuQVdxwGkg1mfFxTJtfDPnge3PxbPdpUq9kMjiY1d1XuF2DrA9IWBvQub18U" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="393" data-original-width="694" height="226" src="https://blogger.googleusercontent.com/img/a/AVvXsEggwl-Sj6Y1GTc5HrjnW1SGdPynvBM8Vu5r6pLZY9fUIuiaD3tHbzyrAA-RZx0hKgyzFhxa11FLPRF9Mn8FLpXB4yXH3uO1100qou2o4hXg434BSjTchT10sXpQIDuNpuQVdxwGkg1mfFxTJtfDPnge3PxbPdpUq9kMjiY1d1XuF2DrA9IWBvQub18U=w400-h226" width="400" /></a></div></div><div><br /></div><div>In this example the candidates are communicated as soon as they are retrieved. This technique is called <a href="https://www.rfc-editor.org/rfc/rfc8838.html" rel="nofollow" target="_blank">Trickle ICE</a> and was designed to ensure that the connectivity checks can happen as soon as the candidates are available.</div><div><br /></div><div>Without Trickle ICE, the WebRTC client would need to wait for all the candidates to be collected before sending an offer or an answer, increasing the session set up time.</div><div><br /></div><div>The candidates are transmitted over a signalling system established between the two parties. This is outside of the WebRTC specifications and application-specific.</div><div><br /></div><div>The WebRTC client will then receive the ICE candidates from the other party, and it will build a list of “candidate pairs”: each local candidate will be paired with each remote candidate.</div><div><br /></div><div>After this operation the connectivity checks can begin.</div><div><br /></div><div>Let’s see it with an example, assuming UDP as transport for all cases. The WebRTC client has a local host candidate, IP1:port1, and has received a remote host candidate, IP2:port2. It builds a “candidate pair” with the two: {IP1:port1, IP2:port2}.</div><div><br /></div><div>The WebRTC client will start sending STUN Binding Requests with source IP1:port1 and destination IP2:port2. These requests use STUN short-term authentication, and contain a username and password that were previously exchanged when transmitting the candidates inside the SDP offer/answer.</div><div><br /></div><div>If the Binding Request reaches the other party on IP2:port2, then the other party will authenticate the request, and respond with a Binding Success Response, including the MAPPED-ADDRESS attribute, containing the source of the request.</div><div><br /></div><div>If the Binding Success Response reaches the WebRTC client, then it will identify the request by looking into the transaction ID and mark the check as Successful, and so suitable for exchanging media.</div><div><br /></div><div>If for any reason the Binding Success Response is not received, then the candidate pair will remain in a In Progress state for some time, and then move to Failed: that pair cannot be used to exchange media.</div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEimhErTpOE6gUIaqvV3E-1JrrBi277LhHrt6cxFlHFYJzy0sRtS6tF15ri-Ys0rRcbiA98DJJPvwOnoQdx001wLgfKlJyzoTGmhrT8sIkoGkuMH3z9sir47-kjqmRQDIGroCfxZpF6TNr_rcAPtNs3qThL1iSCkh0-_fMWqh_SGjrWAIhzNiIWINEO5" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="333" data-original-width="694" height="154" src="https://blogger.googleusercontent.com/img/a/AVvXsEimhErTpOE6gUIaqvV3E-1JrrBi277LhHrt6cxFlHFYJzy0sRtS6tF15ri-Ys0rRcbiA98DJJPvwOnoQdx001wLgfKlJyzoTGmhrT8sIkoGkuMH3z9sir47-kjqmRQDIGroCfxZpF6TNr_rcAPtNs3qThL1iSCkh0-_fMWqh_SGjrWAIhzNiIWINEO5" width="320" /></a></div><br /><br /></div><div>It’s important to note that this is symmetrical: the other party too can start the connectivity check from IP2:port2 towards IP1:port1. To avoid a conflict, the parties assume the role of "controlling" or "controlled" agent. The controlling agent will be the one deciding which candidate pair will be used for exchanging media.</div><div><br /></div><div>Before media can flow through a TURN server, a client must create a Permission. This is important during ICE connectivity checks.</div><div><br /></div><div>For ICE candidates of type 'relay', the connectivity check will be performed sending Binding Requests that traverse the TURN server and reach the other party on the relay side. The Binding Requests will be carried by a Send Indication, destined to the remote candidate as peer address. The TURN server will only accept it and relay it to the destination if a Permission has been granted for that peer.</div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiBAgdScgI0iNtx3nNry-9cfiksVA7By2MPVo-RaYXsP-6SdjTz3lIoe4nxJwtC3YtS0NyUnnO8h6uBpT7YCSHSbGXy4tXaY6tFie_rnXVSvnB5AbqV23PHo3C3AOp3qZpmbsZUYmog1FvMoZli1urMfDO7C-RL8XXC7WNVYfYSoPeu6HMXHe886Xmi" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="359" data-original-width="694" height="208" src="https://blogger.googleusercontent.com/img/a/AVvXsEiBAgdScgI0iNtx3nNry-9cfiksVA7By2MPVo-RaYXsP-6SdjTz3lIoe4nxJwtC3YtS0NyUnnO8h6uBpT7YCSHSbGXy4tXaY6tFie_rnXVSvnB5AbqV23PHo3C3AOp3qZpmbsZUYmog1FvMoZli1urMfDO7C-RL8XXC7WNVYfYSoPeu6HMXHe886Xmi=w400-h208" width="400" /></a></div><br /><br /></div><div><i>(NAT has been omitted in this diagram for simplicity)</i></div><div><br /></div><div>Of course a TURN allocation must exist for the CreatePermission request to succeed, but that’s already been created during the candidate gathering phase.</div><div><br /></div><div>If the candidate pair selected for exchanging media will be one with a local 'relay' candidate, then typically the WebRTC client binds a TURN Channel to the other party, and starts exchanging media using ChannelData messages, instead of Send/Data Indications.</div><div><br /></div><div>There are more details that can be discussed, like managing timeouts, role conflicts, ICE Lite, etc, which we will address in other articles. One additional aspect is important here: we mentioned three types of candidates, host, server reflexive and relayed, but there’s a fourth one, "peer reflexive".</div><div><br /></div><div>Peer reflexive candidates are not provided directly by a party during candidate exchange, but are instead discovered dynamically during the connectivity checks. Getting back to the previous example with a candidate pair {IP1:port1, IP2:port2}, depending on NAT conditions, the response to the Bind Request from IP1:port1 can be received by another source, IP3:port3.</div><div><br /></div><div>The WebRTC client can verify IP3:port3 is sending a valid response by checking the transaction ID, and if successful it will dynamically add a remote candidate of type peer reflexive. Sometimes the peer reflexive candidate is the only one suitable and will be used to exchange media.</div><div><br /></div><div>Chrome’s chrome://webrtc-internals, Firefox’s about:webrtc and Safari's "WebRTC Logging" will show the list of candidates and the pair that was selected, so those tools are of great value when troubleshooting.</div><div><br /></div><div>If you take a trace on the computer running the WebRTC client, you’ll be able to see the STUN Binding Requests and Responses, and CreatePermission, Send/Data Indications for connectivity checks if unencrypted TURN is used. Wireshark will filter those messages for you if you use the ‘stun’ filter, and will also be able to interpret the Binding Request/Response carried inside Send/Data Indications (and also the RTP streams, but that’s for another article).</div><div><br /></div><div>If you're interested about troubleshooting TURN sessions, take a look at this other article, <a href="https://www.giacomovacca.com/2022/05/troubleshooting-turn.html" rel="nofollow" target="_blank">"Troubleshooting TURN"</a>.</div>Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-52604874343008569442022-05-16T17:29:00.001+02:002022-05-16T17:29:46.300+02:00Troubleshooting TURN<p> </p><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">WebRTC applications use the <a href="https://datatracker.ietf.org/doc/html/rfc8445" target="_blank">ICE negotiation</a> to discovery the best way to communicate with a remote party. I</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; white-space: pre-wrap;">t dynamically finds a pair of candidates (IP address, port and transport, also known as “transport address”) suitable for exchanging media and data.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The most important aspect of this is “dynamically”: a local and a remote transport address are found based on the network conditions at the time of establishing a session. For example, a WebRTC client that normally uses a server reflexive transport address to communicate with an SFU. when running inside the home office, may use a relay transport address over TCP when running inside an office network which limits remote UDP targets. The same configuration (defined as “iceServers” when creating an <a href="https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection" target="_blank">RTCPeerConnection</a> will work in both cases, producing different outcomes.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This means that a certain portion of WebRTC sessions happen over TURN, i.e. they are relayed through a TURN service, when the choice is left to the client. ‘host’, ‘server reflexive’ and ‘relay’ candidates are left to compete with each other, and the best will win, with the caveat that ‘host’ candidates have the highest priority, and ‘relay’ the lowest. This prioritization originates from the logical assumption that a relayed connection may be less performant than a direct one.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">There are cases though when using a TURN service is not optional, but mandatory; an RTCPeerConfiguration setting, ‘iceTransportPolicy’ allows this.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In any case, when TURN is used, it’s important to be able to troubleshoot the session establishment, and this article aims to provide some important guidelines.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">These are the key points:</span></p><ul style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space: pre-wrap;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Acquiring the TURN settings</span></p></li><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space: pre-wrap;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Confirming the reachability of the TURN server</span></p></li><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space: pre-wrap;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Creating a relay allocation on the TURN server</span></p></li><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space: pre-wrap;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Setting permissions for using the created allocations</span></p></li><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space: pre-wrap;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Exchanging ICE connectivity checks over TURN</span></p></li><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space: pre-wrap;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Exchanging media and/or data over TURN</span></p></li></ul><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><h1 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Acquiring the TURN settings</span></h1><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">While STUN servers are typically used without the need for authentication, it’s unlikely that a TURN service can. The resources involved in a TURN service are expensive, in particular in the case of highly scalable and distributed systems, and for this reason are only allowed for authenticated customers.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The required TURN settings are:</span></p><ul style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space: pre-wrap;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">A URL (in the form ‘turn:<FQDN or IP address>:port)</span></p></li><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space: pre-wrap;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">An username</span></p></li><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space: pre-wrap;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">A password (called ‘credential’)</span></p></li></ul><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">These are provided inside the ‘iceServers’ configuration structure passed to the <a href="https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection" target="_blank">RTCPeerConnection</a> at the moment of creation.</span></p><h2 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Troubleshooting points</span></h2><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">It’s important to verify that the TURN settings are correctly configured; in Chrome, open Developer Tools and check in the JavaScript code that the ‘iceServers’ structure contains valid values.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Check also the ‘iceTransportPolicy’ (which default value is ‘all’).</span></p><h1 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Confirming the reachability of the TURN server</span></h1><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">When the ICE candidates gathering phase begins, the ICE client verifies that the TURN URL defines a reachable service by sending a STUN Binding Request towards the IP and port resolved from the ‘iceServer’ settings.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This request originates from the IP address and port that will be used to access the TURN service, and so it will check that it’s suitable for it.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">If the STUN Binding Request is received by the TURN server, then it will respond with a STUN Binding Success, carrying an attribute (XOR-MAPPED-ADDRESS) that tells what source IP and port was seen by the server.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">If the STUN Binding Success response is received by the client, then there’s proof that the TURN server is reachable.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">For example:</span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikQ32P8qmYtE674lYrVTte9nV1pjjX3c2OHKqiVIJ-6tCtHf38YFONiFV3sYrIYYeIwCXuZVZJXVqAmQwMt250GEy_hjuC5KD6e2zQXQGqSIQ2BwbhlsyAfhYoK6h5Byn-BdPcGUDsMdsOkw2FjViwM2noY5QNylabvY5_r6nG_IrZt5fvUk9U19Tb/s2968/binding.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="178" data-original-width="2968" height="38" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikQ32P8qmYtE674lYrVTte9nV1pjjX3c2OHKqiVIJ-6tCtHf38YFONiFV3sYrIYYeIwCXuZVZJXVqAmQwMt250GEy_hjuC5KD6e2zQXQGqSIQ2BwbhlsyAfhYoK6h5Byn-BdPcGUDsMdsOkw2FjViwM2noY5QNylabvY5_r6nG_IrZt5fvUk9U19Tb/w640-h38/binding.png" width="640" /></a></div><br /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><br /></p><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Now it’s possible to negotiate a relay allocation.</span></p><h3 style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt; text-align: left;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Troubleshooting points</span></h3><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In the host running the WebRTC client, take a network trace and verify that the STUN Binding Request is addressed to the expected destination (in particular if the TURN URL required a DNS resolution and so there are multiple IP addresses that could be used).</span></p><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Verify in the trace that the STUN Binding Success Response is received.</span></p><h1 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Creating a relay allocation on the TURN server</span></h1><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This is the key element: the client asks the TURN server to become a relay on its behalf.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Here the TURN protocol is used, and the client issues an Allocate Request towards the TURN server. This request must be authenticated, for the reasons discussed earlier, and so it’s challenged with a 401 Unauthenticated response, carrying a realm and a nonce.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The client will use the provided credentials (username and credential), together with the given realm and nonce, to compute a MESSAGE-INTEGRITY attribute and send again the Allocate Request with this attribute.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">If the credentials are correct (and also the user is allowed to access the service), then the TURN service will reserve a transport address for that allocation: this is the relay transport address. An Allocate Success Response is transmitted to the client, with a XOR-RELAYED-ADDRESS attribute.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">At this point the client has gained a ‘relay’ candidate and transmits it to the remote party through the signalling system in use (this is service-specific and not standardized).</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Here’s an example of a successful allocation:</span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgoqN1Z7BWWitbEctb4ehc3of4PUEa4uGM0NNsEGwfIlnRLlHE1htxDS4A3NGiN3ry-FGQ2H9PrMTWRLkjOCofVWWhelMRhXSR3aWGf5OhWOmi1K3oSed-rTDQPPaiV2d6eCnmphzdzhwIA33lhZzJyrJmY0MjCcHid6pfh1p04g-uQ4sS67LTshbG/s3350/allocate_ok.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="192" data-original-width="3350" height="36" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgoqN1Z7BWWitbEctb4ehc3of4PUEa4uGM0NNsEGwfIlnRLlHE1htxDS4A3NGiN3ry-FGQ2H9PrMTWRLkjOCofVWWhelMRhXSR3aWGf5OhWOmi1K3oSed-rTDQPPaiV2d6eCnmphzdzhwIA33lhZzJyrJmY0MjCcHid6pfh1p04g-uQ4sS67LTshbG/w640-h36/allocate_ok.png" width="640" /></a></div><br /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><br /></p><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Note that a client may create more than one allocation for the same session; each one will be identified by a different source port, so it will be easily identifiable. You can filter them out with something like ‘stun and udp.port==PORT`, where PORT is the client source port for a transaction you’re interested in.</span></p><h2 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Troubleshooting points</span></h2><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In the host running the WebRTC client, take a network trace and confirm that there’s an Allocate Success Response.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><h3 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 4pt; margin-top: 16pt;"><span style="background-color: transparent; color: #434343; font-family: Arial; font-size: 14pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Wrong credentials</span></h3><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In case of wrong credentials, instead of an Allocate Success Response you’ll see another 401 Unauthenticated response. In this case you must check that the credentials are correct, and the user is authorized to access the service.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaJxvYurO9wz179vsDOKUuvDnuQjkcqMm7kkeJp7PuPzBgO1aGCpCjWqAPaaSKaao7dnHKqFhKdFp3PRlwjR185vcoUWaKr0zjU4u2wItyzmg6L_pTC5CjuP7frkW6eREDKRKNGgvpcZ6TVYT3uly5Jhc3Xg2mLmN1O-KYJQ2LxzJttMItgwioUxAD/s3140/allocation_401.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="224" data-original-width="3140" height="46" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaJxvYurO9wz179vsDOKUuvDnuQjkcqMm7kkeJp7PuPzBgO1aGCpCjWqAPaaSKaao7dnHKqFhKdFp3PRlwjR185vcoUWaKr0zjU4u2wItyzmg6L_pTC5CjuP7frkW6eREDKRKNGgvpcZ6TVYT3uly5Jhc3Xg2mLmN1O-KYJQ2LxzJttMItgwioUxAD/w640-h46/allocation_401.png" width="640" /></a></div><br /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: #434343; font-family: Arial; font-size: 14pt; white-space: pre-wrap;">Other errors for Allocate Request</span></p><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Any other error for the Allocate Request will have a detailed error code (in a similar fashion as HTTP or SIP have), so take a note on that and search for its root cause.</span></p><h1 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Setting permissions for using the created allocations</span></h1><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">For security reasons, before media or data is exchanged through the relay, the client must set specific permissions for the remote party.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Once the client has a valid relay allocation, every time it receives an ICE candidate from the remote it must set a permission for the remote IP address.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This is accomplished with a TURN CreatePermission Request. The allocation the permission refers to is implicit from the client source IP address and port. The TURN server will respond with a CreatePermission Success if the request is accepted; note that often a client receives ICE candidates with private or reserved IP addresses: in that case the TURN server will most probably reject the request with a 403 Forbidden response.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Example:</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8iv1VHLyvLMhV6_pHYI6sfwbqu3tX14yLR9HoRAR8ZUsTgJPHpWZAsRoscgrjP4JK6iLVZ18gBN7vt8Bd_veiCjlIVtplhBxNyBjFC4RyfRFl0OWNJ-gnPxxGYgdpgFyOARiK6WK2fDuEdY_VIERGrKI5pajLDZcno2uBdJiZbI8OGDGaqcGXplAe/s3480/permissions.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="174" data-original-width="3480" height="32" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8iv1VHLyvLMhV6_pHYI6sfwbqu3tX14yLR9HoRAR8ZUsTgJPHpWZAsRoscgrjP4JK6iLVZ18gBN7vt8Bd_veiCjlIVtplhBxNyBjFC4RyfRFl0OWNJ-gnPxxGYgdpgFyOARiK6WK2fDuEdY_VIERGrKI5pajLDZcno2uBdJiZbI8OGDGaqcGXplAe/w640-h32/permissions.png" width="640" /></a></div><br /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 16pt; white-space: pre-wrap;">Troubleshooting points</span></p><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In the host running the WebRTC client, take a network trace and confirm that there’s a CreatePermission Success for at least one of the remote candidates.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">If no CreatePermission requests are sent, or none of them is successfully accepted, then no relaying will be possible.</span></p><h1 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Exchanging ICE connectivity checks over TURN</span></h1><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Once the TURN server is reached, a relay allocation reserved and a permission created, there are the conditions for exchanging ICE connectivity checks over TURN.</span></p><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">These are performed by sending STUN <a href="https://www.rfc-editor.org/rfc/rfc8489#section-9.1" target="_blank">Binding Requests with short term credentials</a>; the peculiarity with TURN is that these Binding Requests are encapsulated inside a TURN Send Indication, addressed to the remote peer.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Wireshark will nicely solve this encapsulation for you, and instead of showing a Send Indication will show you its content, the Binding Request.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The TURN server will relay the Binding Request to the remote peer, performing the relay for the first time. The expected outcome is that the remote entity will respond with a Binding Success, which the TURN server will encapsulate inside a Data Indication and deliver to the client.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">If that happens, then the client has learned that the remote candidate is indeed reachable via TURN and that’s a suitable candidate pair for exchanging media and data.</span></p><h2 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Troubleshooting points</span></h2><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In the host running the WebRTC client, take a network trace and confirm that there are Binding Requests carried over TURN that receive a Binding Success.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">If Binding Success responses are not received, then something is preventing it and the best way to investigate is to take network traces on the TURN server host, if possible. Those traces will tell you whether the Binding Requests are correctly leaving the TURN server towards the remote party and whether the Binding Success responses are being received or not.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">It’s possible that the remote endpoint is simply unreachable from the TURN service, and in this case the ICE candidates pair will be marked as unusable.</span></p><h1 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Exchanging media and/or data over TURN</span></h1><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The last fundamental step is the actual exchange of packets through the relay. The typical type of packets is RTP.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Once the connectivity checks will be successful, if the client has elected the relay candidate as the one to be used, then RTP can start flowing. You’ll be able to see the RTP packets flowing in both directions, typically with video and audio multiplexed.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">There are two ways for transmitting data:</span></p><ul style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; margin-bottom: 0px; margin-top: 0px;"><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space: pre-wrap;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Indications</span></p></li><li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; margin-left: 15px; vertical-align: baseline; white-space: pre-wrap;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Channels</span></p></li></ul><div><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">A Send Indication carries the data (RTP) and destination from the client to the TURN server. The TURN server, granted the allocation exists and the permission allows it, will extract the data and send it to the destination from the allocated relay transport address.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">When the data arrives from the remote peer to the relay transport address, then the TURN server, after performing the above checks, will encapsulate the data inside a Data Indication and send it to the client.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">There is a more efficient way though to exchange data: the client can define a Channel (through the ChannelBind request), which associates a channel ID to a remote party. From that moment both the client and the TURN server can exchange data via ChannelData messages carrying just the channel ID and data, omitting the remote transport address. This reduces the network and computing overhead and it is typically chosen against the use of Indications.</span></p><h2 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Troubleshooting points</span></h2><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In the host running the WebRTC client, take a network trace and confirm that data is being sent from the client with Send Indications or ChannelData messages, and to the client with Data Indications and ChannelData messages.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In case of monodirectional media, it’s advisable to take network traces on the TURN server host to clarify whether the media is being exchanged or not on the relay side with the remote peer.</span></p><h1 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Encrypted TURN</span></h1><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">It’s possible to use TURN over TLS, with all the data exchanged encrypted. In this case using Wireshark as described won’t allow you to see the details of the requests and responses, and troubleshooting is harder.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">One possible approach is to first of all ensure that all the operations described previously happen correctly when using unencrypted TURN (over UDP or TCP). It’s very likely that the TURN service you are using is accessible over unencrypted UDP (default behavior): before moving to TLS ensure UDP works fine.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Wireshark will show you anyway the TLS connections established with the server, so that will confirm whether the connection was successful, the TLS session established, and some application data exchanged.</span></p><h1 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Useful tools</span></h1><h2 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Wireshark</span></h2><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><a href="https://www.wireshark.org/download.html" target="_blank">Wireshark</a> is available for a variety of platforms; it’s a fundamental tool to understand what’s happening between the local WebRTC client and the remote server.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">It comes with filters that detect the type of packets. You can use `stun` to filter out STUN and TURN packets, and even select specific TURN transactions, like `stun.type.method==0x0003` to show Allocate Request and Responses.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Saving a trace into a pcap file and making it available to others helps enormously the ability to troubleshoot.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Wireshark can be used for both capturing and just displaying captures.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">There are cases where the dissectors, i.e. the interpreters of the packets, don’t recognize a TURN transaction. For example this happens when they happen over a non-default port (3478 for UDP and TCP, 5349 for TLS). To “help” Wireshark, right click on a packet, select “Decode As…” and set ‘STUN’ as protocol: it will correctly interpret all the packets using that non-default port.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The same applies for RTP: when signalling is not available to Wireshark, then UDP packets containing RTP may not be correctly interpreted. Use the same “Decode As…” method.</span></p><h2 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">tcpdump</span></h2><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">On the server side, any tool for packet capture would do, with <a href="https://www.tcpdump.org/" target="_blank">tcpdump</a> being a common solution.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Save the trace into a pcap file with the `-w` option, e.g. `tcpdump -n -v -w trace_1.pcap`, copy it to your machine and use Wireshark to display the packets.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><h2 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">WebRTC samples, Trickle ICE</span></h2><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This <a href="https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/" target="_blank">open source tool</a> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">allows you to verify the browser can correctly gather `relay` candidates with the given TURN server details (URL, username, credential).</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Before troubleshooting a client implementation, ensure that this tool can correctly access the TURN resources you’re referring to.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><h2 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">turnutils_uclient</span></h2><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The popular open source implementation of a TURN server, coturn, comes with a <a href="https://github.com/coturn/coturn/blob/master/README.turnutils" target="_blank">tool</a> that simulates a client. A plethora of options are available, allowing you to test specific aspects of the TURN operations, e.g. using Send Indications or using Channels, etc.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Use `turnutils_uclient` to ensure the TURN service you want to use is accessible correctly with the given TURN settings, You’ll also get information about the round trip time and jitter.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><h2 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Chrome webrtc-internals</span></h2><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">When using Chrome, the best way to understand what’s happening is to open a tab on chrome://webrtc-internals/. It will show you all the information related to each RTCPeerConnection being managed by the browser at that point, including the list of ICE candidates, the details of the TURN server being used (except the credential for obvious reasons), including the iceTransportPolicy (‘all’ or ‘relay’), the chosen ICE candidates pair, statistics on media transfer, etc.</span></p><br style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;" /><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Search for `relay` candidates and verify the client is able to retrieve them from the TURN service, and whether they are selected as the candidate pair or not.</span></p><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><h2 dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 16pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Conclusions</span></h2><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"></span></p><p dir="ltr" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This article should provide a good checklist for troubleshooting the connection to a TURN service. There is much more to say, in particular for what concerns browsers different than Chrome and server-side investigations: I plan to write about it in the future.</span></p><h1 style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;"><br /></h1><div><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div><div class="yj6qo" style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;"></div>Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com1tag:blogger.com,1999:blog-9031922773224056133.post-22195510499092653702021-03-11T11:01:00.000+01:002021-03-11T11:01:17.333+01:00[off topic] Differences between running and cycling<p> I'm a passionate runner, and always considered cycling as something fun, e.g. mountain-biking, but difficult to practice regularly. There's a lot of overhead in cycling, like the preparation, bike maintenance, dealing with city traffic, etc.</p><p>Anyway about eight months ago I bought a road bike and felt in love with it. Soon after that I discovered <a href="https://zwift.com" rel="nofollow" target="_blank">Zwift</a> and that gave an additional dimension to the sport: practice whenever you want from home, with accurate power measurements and a way to socialise with distant people. That was a game changer.</p><p>In five months I cycled 1600 virtual Km and climbed almost 17 virtual Km. Meanwhile my running performance, instead of degrading, improved, and that surprised me.</p><p>Anyway what I wanted to write about is a great article I read, <a href="https://www.researchgate.net/publication/24205037_Physiological_Differences_Between_Cycling_and_Running" rel="nofollow" target="_blank">"Physiological Differences Between Cycling and Running"</a>. It's a review of articles published in that area. Some conclusions are very interesting.</p><p>In general it seems sports medicine is still inconclusive for many aspects, and coaches may still have an advantage by following empirical/heuristic approaches in comparison with research-driven indications.</p><p>But more specifically, some notes from the conclusions:</p><p>- For the same person, VO2max depends on the speciality (i.e. runners achieve higher values on treadmill than cycle ergometer)</p><p>- There seems to be more physiological transfer from running to cycling than the other way around</p><p>- Pedalling cadence impacts the metabolic response during cycling, but also during a following run (at least in the short term)</p><p>- The Lactate Threshold is lower for athletes when not practicing their speciality, i.e. the Lactate Threshold depends on the training method</p><p>- Both female and male are impacted in the same way when comparing VO2max for running and cycling</p><p>- Triathletes have similar max Heart Rate when running and cycling, again pointing to the importance of the actual speciality used in training</p><p>- The position when cycling makes it harder to breathe</p><p>and probably other important elements that I wasn't able to fully grasp.</p><p><br /></p>Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-22363043834595269092021-03-10T11:55:00.002+01:002021-03-10T11:55:33.755+01:00Notes on STUN protocol<p> Since I needed to see a few details in the handling of attributes in STUN responses, I thought of going through the whole STUN protocol RFC again and take notes on the most important parts.</p><p>I put my notes in some slides, and I'm sharing then in Slideshare in case somebody else may find them useful too:</p><p><br /></p>
<iframe allowfullscreen="" frameborder="0" height="485" marginheight="0" marginwidth="0" scrolling="no" src="//www.slideshare.net/slideshow/embed_code/key/EfRt24YCe4G6cz" style="border-width: 1px; border: 1px solid #CCC; margin-bottom: 5px; max-width: 100%;" width="595"> </iframe> <div style="margin-bottom: 5px;"> <strong> <a href="//www.slideshare.net/GiacomoVacca/stun-protocol" target="_blank" title="STUN protocol">STUN protocol</a> </strong> from <strong><a href="https://www.slideshare.net/GiacomoVacca" target="_blank">Giacomo Vacca</a></strong> </div>Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-46295657852518213982021-02-19T09:45:00.001+01:002021-02-19T09:45:36.930+01:00Extracting RTP streams from network captures<p><span style="font-family: "Helvetica Neue"; font-size: 13px;">I needed an efficient way to programmatically extract RTP streams from a network capture.</span></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">In addition I wanted to:</p><ul class="ul1"><li class="li3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span class="s1" style="font-size: 12px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"></span>save each stream into a separate pcap file.</li><li class="li3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span class="s1" style="font-size: 12px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"></span>extract SRTP-negotiated keys if present and available in the trace, associating them to the related RTP (or SRTP if the negotiation succeeded) stream.</li></ul><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">Some caveats:</p><ul class="ul1"><li class="li3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span class="s1" style="font-size: 12px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"></span>In normal conditions the negotiation of SRTP sessions happens via a secure transport, typically SIP over TLS, so the exchanged crypto information may not be available from a simple network capture.</li><li class="li3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span class="s1" style="font-size: 12px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"></span>There are ways to extract RTP streams using Wireshark or tcpdump; it’s not necessary to do it programmatically.</li></ul><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">All this said I wrote a small tool (<a href="https://github.com/giavac/pcap_tool">https://github.com/giavac/pcap_tool</a>) that parses a network capture and tries to interpret each packet as either RTP/SRTP or SIP, and does two main things:</p><ul class="ul1"><li class="li3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span class="s1" style="font-size: 12px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"></span>save each detected RTP/SRTP stream into a dedicated pcap file, which name contains the related SSRC.</li><li class="li3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span class="s1" style="font-size: 12px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"></span>print a summary of the crypto information exchanged, if available.</li></ul><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">With those two elements, it’s then possible to decrypt an SRTP stream, depending on the availability of the exchanged crypto information, and also decode it into audio, depending on the codec.</p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">Decryption and decoding is not part of my tool, but can be achieved easily with other tools, like <b>pjsip’s pcaputil</b>.</p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">I might integrate that part into pcap_tool in the future. Again not because it’s strictly necessary, but to start getting more control on the parsing and manipulation. This may reveal to be useful in the future.</p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><a href="https://github.com/giavac/pcap_tool" rel="nofollow" target="_blank">pcap_tool is available here</a> for anybody interested in using it and may perhaps wish to change or extend some parts.</p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">You can just clone it and build it as described in the README.</p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">An example output:</p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><i>./pcap_tool -d ../../trace_20210218_1.pcap</i></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><i><br /></i></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><i>[…]</i></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><i><br /></i></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><i>Extracted 1092 RTP frames</i></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><i><span class="Apple-tab-span" style="white-space: pre;"> </span>Detected RTP Stream: 0x7a2179fa<span class="Apple-tab-span" style="white-space: pre;"> </span>Source port:22248 - Destination port:4000 - Packets: 544 (./stream-0x7a2179fa.pcap)</i></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><i><span class="Apple-tab-span" style="white-space: pre;"> </span>Detected RTP Stream: 0x772dc5d7<span class="Apple-tab-span" style="white-space: pre;"> </span>Source port:4000 - Destination port:22248 - Packets: 548 (./stream-0x772dc5d7.pcap)</i></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><i><br /></i></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><i><br /></i></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><i>source port: 22248 - tag: 3 - suite: AES_CM_128_HMAC_SHA1_80 - key: /1TI6DJWHk7fBJY1yBp7L51uEz1JJ2n6CcQAAsJM</i></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><i>-----</i></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><i>source port: 4000 - tag: 4 - suite: AES_CM_128_HMAC_SHA1_32 - key: mPytX24bRmyNgMaqQSxP8dMMqdkkmQeHgC2Ttb3v</i></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><i>source port: 4000 - tag: 3 - suite: AES_CM_128_HMAC_SHA1_80 - key: J1YS1owJDKAFdq5cRF+JtektYDf6IiowCAeijeal</i></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><i>source port: 4000 - tag: 2 - suite: AES_256_CM_HMAC_SHA1_32 - key: 5A9R8O8MCzbuGvJ08WWNJcNHsPaEcEp1ZDp5DunknZ+bZ2JQaVpZ2qmqraTmgQ==</i></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><i>source port: 4000 - tag: 1 - suite: AES_256_CM_HMAC_SHA1_80 - key: ZcZn1IY++2xsSIk/U1GsHSGp+OI/BYIocv/40ldJB28bcNeMmYzs4z4ozrNQ5Q==</i></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><i>-----</i></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">That network capture contained 2 SRTP streams, which have been saved separately into <b>stream-0x7a2179fa.pcap</b> and <b>stream-0x772dc5d7.pcap</b> files respectively.</p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">For the negotiation it’s visible what the sender from port 22248 (owner of the 0x7a2179fa stream) used as crypto information, and looking at the same tag (3 in this case) it’s possible to see what crypto information was used by the sender of 0x772dc5d7 stream from port 4000.</p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">With this it’s possible to decrypt (and decode since G.711 was used) with pjsip’s pcaputil with something like:</p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><i>pcaputil -c AES_CM_128_HMAC_SHA1_80 -k /1TI6DJWHk7fBJY1yBp7L51uEz1JJ2n6CcQAAsJM stream-0x7a2179fa.pcap stream-0x7a2179fa.wav</i></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">and have the audio from that stream into a WAV file.</p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">How to build pcaputil (in fact all pjsip’s applications) is widely documented but I also described it in the appendix of <a href="https://www.giacomovacca.com/2020/11/testing-sip-platforms-and-pjsip.html"><span class="s2" style="color: #dca10d;">https://www.giacomovacca.com/2020/11/testing-sip-platforms-and-pjsip.html</span></a><span class="Apple-converted-space"> </span></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;">The call in the example was generated in fact with pjsua.</p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 13px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p>Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-27678734275844277772020-12-04T09:52:00.000+01:002020-12-04T09:52:09.264+01:00SIP - Connection reuse vs Persistent connection<p>It goes without saying that SIP solutions are impacted by NAT. So much that some scenarios required integration to <a href="https://tools.ietf.org/html/rfc3261" rel="nofollow" target="_blank">RFC 3261</a>, e.g. with <a href="https://tools.ietf.org/html/rfc3581" rel="nofollow" target="_blank">RFC 3581</a>, which defined the '<i>rport</i>' attribute to be added in the Via header (integrating the '<i>received</i>' attribute): with that information, responses could be routed to the source port of the related request, and not on the advertised port in the original Via.</p><p>That was called <b>Symmetric Response</b>, and applied to connection-less transports (UDP), while, as mentioned in <a href="https://tools.ietf.org/html/rfc6314" rel="nofollow" target="_blank">RFC 6314</a>, it's not necessary when using reliable transports (TCP in most cases): SIP responses can be sent back on the same connection on which the request arrived.</p><p>Also from RFC 3261, chapter 18, Transport layer, client behaviour:</p><div data-en-clipboard="true" data-pm-slice="1 1 []"><blockquote>"For reliable transports, the response is normally sent on the connection on which the request was received."</blockquote></div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []">But the client needs to be prepared to receive the response on a new connection:</div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []"><div data-en-clipboard="true" data-pm-slice="1 1 []"><blockquote>"[...] the transport layer MUST also be prepared to receive an incoming connection on the source IP address from which the request was sent and port number in the "sent-by" field."</blockquote></div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []">That obviously would require the ability to create such connection from the server to the client.</div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []">Anyway when a reliable connection between two SIP entities is up, after a transaction is already concluded, there are two interesting opportunities:</div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []">- Use that same connection for more requests from the client</div><div data-en-clipboard="true" data-pm-slice="1 1 []">- Use that same connection for more requests from the server</div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []">where for "client" I refer to the entity that created the connection and sent the initial request, and "server" is the entity that accepted the connection and delivered the response(s).</div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []">The first case is mentioned in the same chapter on Transport layer:</div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []"><div data-en-clipboard="true" data-pm-slice="1 1 []"><blockquote>"If a request is destined to an IP address, port, and transport to which an existing connection is open, it is RECOMMENDED that this connection be used to send the request, but another connection MAY be opened and used."</blockquote></div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []">This is what's referred to as "<b>Persistent connection</b>", as mentioned in <a href="https://tools.ietf.org/html/rfc5923" rel="nofollow" target="_blank">RFC 5923</a>:</div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []"><div data-en-clipboard="true" data-pm-slice="1 1 []"></div><blockquote><div data-en-clipboard="true" data-pm-slice="1 1 []">"The SIP protocol includes the notion of a persistent connection</div><div> [...], which is a mechanisms to insure that</div><div> responses to a request reuse the existing connection that is</div><div> typically still available, as well as reusing the existing</div><div> connections for other requests sent by the originator of the</div><div> connection."</div></blockquote><div></div><div><br /></div><div>The second case (using the same connection from future requests from the server) is instead the subject of RFC 5923, and it is defined as "<b>Connection reuse</b>".</div><div><br /></div><div>Once the connection is up, it seems a good opportunistic approach to reuse it, but an important limitation is mandated:</div><div><br /></div><div><div data-en-clipboard="true" data-pm-slice="1 1 []"><blockquote>"Unlike TCP, TLS connections can be reused to send requests in the backwards direction since each end can be authenticated when the connection is initially set up."</blockquote></div></div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []">In other words, only TLS connections formed by exchanging certificates can be reused, because the identities have been mutually verified.</div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []">The way the client can tell the server that connection reuse is desired is with a new parameter to be added in the Via header: '<i>alias</i>'.</div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []">In general, RFC 5923 at chapter 5 clarifies:</div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []"><div data-en-clipboard="true" data-pm-slice="1 1 []"></div><blockquote><div data-en-clipboard="true" data-pm-slice="1 1 []">"The act of reusing a connection needs</div><div> the desired property that requests get delivered in the backwards</div><div> direction only if they would have been delivered to the same</div><div> destination had connection reuse not been employed."</div></blockquote><div></div><div><br /></div><div><br /></div></div><div data-en-clipboard="true" data-pm-slice="1 1 []">Last important bit, not to be left implicit: persistent connections don't imply connection reuse, as RFC 5923 clarifies:</div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []"></div><blockquote><div data-en-clipboard="true" data-pm-slice="1 1 []">"[...] Persistent connections do not</div><div> imply connection reuse."</div></blockquote><div></div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div>So this post is basically sharing my own notes on this topic, which maybe somebody else (including me) can find useful in the future.</div></div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div></div><div data-en-clipboard="true" data-pm-slice="1 1 []"><br /></div></div>Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-1280042885645203462020-11-24T09:54:00.003+01:002020-12-04T09:54:00.187+01:00Dissecting traces with DTMF tones<p>I'm sure I belong to the large group of people who love to analyse network traces with tools like Wireshark. Being able to see the details of a packet or datagram down to the level of the bits is not only extremely useful, but also fascinating.</p><div><br /></div><div>Time ago I wrote a dissector for Wireshark, using the Lua interface, and that was fun (I see it's still available <a href="https://github.com/sipcapture/hep-wireshark" rel="nofollow" target="_blank">here</a>). <a href="https://gitlab.com/wireshark/wireshark/-/wikis/Lua" rel="nofollow" target="_blank">The official recommendation</a> is to use Lua only for prototyping and testing, but when performances are not key and there isn't the intent to add the dissector to the official distribution, it's fast and effective.</div><div><br /></div><div>In order to parse network traces with audio and extract it into the payload first, and then decode it into a WAV file, C is a viable solution. I wrote about <a href="https://www.giacomovacca.com/2013/06/voip-calls-encoded-with-silk-from-rtp.html" rel="nofollow" target="_blank">a program that does that here </a>and since it attracted some attention and feedback I wrote <a href="https://www.giacomovacca.com/2017/01/voip-calls-encoded-with-silk-from-rtp.html" rel="nofollow" target="_blank">an updated version later</a>.</div><div><br /></div><div>More recently I wanted to identify programmatically the presence (and value) of DTMF tones - as RTP Events, RFC 2833 - in network traces. This time rather than using C, I wanted to integrate it with python, and <a href="https://scapy.net/" rel="nofollow" target="_blank">scapy</a> seemed a good choice.</div><div><br /></div><div><i>scapy</i> is quite complete, but interestingly it doesn't have a parser for the RTP Event extension. So I thought of mapping the raw content in the RTP payload to a structure, with the help of <a href="https://docs.python.org/3/library/ctypes.html" rel="nofollow" target="_blank">C types</a>.</div><div><br /></div><div>This it the core of the program:</div><div><br /></div><div></div><blockquote><div>def process_pcap(file_name, sut_ip):</div><div><br /></div><div> for (pkt_data, pkt_metadata,) in RawPcapReader(file_name):</div><div><br /></div><div> ether_pkt = Ether(pkt_data)</div><div><br /></div><div> # A little housekeeping to filter IPv4 UDP packets goes here</div><div> ...</div><div><br /></div><div> if ether_pkt.haslayer(UDP):</div><div> udp_pkt = ether_pkt[UDP]</div><div><br /></div><div> # Get the raw UDP packet into an RTP structure</div><div> rtp_pkt = RTP(udp_pkt["Raw"].load)</div><div><br /></div><div> ptype = rtp_pkt.payload_type</div><div><br /></div><div> # Assume payload type 96 or 101 are used for RTP events</div><div> if (ptype == 96 or ptype == 101):</div><div> rtpevent_content = rtp_pkt.payload</div><div><br /></div><div> # map the payload into an RTPEvent object</div><div> rtpevent_struct = RTPEvent.from_buffer_copy(rtpevent_content.load)</div><div></div></blockquote><div><br /></div><div>The RTPEvent class looks like this:</div><div><br /></div><div></div><blockquote><div>class RTPEvent(ctypes.BigEndianStructure):</div><div> _fields_ = [</div><div> ('event_id', ctypes.c_uint8),</div><div> ('end_of_event', ctypes.c_uint8, 1),</div><div> ('reserved', ctypes.c_uint8, 1),</div><div> ('volume', ctypes.c_uint8, 6),</div><div> ('duration', ctypes.c_uint16)</div><div> ]</div></blockquote><div></div><div><br /></div><div>so once mapped, the <i>rtpevent_struct</i> object will have its DTMF-specific details, in particular with the digit contained in <i>rtpevent_struct.event_id</i>, and the indication whether it's the marker of end of the event in the <i>end_of_event </i>bit.</div><div><br /></div><div>All the other information (source/destination IP address and port, timestamp, SSRC) is obviously available in the UDP and RTP portion, so it's easy to adapt to your needs and filter out the DTMF tones for the streams you're interesting in.</div><div><br /></div><div><br /></div>Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-18480762941675376002020-11-23T10:07:00.002+01:002020-12-04T09:54:27.974+01:00Kubernetes role-based authorisation for controller applications<p>There are many scenarios where an application running inside a Kubernetes environment may need to interact with its API.</p><div>For example, an application running inside a Pod may need to retrieve real time information about the availability of other applications' endpoints.</div><div><br /></div><div>This may be a form of service discovery that integrates or extend the native Kubernetes internal service discovery. In most cases, DNS records are associated to a Service and provide the list of active Endpoints for that Service, with a proper TTL. There are situations though where those DNS records are not available, an application is not able to use them directly, or what's needed is more than the private IP addresses associated with the Endpoints.</div><div><br /></div><div>If interacting with the Kubernetes API from inside an application is needed, then there are two main areas to consider: Authentication and Authorisation.</div><div><br /></div><div>Every Pod has a Service Account associated to it, and applications running inside that Pod can use that Service Account. Without a specific configuration the Service Account will default to a generic namespace and generic authorisation.</div><div><br /></div><div>It's possible instead to define a more specific Service Account, with fine grained permissions to access the API. This Service Account can then be linked to a Pod with a Role-based approach.</div><div><br /></div><div>A reference: <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account" rev="en_rl_none">https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account</a>/ </div><div><br /></div><div>You can get a list of available Service Accounts with an intuitive:</div><div><br /></div><div><blockquote># kubectl get serviceaccounts</blockquote></div><div><br /></div><div>which is likely to show you a single 'default' service.</div><div><br /></div><div>Service Accounts may have a namespace scope. You can check what Service Accounts are associated to a pod with a command like:</div><div><br /></div><div><blockquote># kubectl -n NAMESPACE get pods/PODNAME -o yaml | grep serviceAccountName</blockquote></div><div><br /></div><div>Service Account authentication can use the token reachable from inside a Pod, in the <i>/var/run/secrets/kubernetes.io/serviceaccount </i>directory, under the namespace-specific directory, e.g. <i>/var/run/secrets/kubernetes.io/serviceaccount/NAMESPACE/token.</i></div><div><i><br /></i></div><div>Those tokens are also visible as Mounts in the related containers.</div><div><i><span data-markholder="true"></span></i></div><div><br /></div><div> For internal requests, Kubernetes provides a local default HTTPS endpoint at <i>https://kubernetes.default.svc - </i> so a way to discover the details of a Service Account for a given namespace could be:</div><div><br /></div><div><i></i></div><blockquote><div><i>#!/bin/bash</i></div><div><br /></div><div><i># Point to the internal API server hostname</i></div><div><i>APISERVER=https://kubernetes.default.svc</i></div><div><br /></div><div><i># Path to ServiceAccount token</i></div><div><i>SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount</i></div><div><br /></div><div><i># Read this Pod's namespace</i></div><div><i>NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)</i></div><div><br /></div><div><i># Read the ServiceAccount bearer token</i></div><div><i>TOKEN=$(cat ${SERVICEACCOUNT}/token)</i></div><div><br /></div><div><i># Reference the internal certificate authority (CA)</i></div><div><i>CACERT=${SERVICEACCOUNT}/ca.crt</i></div><div><br /></div><div><i># Explore the API with TOKEN</i></div><div><i>curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api/v1/namespaces/${NAMESPACE}/pods</i></div></blockquote><div><i></i></div><div><br /></div><div><br /></div><h2 style="text-align: left;">Accessing the internal API programmatically</h2><div><br /></div><div>With common python libraries as <a href="https://github.com/kubernetes-client/python" rev="en_rl_none">https://github.com/kubernetes-client/python</a> (available on debian with the 'python3-kubernetes' package) it's extremely easy to automate the invocation of the internal APIs.</div><div><br /></div><div>Most of the examples assume you're running your program as a user, and refer to the local kube config file, but when running inside a container it's possible to inherit the Service Account token associated with the hosting Pod.</div><div><br /></div><div>For this, instead of using </div><div><br /></div><div><blockquote>config.load_kube_config()</blockquote></div><div><br /></div><div>use</div><div><br /></div><div><blockquote>config.load_incluster_config()</blockquote></div><div><br /></div><div>Then you can instantiate your API client object:</div><div><br /></div><div><blockquote>v1 = client.CoreV1Api()</blockquote></div><div><br /></div><div>and either do a single request, like getting a list of all pods inside any namespace:</div><div><br /></div><div><blockquote>ret = v1.list_pod_for_all_namespaces(watch=False)</blockquote></div><div><br /></div><div>or a list of pods belonging to a namespace and matching a specific application label, like:</div><div><br /></div><div><blockquote>ret = v1.list_namespaced_pod(namespace, label_selector=app_name, watch=False)</blockquote></div><div><br /></div><div>or you can "watch" some resources, which basically means subscribing to such resource updates and getting a notification at each change:</div><div><br /></div><div></div><blockquote><div>w = watch.Watch()</div><div>for event in w.stream(v1.list_namespace, _request_timeout=60):</div><div> ...</div></blockquote><div></div><div><br /></div><div>Each event can be ADDED, DELETED and MODIFIED, and carries a rich set of information associated to the current status of the resource.</div><div><br /></div><h2 style="text-align: left;">Defining your specific ServiceAccount</h2><div><br /></div><div>Before getting to that point, though, you need to define your non-default Service Account and assign specific permissions to it. To achieve this, the role-based approach can be used.</div><div><br /></div><div>First of all define a ServiceAccount resource:</div><div><br /></div><div></div><blockquote><div>apiVersion: v1</div><div>kind: ServiceAccount</div><div>metadata:</div><div> labels:</div><div> app.kubernetes.io/component: mycomponent</div><div> name: mycomponent-serviceaccount</div></blockquote><div></div><div><br /></div><div>Then define a role associated to this resource:</div><div><br /></div><div></div><blockquote><div>apiVersion: rbac.authorization.k8s.io/v1</div><div>kind: Role</div><div>metadata:</div><div> name: myrole</div><div> namespace: mynamespace</div><div> labels:</div><div> [...]</div><div> annotations:</div><div> [...]</div><div>rules:</div><div>[...]</div><div> - apiGroups:</div><div> - ""</div><div> resources:</div><div> - pods</div><div> - endpoints</div><div> verbs: ["get", "list", "watch"]</div></blockquote><div></div><div><br /></div><div>This example adds the permission to get, list, or watch the list of pods and endpoints in the given namespace.</div><div><br /></div><div>Create a role binding:</div><div><br /></div><div></div><blockquote><div>apiVersion: rbac.authorization.k8s.io/v1</div><div>kind: RoleBinding</div><div>metadata:</div><div> name: myrolebinding</div><div> namespace: mynamespace</div><div> labels:</div><div>[...]</div><div> annotations:</div><div>[...]</div><div>roleRef:</div><div> apiGroup: rbac.authorization.k8s.io</div><div> kind: Role</div><div> name: myrole</div><div>subjects:</div><div> [...]</div><div> - kind: ServiceAccount</div><div> name: mycomponent-serviceaccount</div><div> namespace: mynamespace</div></blockquote><div></div><div><br /></div><div>Whenever a Role cannot be restricted to a namespace, for example if it needs to access cluster-wide resources like Nodes, then the ClusterRole resource is available.</div><div><br /></div><h2 style="text-align: left;">References and other sources</h2><div>"Access clusters using the Kubernetes API", https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/</div><div><br /></div><div>An interesting post about asynchronous watches with Python: https://medium.com/@sebgoa/kubernets-async-watches-b8fa8a7ebfd4</div><div><br /></div><div>"Kubernetes Patterns", an ebook by Redhat: https://www.redhat.com/cms/managed-files/cm-oreilly-kubernetes-patterns-ebook-f19824-201910-en.pdf</div><div><br /></div><div><br /></div><div><br /></div><div><i><span data-markholder="true"></span></i></div><div><br /></div><div><br /></div>Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-72094526522792122452020-11-20T13:27:00.014+01:002020-12-16T13:10:03.105+01:00Testing SIP platforms and pjsip<p>There are various levels of testing, from unit to component, from integration to end-to-end, not to mention performance testing and fuzzing.</p><div>When developing or maintaining Real Time Communications (RTC or VoIP) systems, all these levels (with the exclusion maybe of unit testing) are made easier by applications explicitly designed for this, like <a href="http://sipp.sourceforge.net/doc/reference.html" rel="nofollow" target="_blank">sipp</a>.</div><div><br /></div><div><i>sipp</i> has a deep focus on performance testing, or using a simpler term, load testing. Some of its features allow to fine tune properties like call rate, call duration, simulate packet loss, ramp up traffic, etc. In practical terms though once you have the flexibility to generate SIP signalling to negotiate sessions and RTP streams, you can use <i>sipp</i> for functional testing too.</div><div><i>sipp</i> can act as an entity generating a call, or receiving a call, which makes it suitable to surround the system under test and simulate its interactions with the real world.</div><div><br /></div><div>What <i>sipp</i> does can be generalised: we want to be able to simulate the real world that surrounds (or will surround) our system in Production. From this point of view <i>sipp</i> is not the only answer, and projects often use other tools, or a combination of other tools.</div><div><br /></div><div>One simple and effective approach is re-using RTC applications and build the testing tool around them. When a system is built around an application, it's likely that the people working on it are familiar enough with the application to re-use it to mock the external world. This is often achieved with <a href="https://www.asterisk.org/" rel="nofollow" target="_blank">Asterisk</a> or <a href="https://freeswitch.com/" rel="nofollow" target="_blank">FreeSWITCH</a>. They both expose an API for originating calls, and surely can play the role of called party (or "absorbers", or "parrots" depending on their main scope and terminology).</div><div><a href="https://www.kamailio.org/w/" rel="nofollow" target="_blank">Kamailio</a> also can be used to generate calls, even though its core focus on signalling makes it slightly more complex to use in generic cases.</div><div><br /></div><div>Unless behavioural changes are put in place, such solutions imply compromising on the SIP stacks in use. <i>Asterisk</i> or <i>FreeSWITCH</i> won't make it too easy to generate an INVITE with a wrongly formatted SIP header, for example, while <i>sipp</i> is much more flexible, ad the SIP messages can be mocked down to the single character. What typically happens is that <i>sipp</i> is used to generate or receive calls when specific syntax requirements for the signalling are needed, while <i>Asterisk</i> and <i>FreeSWITCH</i> can be used in more permissive cases, where what's important is a generic session establishment.</div><div><br /></div><div>When dealing with media (typically, RTP streams) is necessary, then <i>sipp</i> provides at least two methods: re-playing an RTP stream from a trace (pcap file), or encoding a WAV file into a stream. Recently <i>sipp</i> added the ability to play RTP Events separately (DTMF tones as in RFC 2833 - I think the first patch with this functionality was <a href="https://sourceforge.net/p/sipp/patches/50" rel="nofollow" target="_blank">this</a>). sipp is not able to transcode or generate non-PCM streams, but still it can play a non-PCM stream with just some limitations, which covers most of typical cases.</div><div><br /></div><div>Less generic scenarios where RTC applications like <i>Asterisk</i> and <i>FreeSWITCH</i> can be useful are the ones requiring SRTP (encrypted RTP). Even though <i>sipp</i> can be used to negotiate SRTP, by adapting the SDP portion of the offer/answer, it doesn't provide a solution to <i>generate</i> SRTP streams.</div><div><br /></div><div>In this case a very useful item to add to your toolbox is <a href="https://www.pjsip.org/" rel="nofollow" target="_blank">pjsip</a>, which is a SIP stack library (used also by <i>Asterisk</i> and <b>chan_pjsip</b> being the current recommended SIP channel, as opposed to the older chan_sip) that exposes an API and also a command-line option (<b>pjsua</b>). <i>pjsua</i> can be used directly, with either command line arguments or a configuration file, or it's possible to use <i>pjsip</i> library to write programs with languages like python: this makes it very flexible and helps its integration in existing and new testing systems.</div><div><br /></div><div>With <i>pjsip</i>, it's possible to generate calls that play audio and DTMF tones, in a similar way than <i>sipp</i>, but also encrypt RTP and establish SRTP streams.</div><div><br /></div><h2 style="text-align: left;">pjsua </h2><div><br /></div><div>The easiest approach is to build the pjsip project and use the pjsua binary (you can see a procedure in the Appendix).</div><div><i>pjsua</i> accepts command-line arguments, but can receive arguments from a configuration file, which makes it easier to read. For example you could just</div><div><br /></div><div><blockquote># pjsua --config-file pjsua.cfg</blockquote></div><div><br /></div><div>where pjsua.cfg contains just the caller and callee:</div><div><br /></div><div></div><blockquote><div>sip:bob@example.com</div><div>--id=sip:alice@example.com</div></blockquote><div></div><div><br /></div><div>A more sophisticated configuration file contains instructions on codecs and encryption, e.g.</div><div><br /></div><div></div><blockquote><div>sip:bob@example.com</div><div>--id=sip:alice@example.com</div><div>--use-srtp=0</div><div>--srtp-secure=0</div><div>--realm=*</div><div>--log-level=6</div><div>--no-vad</div><div>--dis-codec GSM</div><div>--dis-codec H263</div><div>--dis-codec iLBC</div><div>--dis-codec G722</div><div>--dis-codec speex</div><div>--dis-codec pcmu</div><div>--dis-codec pcma</div><div>--dis-codec opus</div><div>--add-codec pcma</div><div>--null-audio</div><div>--auto-play</div><div>--play-file /some_audio.wav</div></blockquote><div></div><div><br /></div><div>Since I mentioned SRTP as a possible key element for using <i>pjsip</i>, let's look into the related options:</div><div><br /></div><div><b>--use-srtp=0</b></div><div><b>--srtp-secure=0</b></div><div><br /></div><div>'use-srtp' can be 0, 1 or 2, and means "disabled", "optional" and "mandatory", respectively.</div><div><br /></div><div>With "optional" pjsua offers both plain and encrypted RTP at the same time, and the callee entity can decide. With "mandatory" it will only offer SRTP, and the callee will have to either accept or reject.</div><div><br /></div><div>'srtp-secure' refers to the use of TLS, and can also be 0, 1 or 2, meaning "not required", use "tls", or use "sips" respectively. Needless to say, in normal scenarios you want to protect the SRTP crypto information carried in the SDP, so you want to encrypt signalling too. SIP over TLS is the typical solution. For testing purposes you may prefer making it easier to check the content of signalling, and use 'srtp-secure=0'.</div><div><br /></div><div>'no-vad' formally should be used to disable silence detection; in practice you want this option when generating a call from a machine that doesn't have a sound card.</div><div><br /></div><div>Similarly, 'null-audio' disables the requirement to play the audio, required when the calls are generated from a host with no sound interfaces.</div><div><br /></div><div>'dis-codec' is used to disable a codec from the negotiation, and 'add-codec' instead selects a codec to be added to the offer. This adds flexibility, and it's also worth noting that video codecs are available too.</div><div><br /></div><div><br /></div><h2 style="text-align: left;">Using pjsip library with python</h2><div><br /></div><div>It's possible to use the <i>pjsip</i> library's API with high level programming languages like python. This makes test automation quite versatile, and I remember seeing this approach as early as 2012, where the project I was working on had the client applications built on top of pjsip: it was extremely valuable to simulate programmatically the clients from linux machines.</div><div><br /></div><div>Being designed for interactive applications, <i>pjsip</i> comes with a nice event-based model, so in principle you need to trigger the desired actions and register callback functions that will be called at the proper moment.</div><div><br /></div><div>A complete reference to the python library can be found <a href="https://www.pjsip.org/python/pjsua.htm" rel="nofollow" target="_blank">here</a>.</div><div><br /></div><div>In general, after you import the library:</div><div><br /></div><div><blockquote>import pjsua as pj</blockquote></div><div><br /></div><div>then the library is imported in an object, the configuration objects are populated, and a call is triggered, e.g.:</div><div><br /></div><div></div><blockquote><div> lib = pj.Lib()</div><div> media_cfg = pj.MediaConfig()</div><div> media_cfg.no_vad = 0</div><div> lib.init(log_cfg = pj.LogConfig(level=3, callback=log_cb), media_cfg=media_cfg)</div><div> lib.set_null_snd_dev()</div><div><br /></div><div> lib.set_codec_priority("GSM", 0)</div><div> lib.set_codec_priority("iLBC", 0)</div><div> lib.set_codec_priority("G722", 0)</div><div> lib.set_codec_priority("speex", 0)</div><div> lib.set_codec_priority("pcmu", 0)</div><div> lib.set_codec_priority("pcma", 1)</div><div><br /></div><div><br /></div><div> transport = lib.create_transport(pj.TransportType.UDP)</div><div> lib.start()</div><div> acc = lib.create_account_for_transport(transport)</div><div><br /></div><div> call = acc.make_call(sys.argv[1], MyCallCallback(), hdr_list=custom_headers)</div></blockquote><div></div><div><br /></div><div>You can see that set_codec_priority to 0 is equivalent to the <i>--dis-codec</i> command line option.</div><div><br /></div><div>MyCallCallback() is the callback function that will be invoked at each change of call state, with an event object passed as argument. You'll have something like:</div><div><br /></div><div></div><blockquote><div>class MyCallCallback(pj.CallCallback):</div><div> def __init__(self, call=None):</div><div> pj.CallCallback.__init__(self, call)</div><div><br /></div><div> def on_state(self):</div><div> ...</div><div> if self.call.info().state == pj.CallState.CONFIRMED:</div><div> # The call has been answered</div><div> # Here you can create a player to generate audio into an RTP stream, send DTMF, log information, etc</div><div> # You can even invoke other APIs to interact with more complex systems</div><div>...</div><div> def on_media_state(self):</div><div> global lib</div><div> if self.call.info().media_state == pj.MediaState.ACTIVE:</div><div> ...</div><div> # Media is now flowing, so you can connect it to the internal conference object</div><div> # Connect the call to sound device</div><div> call_slot = self.call.info().conf_slot</div><div> lib.conf_connect(call_slot, 0)</div><div> lib.conf_connect(0, call_slot)</div><div> print "on_media_state - MediaState ACTIVE"</div></blockquote><div></div><div><br /></div><div>As it can be expected, exceptions can be caught and errors displayed:</div><div><br /></div><div></div><blockquote><div>except pj.Error, e:</div><div> print "Exception: " + str(e)</div><div> lib.destroy()</div><div> lib = None</div><div> sys.exit(1)</div></blockquote><div></div><div><br /></div><div>If you happen to need DTMF tones, <i>pjsip</i> offers the <b>dial_dtmf()</b> function, as part of the Call object, e.g.:</div><div><br /></div><div><blockquote>self.call.dial_dtmf("0")</blockquote></div><div><br /></div><div>Just remember that these calls are asynchronous, non-blocking: you need to add explicitly a delay to separate the beginning of a tone from other actions.</div><div><i>pjsip</i> will generate proper RTP Event packets of the given duration, inside the existing RTP stream (and so they will have the same SSRC and proper timestamp reference).</div><div><br /></div><div>I'll write about analysing pcap traces to extract information on RTP events in a separate article.</div><div><br /></div><h2 style="text-align: left;">Wrap up</h2><div><br /></div><div>This article is somehow what I would have wanted to read on the topic some time ago, but I had to infer from various sources and after various experiments. I hope it will be useful to some of the readers.</div><div><br /></div><h2 style="text-align: left;">Appendix - pjsua build and install</h2><div><br /></div><div>To build pjsua on debian you can do something like:</div><div><br /></div><div></div><blockquote><div>apt install python-dev gcc make gcc binutils build-essential libasound2-dev wget</div><div>wget https://github.com/pjsip/pjproject/archive/2.10.tar.gz</div><div>tar -xvf 2.10.tar.gz</div><div>cd pjproject-2.10</div><div>export CFLAGS="$CFLAGS -fPIC"</div><div>./configure && make dep && make</div></blockquote><div></div><div><br /></div><div>The binary will be available at ./<i>pjsip-apps/bin/pjsua-x86_64-unknown-linux-gnu</i>, which of course you can link to something easier to use, or copy to a directory in the PATH.</div><div><br /></div>Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com3tag:blogger.com,1999:blog-9031922773224056133.post-10822976906506521592019-11-22T09:26:00.000+01:002019-11-22T09:30:25.567+01:00kamailio-tests, a testing framework for Kamailio developers<div style="-en-clipboard: true;">
<a href="https://www.kamailio.org/w/" target="_blank">Kamailio</a> is an open source VoIP server, widely used in the VoIP industry for its performance and feature set.</div>
<div style="-en-clipboard: true;">
<br /></div>
<div style="-en-clipboard: true;">
<a href="https://github.com/kamailio/kamailio-tests" target="_blank">kamailio-tests</a> is a project that aims to provide a level of automated testing for developers.
</div>
<div style="-en-clipboard: true;">
<br /></div>
<div>
The main idea is that simple things like loading a module or calling a core or module function can be tested without building an entire infrastructure around Kamailio.
</div>
<div>
<br /></div>
<div>
Also, we want to be able to perform the tests against various versions of Kamailio, and on various OS distributions. In this way a backward incompatible function or a regression can be discovered by a developer even though their system uses one single version and OS combination.
</div>
<div>
<br /></div>
<div>
This includes also third party libraries, e.g. you may think of testing the HTTP clients with different curl libraries, while Kamailio and OS stay at the same version.
</div>
<div>
<br /></div>
<div>
kamailio-test doesn't perform tests at function level (but it requires the application to run) or "integration testing" (because in order to remain generic and compact it doesn't cover complex architectures), but aims to test Kamailio as an isolated application.</div>
<div>
<br /></div>
<div>
In order to achieve this, kamailio-tests relies on Docker to allow developers to experiment and perform their tests in perfect isolation. For example, once you've built the base Docker images you can add and modify tests while travelling on a plane.
</div>
<div>
<br /></div>
<div>
Most VoIP platforms have extensive and complex test scenarios that involve setting up accounts, other components, specific logging systems, etc. We can't provide a compact, generic solution that would replace that, but instead we can focus on self-contained (pun intended), simple tests that give us key feedback on issues (and also the opportunity to debug them in that same infrastructure).
</div>
<div>
<br /></div>
<div>
The internal structure of the project and how to use it is described in its <a href="https://github.com/kamailio/kamailio-tests/blob/master/README.md" target="_blank">README</a>, but here I'd like to reiterate how tests are run and how new tests can be added.</div>
<div>
<br /></div>
<div>
Of course you'll need Docker, and it can be a Desktop or server version.
</div>
<div>
<br /></div>
<div>
Currently CentOS 7, Debian 9 and Debian 10 are supported via dedicated Dockerfiles.
</div>
<div>
<br /></div>
<div>
What version of Kamailio to test can be chosen by simply checking out the desired git branch or tag from <a href="https://github.com/kamailio/kamailio" target="_blank">Kamailio github repo</a>.</div>
<div>
<br /></div>
<div>
The installation process is described in the README, but let's see comment it here too:
</div>
<div>
<br /></div>
<div>
First create a directory where to store the resources and go to it, e.g.:
</div>
<div>
<br /></div>
<blockquote class="tr_bq">
mkdir kamailio-testing<br />
cd kamailio-testing
</blockquote>
<div>
<br /></div>
<div>
Clone the kamailio-tests git repository inside that work directory:
</div>
<div>
<br /></div>
<blockquote class="tr_bq">
git clone <a href="https://github.com/kamailio/kamailio-tests">https://github.com/kamailio/kamailio-tests</a></blockquote>
<div>
<br /></div>
<div>
Clone the kamailio git repository inside that work directory:
</div>
<div>
<br /></div>
<blockquote class="tr_bq">
git clone <a href="https://github.com/kamailio/kamailio">https://github.com/kamailio/kamailio</a></blockquote>
<div>
<br /></div>
<div>
Here you can git checkout the branch or tag you want to test, e.g.</div>
<div>
<br /></div>
<blockquote class="tr_bq">
cd kamailio<br />
git checkout 5.3.1<br />
cd ../</blockquote>
<div>
<br /></div>
<div>
Copy the desired Dockerfile from kamailio-tests in the current folder, based on the OS distribution you want, e.g. for Debian Stretch:
</div>
<div>
<br /></div>
<blockquote class="tr_bq">
cp kamailio-tests/docker/Dockerfile.debian9 Dockerfile</blockquote>
<div>
<br /></div>
<div>
Build the Docker image. Give it the name you want: you'll just have to refer to it when launching the tests. e.g.:
</div>
<div>
<br /></div>
<blockquote class="tr_bq">
docker build -t kamailio-tests-deb9x .</blockquote>
<div>
<br /></div>
<div>
Run the tests. If you're happy with the default behaviour you can just:
</div>
<div>
<br /></div>
<blockquote class="tr_bq">
docker run kamailio-tests-deb9x
</blockquote>
<div>
<br /></div>
<div>
and all not explicitly excluded tests will be executed.
</div>
<div>
<br /></div>
<div>
Some tests may be excluded from execution by listing them in the etc/excludeunits.txt.DISTRIBUTION file, e.g. etc/excludeunits.txt.centos7.
</div>
<div>
<br /></div>
<div>
How to add a test?
</div>
<div>
<br /></div>
<div>
Create a new folder in units/, with a name starting with 't', followed by a string indicating the module, like 'geoip', and four digits, e.g. 'tgeoip0001'.
</div>
<div>
<br /></div>
<div>
Inside this folder the minimum amount of data is a shell script that the test framework will execute and a kamailio.cfg file. See for example <a href="https://github.com/kamailio/kamailio-tests/tree/master/units/tgeoip0001" target="_blank">this test for geoip module</a>.</div>
<div>
<br /></div>
<div>
The script must start with:
</div>
<div>
<br /></div>
<blockquote class="tr_bq">
#!/bin/bash<br />
. ../../etc/config<br />
. ../../libs/utils</blockquote>
<div>
<br /></div>
<div>
to prepare configuration and utility commands.
</div>
<div>
<br /></div>
<div>
Then you can launch Kamailio with the configuration required, e.g.:
</div>
<div>
<br /></div>
<blockquote class="tr_bq">
${KAMBIN} -P ${KAMPID} -w ${KAMRUN} -Y ${KAMRUN} -f ./kamailio-tgeoip0001.cfg -a no -ddd -E 2>&1 | tee /tmp/kamailio-tgeoip0001.log &</blockquote>
<div>
<br /></div>
<div>
You can see that logs are being sent to a local file; this is typically how the test result can be verified. Of course you can use a different approach.</div>
<div>
<br /></div>
<div>
Inside the container you have <a href="https://linux.die.net/man/1/sipsak" target="_blank">sipsak</a> available, so you can launch a test call to trigger call processing in Kamailio with something like:
</div>
<div>
<br /></div>
<blockquote class="tr_bq">
sipsak -s sip:alice@127.0.0.1
</blockquote>
<div>
To help triggering the tests also tools like <a href="http://sipp.sourceforge.net/doc/reference.html" target="_blank">sipp</a> and <a href="https://www.pjsip.org/pjsua.htm" target="_blank">pjsua</a> can be considered, but are not currently installed in the docker images.</div>
<div>
<br /></div>
<div>
After waiting for the necessary time, you can just kill Kamailio with
</div>
<div>
<br /></div>
<blockquote class="tr_bq">
kill_pidfile ${KAMPID}
</blockquote>
<div>
<br /></div>
<div>
and check the test result by parsing the log file, e.g.:
</div>
<div>
<br /></div>
<blockquote class="tr_bq">
grep "ip address is registered in US" /tmp/kamailio-tgeoip0001.log<br />
ret=$?<br />
if [ ! "$ret" -eq 0 ] ; then<br />
exit 1<br />
fi
</blockquote>
<div>
<br /></div>
<div>
'exit 1' will make the test fail.
</div>
<div>
<br /></div>
<div>
End the test script with
</div>
<div>
<br /></div>
<blockquote class="tr_bq">
exit 0</blockquote>
<div>
<br /></div>
<div>
to make it pass.
</div>
<div>
<br /></div>
<div>
Adding a README.md inside that same folder is highly recommended.
</div>
<div>
<br /></div>
<div>
You can add one or more tests and run them before committing the changes.
</div>
<div>
<br /></div>
<!--?xml version="1.0" encoding="UTF-8"?-->
<br />
<div>
Once happy please fork the kamailio-tests repo with your changes and raise a pull request.</div>
<div>
<br /></div>
<div>
Thanks for reading; feedback is welcome.</div>
<div>
<br /></div>
Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com1tag:blogger.com,1999:blog-9031922773224056133.post-14570235229945987682019-11-17T19:42:00.001+01:002019-11-18T13:24:25.818+01:00My notes on Kamailio Developer Meeting - November 2019<div style="-en-clipboard: true;">
The Kamailio Developers Meeting is a two-day event held in Dusseldorf, currently at the second edition.
</div>
<div style="-en-clipboard: true;">
<br /></div>
<div>
As described in https://www.kamailio.org/w/developers-meeting/,
</div>
<blockquote class="tr_bq">
"The purpose of the event is to support the interaction between developers and to offer a great environment to work together on relevant topics related to the Kamailio project. It is intended for participants that want to write code for Kamailio and its tools or improve the documentation. There will be no formal presentations, only open discussions, coding or documentation writing sessions."
</blockquote>
<div>
<br /></div>
<div>
The sipgate offices offered a very welcoming environment. Noticeable to have a kitchen with chefs for breakfast and lunch, and a private pub, where the social event was held. I noticed the presence of art works and learned that those offices are also part of an art itinerary in Dusseldorf.
</div>
<div>
<br /></div>
<div>
We started listing the topics that we wanted to tackle during the event, then discussed a plan to go through them.
</div>
<div>
<br /></div>
<div>
The first important activity was <a href="https://www.kamailio.org/w/2019/11/kamailio-v5-3-1-released/" target="_blank">the release of kamailio version 5.3.1</a> (minor bug fix). The release process includes running the tests in <a href="https://github.com/kamailio/kamailio-tests" target="_blank">kamilio-tests repo</a> (more on this later in a dedicated post), and an issue was found and resolved during the release.</div>
<div>
<br /></div>
<div>
Then it was the time to discuss RPM packages building: Sergey Safarov has made an incredible amount of work to ensure that RPM packages for kamailio are available, and now they are available at <a href="https://rpm.kamailio.org/">https://rpm.kamailio.org/</a>. Work is in progress to move all the building phases to infrastructure belonging to the kamailio projects and avoiding personal accounts to access cloud services as much as possible.
</div>
<div>
<br /></div>
<div>
A documentation review was carried out: it was reported that often the return values of module functions are not documented, and work is encouraged in that direction, including a review of documentation of function parameters. Modules may make available several pseudovariables, and not always they are documented. There's also <a href="https://www.kamailio.org/wiki/cookbooks/devel/pseudovariables" target="_blank">a dedicated wiki page on pseudovariables</a> that's useful as reference.</div>
<div>
<br /></div>
<div>
Daniel made a briefing on the <a href="https://kamailio.org/docs/tutorials/devel/kamailio-kemi-framework/" target="_blank">Kemi framework</a>, explaining how to structure the wrapper functions (two steps, one the parameters manipulation for the standard cfg file, and then encapsulate the code into a separate function. Also in this way executing a function from Kemi doesn't require executing all the typical wrapper manipulations required by cfg functions, impacting positively on performance). Developers need to ensure that when you develop a new function the body of the function is separate from the code that manipulates the parameters.
</div>
<div>
<br /></div>
<div>
An open topic focused on the best process to handle "dialog failover". This may be necessary if a kamailio-based component disappears during the dialog lifetime, or if the architecture allows for in-dialog messages to be processed by different entities during a call, even in the typical case where record-routing applies. Currently it's possible to load dialog information on the fly from a DB, see <a href="https://www.blogger.com/.%20https://www.kamailio.org/docs/modules/stable/modules/dialog.html#dialog.f.dlg_db_load_callid" target="_blank">dlg_db_load_callid()</a>, but when an instance "takes over" the dialog information there's typically the need to manipulate the record-routing, which is complex and not exactly "standard" behaviour.</div>
<div>
<br /></div>
<div>
Modules like <a href="https://www.kamailio.org/docs/modules/devel/modules/evapi.html" target="_blank">evapi</a> and <a href="https://www.kamailio.org/docs/modules/devel/modules/http_async_client.html" target="_blank">http_async_client</a> allow for the management of asynchronous events by spawning dedicated workers which will execute portions of the routing logic when defined events happen. An open discussion is about the best method for managing asynchronous events and at the same time return execution to the main workers when an event is triggered.
</div>
<div>
<br /></div>
<div>
Debian packaging; a PGP signed key was created, to benefit from the physical presence of various holders of PGP signatures. An open point is related to the ability to keep earlier versions of the kamailio packages every time a new version is released. This is a known behaviour that may limit the options in some environments, and I've experienced it directly. This is caused by reprepro and what seems to be the right way to go is moving from reprepro and use aptly instead. We'll update on this.</div>
<div>
<br /></div>
<div>
Use of TLSv1.3: the only limitation appears to be in libssl library. <a href="https://kamailio.org/docs/modules/devel/modules/tls.html#tls.p.tls_method" target="_blank">It's possible to allow the usage of version 1.3, but only by choosing tlsv1.2+</a>, which admits using v1.2, and so doesn't enforce v1.3 only. When this will be allowed, kamailio administrators will just need to update the configuration and reload the tls module. Remember that only configurations in tls.cfg can be reloaded, while modparam declarations require a restart.
</div>
<div>
<br /></div>
<div>
Finally, we know that a common pattern of usage of kamailio involves querying APIs and basing call processing on the API responses. A simple but powerful solution sees the use of http_async_client in collaboration with <a href="https://www.kamailio.org/docs/modules/devel/modules/rtjson.html" target="_blank">rtjson</a>; see for example this article from wazo developers: <a href="https://wazo-platform.org/blog/kamailio-routing-with-rtjson-and-http-async-client">https://wazo-platform.org/blog/kamailio-routing-with-rtjson-and-http-async-client</a> We are discussing whether extending rtjson may be generally helpful for scenarios like this; in that case I'll post an update.
</div>
<div>
<br /></div>
<div>
For me this event was a great experience, both from a friendship point of view and as a learning experience. I'm very happy about how the kamailio project is managed, decisions are taken and information is shared.
</div>
<div>
<br /></div>
<!--?xml version="1.0" encoding="UTF-8"?-->
<br />
<div>
<i>Here's my report; for obvious reasons this can only be a limited account of what happened during the event, and the information provided is based on my recollection and notes. I'm sure I'm omitting other important material, and I'll be happy to integrate with another post. </i></div>
<div>
<br /></div>
Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-41624372573156859642018-11-27T10:17:00.000+01:002018-11-27T10:18:01.515+01:00You need to slow down<div style="-en-clipboard: true;">
<i>This blog has been historically focused on technical topics in areas related to Real Time Communications, but I'm taking the liberty to digress a little.
</i></div>
<div>
<br /></div>
<div>
I've been reading recently about the dynamics of performance in running.</div>
<div>
<br /></div>
<div>
I'm just an amateur runner, and I always train (and race) alone, so I felt the need to make it more interesting than just reading training tables. I've been studying what is it that limits performance. "Train more", unfortunately, not only doesn't always work, but needs to take into account injuries, and overtraining in general.
</div>
<div>
<br /></div>
<div>
One of the first concepts that struck me is that it's been proven that fatigue, and the consequential slow down, does not mean that the body is unable to continue with that effort. What's behind slowing down is a sort of protective mechanism in our nervous system, which wants to prevent the body to reach exhaustion.
</div>
<div>
<br /></div>
<div>
Our nervous system is constantly getting feedback signals from the body, including the perception of adverse weather conditions, and computing for how longer the current effort can be sustained. Even knowing how long is left to run is a form of external feedback.
</div>
<div>
<br /></div>
<div>
When this computation detects that the effort is too high, the nervous system takes control and doesn't fire as many muscle fibres as the athlete wants to. The athlete thinks the muscles can't continue to work at that level, but in reality they are entering a protective status. Without that, people could literally run until body exhaustion or even death.
</div>
<div>
<br /></div>
<div>
I find this fascinating. Evolution has given us a sophisticated algorithm aimed to prevent body exhaustion by generating fatigue symptoms and consequently reduce the actual effort.
</div>
<div>
<br /></div>
<div>
What's also fascinating is that it seems this system can be "tricked". One way of doing is by training. Training is a way of educating your nervous system that a certain effort is OK. "There's no need to shut me down, brain, I know what I'm doing. I did that thousands of times in my training sessions."
</div>
<div>
So while the body adapts to the stress of training, the nervous system too becomes more familiar with that stress, and gives up a little.
</div>
<div>
<br /></div>
<div>
Another way of tricking that system is by providing false external feedback. It seems that if you get false information about elements like the air temperature, or even the remaining time in your training session or race, then the nervous system acts accordingly. If it believes it's not as hot as it is, it will not intervene to shut the muscle fibre activation.
</div>
<div>
<br /></div>
<div>
Similarly, if the athlete thinks the race is close to the end, the nervous system will allow a prolonged effort. This is why elite marathoners, who clearly haven't underperformed for the first 40 km, can run the last 2 km even faster.
</div>
<div>
<br /></div>
<div>
Of course, all these tricks have limits, and the improvements that can be tricked are in the order of small percentages. But still this shows the importance of external feedback.
</div>
<div>
<br /></div>
<div>
This is properly explained by <a href="https://www.amazon.com/Science-Running-limit-maximize-performance/dp/0615942946" target="_blank">Steve Magness in his "Science of running" book</a>. I then read another book from this author, <a href="https://www.amazon.com/Peak-Performance-Elevate-Burnout-Science/dp/162336793X" target="_blank">"Peak performance"</a>. To be perfectly honest, I was expecting something strictly related to endurance sport, but in this second case the concept of performance was wider.
</div>
<div>
<br /></div>
<div>
There seems though to be an analogy between the shutting down of muscle fibres from the nervous system during running, and something that happens behind a desk and is more widely known: mental burnout. From this point of view, mental burnout can be seen as a way of saying "You can't keep going at this (perceived) level of effort. You need to sleep, hydrate, rest, but you keep working. I'm going to take control and shut you down.".
</div>
<div>
<br /></div>
<!--?xml version="1.0" encoding="UTF-8"?-->
<br />
<div>
As I'm making my own little experiments, in the future I'd like to write more about this, and in particular about the relationship between effort and rest. </div>
Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-88299660178772628442018-11-19T21:20:00.002+01:002018-11-19T21:20:53.043+01:00Docker from scratchSome time ago I prepared an introductory seminar on Docker, which I called "Docker from scratch".<br />
<br />
The audience was a local group of heterogeneous developers.
As it typically happens, preparing that material was a great opportunity to understand better some of the aspects.<br />
<br />
I then published the slides in Slideshare. I notice now that they got a decent number of views and downloads, so <a href="https://www.slideshare.net/GiacomoVacca/docker-from-scratch" target="_blank">I'm linking those slides here as well for reference</a>.<br />
<br />
One big change in respect to 2016 is represented by the choice of abandoning Docker Toolbox in favour of running Docker directly on macOS. I have to say I liked the sandboxing that came with Docker Toolbox, where the docker engine ran inside VirtualBox, <a href="https://docs.docker.com/toolbox/toolbox_install_mac/" target="_blank">now only available for older versions</a>.<br />
<br />
<br />Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-7499834831508727452018-05-29T20:50:00.000+02:002018-05-29T20:50:15.452+02:00On Kamailio World 2018, part II<div style="-en-clipboard: true;">
<a href="http://www.giacomovacca.com/2018/05/on-kamailio-world-2018-part-i.html" target="_blank">In the first part of my brain dump about this year's edition of Kamailio World</a> I focused mainly on testing. Core developers and application designers want to be able to test the behaviour of <a href="https://www.kamailio.org/w/" target="_blank">Kamailio</a>-based architectures with minimal effort and fast feedback.
</div>
<div>
<br /></div>
<div>
A different dimension to testing, that I haven't mentioned in my previous post, was related to <a href="https://en.wikipedia.org/wiki/Fuzzing" target="_blank">Fuzz testing</a>. There were two presentations focused on this: Sandro Gauci's (The easiest way to understand who Sandro is: listen on port 5060 on the public Internet and wait a couple of minutes. You'll see a SIP request from a tool called <a href="https://github.com/EnableSecurity/sipvicious" target="_blank">sipvicious</a> (aka friendly-scanner), a penetration testing tool Sandro wrote (and others misuse)) and <a href="https://www.kamailio.org/w/henning-westerholt/" target="_blank">Henning Westerholt</a>, historical member of the Kamailio community.
</div>
<div>
<br /></div>
<div>
Sandro's presentation focused around fuzzing approaches for RTC in general (<a href="https://www.slideshare.net/sandrogauci/a-tale-of-two-rtc-fuzzing-approaches" target="_blank">slides</a>), while Henning was more specifically focused on Kamailio.
</div>
<div>
<br /></div>
<div>
Fuzzing is a sophisticated technique to verify the robustness of a software application, by sending input that can vary greatly from the typical or expected usage. The objective is to find weaknesses that can lead to crashes or other malfunctions, so that they can be fixed. Of course testing a server like Kamailio is even trickier than testing an application that can read from a file. It is a fascinating topic.</div>
<div>
<br /></div>
<div>
Kamailio proved to be very robust: Henning reported an average of about 1 message every 44 million required to make Kamailio misbehave. The video of Henning's presentation is <a href="https://www.youtube.com/watch?v=bhy7-uxZGqk" target="_blank">here</a> (by the way, <a href="https://www.pascom.net/en/mobydick-voip/" target="_blank">Pascom</a> have done a great work this year too, providing a flawless video streaming and recording. It feels like we are a little spoiled, because we give it for granted and barely notice all the work behind it).</div>
<div>
<br /></div>
<div>
In terms of learning opportunities for architects and administrators of Kamailio-based infrastructure, I found very valuable Daniel's presentations around high-level scripting (with <a href="https://kamailio.org/docs/tutorials/devel/kamailio-kemi-framework/" target="_blank">KEMI</a>) to build the routing logic (<a href="https://www.youtube.com/watch?v=cp3TZkcpFUQ" target="_blank">Video</a> and <a href="https://www.kamailio.org/events/2018-KamailioWorld/Day0/W07-Daniel-Constantin.Mierla-KEMI-Scripting.pdf" target="_blank">slides</a>).</div>
<div>
<br /></div>
<div>
Remember that Lua may not be the most popular - apparently - but it's the one estimated to give you performances closest to the native routing language.
</div>
<div>
<br /></div>
<div>
Another valuable presentation was around the Least Cost Routing techniques that the Kamailio environment makes available. (<a href="https://www.youtube.com/watch?v=ZJa0FVrVo7E" target="_blank">Video</a>, and <a href="https://www.kamailio.org/events/2018-KamailioWorld/Day2/28-Daniel-Constantin.Mierla-Kamailio-LCR-Engines.pdf" target="_blank">slides</a>). Some solutions use out of the box modules (like <a href="https://www.kamailio.org/docs/modules/stable/modules/lcr.html" target="_blank">lcr</a>, <a href="https://www.kamailio.org/docs/modules/stable/modules/carrierroute.html" target="_blank">carrierroute</a>, <a href="https://www.kamailio.org/docs/modules/stable/modules/drouting.html" target="_blank">drouting</a>), some are more indirect (<a href="https://www.kamailio.org/docs/modules/stable/modules/pdt.html" target="_blank">pdt</a>, <a href="https://www.kamailio.org/docs/modules/stable/modules/mtree.html" target="_blank">mtree</a>, <a href="https://www.kamailio.org/docs/modules/stable/modules/dialplan.html" target="_blank">dialplan</a>, <a href="https://www.kamailio.org/docs/modules/stable/modules/prefix_route.html" target="_blank">prefix_route</a>), and others are a combination of them. Must-see if you're working in that area.
</div>
<div>
<br /></div>
<div>
Another learning goldmine has been Lorenzo Miniero's (author of <a href="https://janus.conf.meetecho.com/" target="_blank">Janus</a>, a WebRTC conferencing framework (this definition is mine)) lecture about Privacy, Security and Authentication for WebRTC. (<a href="https://www.youtube.com/watch?v=ewNJMci62rs" target="_blank">Video</a> and <a href="https://www.slideshare.net/LorenzoMiniero/webrtc-securitymore-kamailioworld-2018" target="_blank">slides</a>) Lorenzo does talk fast, but no word is spoken in vain. Worst case, you can watch the video at 0.5 speed (smile). Interesting the case of double encryption for media.</div>
<div>
<br /></div>
<div>
<br /></div>
<!--?xml version="1.0" encoding="UTF-8"?-->
<br />
<div>
I guess there's enough for a part III in the near future! To be continued. </div>
Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-15442476210232930752018-05-24T10:20:00.000+02:002018-05-24T10:40:40.053+02:00On Kamailio World 2018, part I<div style="-en-clipboard: true;">
This was my fifth time in a row attending <a href="https://www.kamailioworld.com/k06/" target="_blank">Kamailio World </a>in Berlin. The weather was warmer and sunnier than usual.
</div>
<div style="-en-clipboard: true;">
<br /></div>
<div>
Apart from the obvious focus on <a href="https://www.kamailio.org/w/" target="_blank">Kamailio</a>, as usual the RTC ecosystem was well represented (with Janus, Asterisk, FreeSWITCH, Homer, RTPEngine, and many others).
</div>
<div>
<br /></div>
<div>
Attendance from the other side of the Atlantic Ocean gave stronger emphasis to the "World" term in the title.
</div>
<div>
<br /></div>
<div>
My personal mission this year was to talk about <a href="https://github.com/kamailio/kamailio-tests" target="_blank">a framework for testing Kamailio</a> as a tool for developers and maintainers of the project: <b>kamailio-tests</b>. The main concept was that early tests that are not focused on a specific business logic (as we all have in our projects) and can be automated will be beneficial to Kamailio's reliability. We want to defer end-to-end testing to later stages, because they are expensive.
</div>
<div>
<br /></div>
<div>
To provide a uniform infrastructure where to run the tests, without requiring permanent test environments, we use Docker for this. This is, of course, not the only possible approach, e.g. you could dynamically spawn VMs, AWS EC2s, etc. But Docker can run on your laptop as well as on a full-fledged CI environment, and this makes it easier to use for the developers.</div>
<div>
<br /></div>
<div>
<a href="https://www.slideshare.net/GiacomoVacca/kamailio-world-workshop-kamailiotests" target="_blank">Please take a look at the slides for more details</a>. The feedback has been great so far, and this proved various points:</div>
<div>
<br /></div>
<div>
1. Conferences for developers are not paid holidays for IT guys, but opportunities for knowledge sharing and collaboration (I would say, in particular if Open Source is in the equation).</div>
<div>
<br /></div>
<div>
2. "Functional" or "component" testing is needed by many, but we haven't a mature solution yet.</div>
<div>
<br /></div>
<div>
3. Docker in RTC is less a fancy technology borrowed by other IT areas and more an everyday tool.
</div>
<div>
<br /></div>
<div>
Some have already volunteered to help me improve kamailio-tests, and their point of view will be very useful. More on this project in the future.</div>
<div>
<br /></div>
<div>
Around the topic of testing, in this case not Kamailio itself but more the business logic built around it, there have been interesting insights from Sebastian Damm (sipgate) and Alex Sosic (evosip). </div>
<div>
<br /></div>
<div>
<a href="https://www.slideshare.net/SebastianDamm2/kamailioworld-2018-modular-and-test-driven-sip-routing-with-lua" target="_blank">Sebastian presented an approach</a> that benefits from moving the Kamailio routing logic from the native language to <a href="https://kamailio.org/docs/tutorials/devel/kamailio-kemi-framework/" target="_blank">KEMI</a> with Lua (<a href="https://github.com/sipgate/lua-kamailio">https://github.com/sipgate/lua-kamailio</a>). <a href="https://www.slideshare.net/sosic/cicd-and-tdd-in-deploying-kamailio" target="_blank">Alex presented a way to</a> verify the routing logic is going through the expected paths, again with Docker, and sipp.</div>
<div>
<br /></div>
<div>
KEMI is an extension of Kamailio that allows developers to write the routing logic in high level languages, like Lua, Python, JS and others. Anedoctical experience made me think Lua was the most popular, while apparently Python is. For what concerns Lua in the RTC world, I wrote a few notes in February: <a href="http://www.giacomovacca.com/2018/02/the-interesting-case-of-lua-in-rtc-world.html">http://www.giacomovacca.com/2018/02/the-interesting-case-of-lua-in-rtc-world.html</a>
</div>
<div>
<br /></div>
<!--?xml version="1.0" encoding="UTF-8"?-->
<br />
<div>
The advantages of working with a high level language are obvious: easier to read and maintain, it's easier to test the functions in isolation, and also easier to involve developers without specific knowledge in Kamailio's routing logic script. They will still need to understand how Kamailio works though, and the underlying protocols, so unless you're doing something extremely basic, it's not a complete abstraction from how Kamailio manages its role as "programmable SIP Proxy".</div>
<div>
<br /></div>
<div>
I have tons of notes from Kamailio World, but if I wait to go through all of them before writing something here, there will be the 2019 edition to talk about. So here's at least a part I.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-63883840575394675242018-02-05T21:44:00.000+01:002018-02-05T22:21:10.650+01:00The interesting case of Lua in RTC world<div>
An interesting pattern that caught my attention is the role that <a href="https://www.lua.org/about.html" target="_blank">Lua</a> is gaining in the RTC (Real-Time Communications) world.</div>
<div>
<br /></div>
<div>
Lua is a small-footprint programming language, powerful while keeping a simple syntax.</div>
<div>
<br /></div>
<div>
I’ve been using <a href="https://freeswitch.org/confluence/display/FREESWITCH/Lua+API+Reference" target="_blank">Lua to script dialplan actions for FreeSWITCH</a> since about 2014. It has provided me with a way to define relatively complex logic and speed up the definition of FS’ behaviour.</div>
<div>
<br /></div>
<div>
Delegating this type of logic to a scripting language had several advantages, such as:</div>
<div>
<ul>
<li>It’s easier to read and understand than native dialplans or native routing logic.</li>
<li>Makes unit testing of the dialplan possible/easier.</li>
<li>Allows changing some pieces of logic easily, in many cases preventing expensive reload of modules or restart of applications.</li>
</ul>
<div>
<br /></div>
</div>
<div>
I’ve been using <a href="https://www.kamailio.org/docs/modules/stable/modules/app_lua.html" target="_blank">Lua for Kamailio</a> as well. <a href="https://www.kamailio.org/w/" target="_blank">Kamailio is an open source programmable SIP Proxy</a>. In a specific case, some bits of the routing logic required regex processing and was expecting to change often: an ideal case for an external script to do that work.</div>
<div>
When the logic changes, it’s sufficient to instruct Kamailio to reload the script, and from that moment on the new requests being processed will use the new logic.</div>
<div>
<br /></div>
<div>
Recent versions of Kamailio though add a framework called <a href="https://kamailio.org/docs/tutorials/devel/kamailio-kemi-framework/" target="_blank">KEMI</a>. This opens up new possibilities, and also provides support for many other scripting languages, python being the most popular, JS, <a href="http://www.squirrel-lang.org/" target="_blank">Squirrel</a>. Still, Lua appears to provide the fastest implementations (with no observable performance degradation) while others have limitations. Python, as you can imagine, provides a rich set of functions and libraries, but it’s not as performant and the reload mechanism currently has some issues.</div>
<div>
<br /></div>
<div>
Wireshark, a tool to capture and analyse network activity, <a href="https://wiki.wireshark.org/Lua" target="_blank">exposes a useful API for Lua</a>. You can use the API to define your own Wireshark dissector (which you’ll need to install as a plugin). This has performance limitations in comparison with dissectors written in C - and so it’s recommended for prototyping only - but still can solve your problem perfectly. Out of need, <a href="https://github.com/sipcapture/hep-wireshark" target="_blank">I wrote a Wireshark dissector for HEP</a>, a binary protocol used in the Homer environment. <a href="http://www.sipcapture.org/" target="_blank">Homer is an open source framework</a> for the monitoring and analysis of Real-Time Communications.</div>
<div>
<br /></div>
<div>
Last weekend a new interesting case was <a href="https://fosdem.org/2018/schedule/event/janus/" target="_blank">presented by Lorenzo Miniero at FOSDEM</a>. The target application was <a href="https://janus.conf.meetecho.com/" target="_blank">Janus, an open source framework to build WebRTC gateways</a>.</div>
<div>
<br /></div>
<div>
Janus allows to build applications by defining the transport and business logic as plugins, on top of the core that implements the WebRTC stack.</div>
<div>
It’s written in C and so far users needed to write plugins with that language. The Janus developers have introduced the possibility to write plugins in Lua.</div>
<!--?xml version="1.0" encoding="UTF-8"?-->
<br />
<div>
<a href="http://www.meetecho.com/blog/tutorial-writing-a-janus-video-call-plugin-in-lua/" target="_blank">In his presentation Lorenzo explains also in detail</a> what approach is best to use for a Real-Time application to interact with single threaded language like Lua in an asynchronous context.</div>
<div>
<br /></div>
<div>
<!--?xml version="1.0" encoding="UTF-8"?--> Just a funny note: Lua uses a double dash for commenting out a line: '--'. Be careful when you watch diffs in a terminal because a removed comment will start with '- --‘ and may not the easiest thing to interpret (experiences may vary depending on the terminal, of course!).</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com1tag:blogger.com,1999:blog-9031922773224056133.post-27816253972122282622018-01-21T20:10:00.001+01:002018-01-21T20:10:32.668+01:00Cache busting when building Docker imagesOne of the handiest features of the docker build system is the caching system.<br />
<br />
'docker build' tries to reuse the layers already built until something changes inside the Dockerfile. In this way, we can save several minutes when rebuilding an image if the changes happen further down the list in the Dockerfile.<br />
<br />
Sometimes, though, we do want to invalidate the cache and ensure the next build won't use it.<br />
<br />
To do this an option is to pass the '--no-cache' argument to 'docker build'.<br />
<br />
When dealing with 'apt-get install' instructions though there are other tricks. I found <a href="https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#run">this document on Dockerfile best practices</a> very useful.<br />
<br />
First of all an observation. If you have 'RUN apt-get update' as a single line of a Dockerfile, followed by the installation of a package, e.g.:<br />
<br />
<i>RUN apt-get update</i><br />
<i>RUN apt-get install -y nginx</i><br />
<i><br /></i>
then changing the list of packages and running again the build command won't trigger an 'apt-get update': that line hasn't changed so docker build reuses the cache. It might not be what you want.<br />
<br />
To force cache invalidation for this specific case the recommendation is to use those commands in the form:<br />
<i><br /></i>
<i>RUN apt-get update && apt-get install -y nginx</i><br />
<br />
This will always install the latest version of the packages. It even has a name: "cache busting".<br />
<br />
Another recommendation I like is to put each package on a single line, and have them in alphabetical order: this will ease visual inspection and prevent duplicates or other undesired conditions.<br />
<br />
Of course, you can also specify exact versions for the packages as you would normally do with 'apt-get install'. That's "version pinning" and it invalidates the cache too.<br />
<br />
You can find all this on the linked page on Dockerfile best practices; this is just my digested interpretation.<br />
<br />
Just one more thing: a way to limit the size of a built image is to clean up the content of '/var/lib/apt/lists' in the same RUN command, e.g.:<br />
<div>
<i><br /></i></div>
<div>
<i>RUN apt-get update && apt-get install -y \</i></div>
<div>
<i> aufs-tools \</i></div>
<div>
<i> automake \</i></div>
<div>
<i> build-essential \</i></div>
<div>
<i>&& rm -rf /var/lib/apt/lists/*</i> </div>
<div>
<br /></div>
<div>
The command above will build an image layer that doesn't contain the apt cache.</div>
<div>
<br /></div>
<div>
If you had instead used this:</div>
<div>
<div>
<i style="background-color: white;"><br /></i></div>
<div>
<i style="background-color: white;">RUN apt-get update && apt-get install -y \</i></div>
<div>
<i style="background-color: white;"> aufs-tools \</i></div>
<div>
<i style="background-color: white;"> automake \</i></div>
<div>
<i style="background-color: white;"> build-essential</i></div>
<div>
<i style="background-color: white;">RUN rm -rf /var/lib/apt/lists/*</i></div>
</div>
<div>
<span style="background-color: white;"><br /></span></div>
<div>
<span style="background-color: white;">you would have had not only a larger layer, containing the apt cache, but also an additional layer generated by the second RUN command.</span></div>
<div>
<span style="background-color: white;"><br /></span></div>
<div>
<br /></div>
Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-29811877705114598662018-01-13T11:56:00.001+01:002018-07-06T14:03:36.037+02:00SIP - ACK loose routingIf you've ever worked with SIP, you must have stumbled upon a trace with 200 OK to INVITE being retransmitted for about 30" and then the call just being set up fail.<br />
<br />
The ACK was never received.<br />
<br />
Then comes the interesting part: discovering why.<br />
<br />
Here are some notes about what should happen, in particular when there are multiple proxies along the path, and with a little additional complexity of one of the proxies with two network interfaces. All this assuming loose routing everywhere. The main reference here of course is <a href="https://tools.ietf.org/html/rfc3261" target="_blank">RFC 3261</a>.<br />
<br />
Isn't an image worth a thousand words? Then here's a sequence diagram:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4a_no538ZxALs72Lfb1ASY2hyym43Anx7cCtKE0ayd6rQVRzr3NpZL6g2iga0IBSVVHSVhGE3C1zpNM7TVXJ6lSG8-7OjsjKmveLBghfumJ6ugwmgXrHYp4JjFBbBuS6zK-4MIs1ICtA/s1600/ACK_routing_multiple_ifs.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="828" data-original-width="1007" height="327" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4a_no538ZxALs72Lfb1ASY2hyym43Anx7cCtKE0ayd6rQVRzr3NpZL6g2iga0IBSVVHSVhGE3C1zpNM7TVXJ6lSG8-7OjsjKmveLBghfumJ6ugwmgXrHYp4JjFBbBuS6zK-4MIs1ICtA/s400/ACK_routing_multiple_ifs.png" width="400" /></a></div>
<br />
All Record-Route headers are assumed to carry loose routing URIs (they have the ;lr attribute).<br />
<br />
B, C and D, working as proxies that want to stay in the path, record route themselves. For this reason E, the UAS and "callee", receives an INVITE with a list of Record-Route headers with B, C and D.<br />
In particular for B there will be two Record-Route headers, since B is using two separate interfaces, one facing A and the other facing C.<br />
<br />
In typical cases the two interfaces represent the interaction with the public Internet on a side and a private infrastructure on the other. But it's not important for this discussion.<br />
<br />
Omitting provisional responses for simplicity, let's assume E responds immediately with a 200 OK. This response will have the same list of R-R headers, in the same order, as received by E.<br />
E will also add its URI in the Contact header of the 200 OK.<br />
<br />
In this loose routing context the IP address in E's Contact's URI will be relevant only for D in the future.<br />
<br />
D, C and B don't modify the list of Record-Route headers, and A receives it as sent by E.<br />
<br />
Apart from the operations related to the media session set up, A will send the ACK to the 200 OK.<br />
This ACK will have a Request URI with E's Contact URI (stripped of anything that can't se inside a Request URI), and a Route header list which is basically the received Record-Route header list inverted (see images).<br />
<br />
A is saying: "Route this ACK to E, routing it via this list of hops".<br />
<br />
When B receives that ACK, it must recognise that the topmost Route headers are B itself, remove them from the Route list, and pick b2 as the interface to deliver the ACK to C.<br />
<br />
C and D will have an easier task to remove a single Route header, the one representing them, and deliver the ACK to the next route.<br />
<br />
For D, the next route will be in fact E, and the ACK will be routed using only the Request URI, as the Route headers have all been eliminated. This is the only step where the IP address that E has set in the Contact of its 200 OK response needs to be visible by another entity, namely D.<br />
<br />
<h2>
ACK routing as explained in RFC 3665</h2>
<br />
To further reiterate this concept, let's look at a somewhat simpler example in <a href="https://tools.ietf.org/html/rfc3665#section-3.2" target="_blank">RFC 3665</a>.<br />
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">The ACK part is:</span><br />
<br />
<script src="https://gist.github.com/giavac/11f9cbfda014595a7bd9bd03d3c52960.js"></script>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;">You can see there's no requirement for Proxy 1 to be able to reach the UAC contact (client.biloxi.example.com might be completely unreachable from Proxy 1).</span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;">It's Proxy 2's responsibility to route the ACK in the last hop towards Bob.</span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;">Proxy 1 must leave the R-URI as is (see below for more details on proxy behaviour), strip itself from the list of Routes and route the ACK to the new topmost Route (Proxy 2).</span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;">Proxy 2 will strip itself from the Route list, being the topmost Route, and forward the ACK to Bob. There are no more Route headers.</span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;">Only at the last hop Bob's contact reachability is relevant, and it is for Proxy 2 only.</span></span></div>
<br />
<h2>
More about the behaviour of the proxies to corroborate the ACK routing</h2>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">From RFC 3261, 16.4:</span></div>
<span id="docs-internal-guid-53052e44-ef21-461a-dff0-cb8aa6158ba8"><br /><span style="font-family: "arial"; font-size: 10pt; vertical-align: baseline; white-space: pre-wrap;">“ If the first value in the Route header field indicates this proxy,</span><span style="font-family: "arial"; font-size: 10pt; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span><span style="font-family: "arial"; font-size: 10pt; vertical-align: baseline; white-space: pre-wrap;"> the proxy MUST remove that value from the request.”</span></span><br />
<span style="font-family: "arial"; font-size: 10pt; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="font-family: "arial";"><span style="font-size: 13.3333px; white-space: pre-wrap;">From RFC 3261, 16.5:</span></span><br />
<span style="font-family: "arial";"><span style="font-size: 13.3333px; white-space: pre-wrap;"><br /></span></span>
<span style="font-family: "arial";"><span style="font-size: 13.3333px; white-space: pre-wrap;">“ A proxy can only change the Request-URI of a request during</span></span><br />
<span style="font-family: "arial";"><span style="font-size: 13.3333px; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<span style="font-family: "arial";"><span style="font-size: 13.3333px; white-space: pre-wrap;"> forwarding if it is responsible for that URI.”</span></span><br />
<span style="font-family: "arial";"><span style="font-size: 13.3333px; white-space: pre-wrap;"><br /></span></span>
<br />
<h2>
APPENDIX - Why is the ACK to 200 OK to INVITE a separate transaction?</h2>
<br />
From RFC 3261, ch. 17:<br />
<br />
The reason for this separation is rooted in the importance of<br />
delivering all 200 (OK) responses to an INVITE to the UAC. To<br />
deliver them all to the UAC, the UAS alone takes responsibility<br />
for retransmitting them (see Section 13.3.1.4), and the UAC alone<br />
takes responsibility for acknowledging them with ACK (see Section<br />
13.2.2.4). Since this ACK is retransmitted only by the UAC, it is<br />
effectively considered its own transaction.<br />
<br />
<br />
<br />
<br />
<br />
<br />Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-81441976968957983712018-01-09T11:29:00.001+01:002018-01-09T11:29:53.246+01:00Copying a file from a Docker container to the hostOften things are more convoluted than you expect. Sometimes they are easier than you fear.<br />
<br />
Here's an example: you're generating files inside a Docker container (with no volumes configured) and you realise you need some of those files available in the host.<br />
<br />
A simple solution is to use 'docker cp', with a format like<br />
<br />
<i>docker cp CONTAINER_ID:FILE_PATH HOST_FILE_PATH</i><br />
<br />
<a href="https://docs.docker.com/engine/reference/commandline/cp/" target="_blank">Official documentation</a>.<br />
<a href="https://stackoverflow.com/questions/22049212/copying-files-from-docker-container-to-host" target="_blank">A Stackoverflow question with more discussions</a>.<br />
<br />Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-35554125942316221962017-11-30T12:05:00.000+01:002017-11-30T12:42:13.295+01:00Around sipp with pcap and authentication supportThis is just a practical guide to have latest <a href="http://sipp.sourceforge.net/doc/reference.html" target="_blank">sipp</a> on <a href="https://www.debian.org/" target="_blank">debian</a>, to benefit from features that are not available in the stock package (<a href="https://packages.debian.org/jessie/sip-tester" target="_blank">sip-tester</a> - version 3.2.1). For example support for playing pcap files or computing authentication hashes.<br />
<br />
<h2>
Build sipp with pcap support</h2>
<br />
<i>apt-get install autoconf libncurses5-dev libpcap-dev g++</i><br />
<i>cd /usr/local/src/</i><br />
<i>git clone https://github.com/SIPp/sipp.git</i><br />
<i>cd sipp/</i><br />
<i>./build.sh --with-pcap</i><br />
<br />
./sipp is the built binary. You can see the version and capabilities with './sipp -v', e.g.:<br />
<i><br /></i>
<i>$ ./sipp -v</i><br />
<i>SIPp v3.6-dev-149-gb95f98f-PCAP-RTPSTREAM.</i><br />
<i>...</i><br />
<br />
This version will be able to use actions like<br />
<br />
<i>exec rtp_stream="file.wav"</i><br />
<br />
or<br />
<br />
<i>exec play_pcap_audio="pcap/g711a.pcap"</i><br />
<br />
(see details in <a href="http://sipp.sourceforge.net/doc/reference.html" target="_blank">current documentation</a>).<br />
<br />
<b>A little caveat for 'rtp_stream' and WAV files</b>. As the documentation says sipp expects a WAV file encoded with PCAM ('A-law'). You can also loop that audio for as long as you need, by adding '<b>,-1</b>' after the file name (and within the double quotes).<br />
But sipp will also send the WAV header in the RTP payload. If this is not acceptable for you, then you can strip the WAV header with something like:<br />
<br />
<i>sox -t WAV -r 8000 -c 1 -e a-law audio.wav audio.raw</i><br />
<br />
(where sox is <a href="https://en.wikipedia.org/wiki/SoX" target="_blank">this tool</a>).<br />
<br />
play_pcap_audio is useful when you want full control of the RTP produced by sipp, e.g. the exact RTP SSRC, missing packets, Mark set bits, etc. Very powerful.<br />
<br />
<h2>
Add authentication support</h2>
<br />
Same dependencies as above, but add:<br />
<div>
<i>apt-get install libssl-dev
</i></div>
<div>
<i><br /></i></div>
<!--?xml version="1.0" encoding="UTF-8"?-->
<br />
<div>
<i>./build.sh --with-pcap --with-openssl</i></div>
<div>
<br /></div>
<div>
./sipp is the built binary. You can see the version and capabilities with './sipp -v', e.g.:<br />
<br />
<i>$ ./sipp -v</i><br />
<!--?xml version="1.0" encoding="UTF-8"?--><i>
SIPp v3.6-dev-149-gb95f98f-TLS-PCAP-RTPSTREAM.</i><br />
<i>...</i></div>
<div>
<br /></div>
<div>
See the authentication features in <a href="http://sipp.sourceforge.net/doc/reference.html" target="_blank">current documentation</a>.</div>
<div>
<br /></div>
<div>
More to come around these topics soon.</div>
<div>
<br /></div>
Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-35224159908711828392017-05-07T21:13:00.000+02:002017-05-07T21:13:20.093+02:00Monitoring FreeSWITCH with Homer - adding non-SIP events with hepipe.js<div>
FreeSWITCH (from now on FS) provides a very powerful tool to interact with it: the Event Socket (ESL), made available via the mod_event_socket module (<a href="https://freeswitch.org/confluence/display/FREESWITCH/mod_event_socket">https://freeswitch.org/confluence/display/FREESWITCH/mod_event_socket</a>).
</div>
<div>
<br /></div>
<div>
ESL is a TCP socket where applications can connect to, and perform two types of action:
</div>
<div>
1. Send commands.
</div>
<div>
2. Subscribe to events.
</div>
<div>
<br /></div>
<div>
The applications subscribing to events will receive the expected notifications through the same TCP connection.
</div>
<div>
A simple protocol and transport made it possible for various libraries in various languages to be written.
</div>
<div>
<br /></div>
<div>
Events from FS can serve multiple purposes. In this article I'm interested in monitoring and event correlation.
</div>
<div>
<br /></div>
<div>
Homer (<a href="http://sipcapture.org/">http://sipcapture.org/</a>) is a widely used, open source tool to monitor RTC infrastructures. It has a multitude of features, but the core is the ability to collect SIP signalling and other events from RTC applications, and perform a form of correlation. In particular, it's able to correlate the SIP signalling involved in a call with other events like RTCP reports or log lines associated to the same call.
</div>
<div>
<br /></div>
<div>
While FS, through the sofia module, has native support for transmitting SIP signalling to Homer, the acquisition of other events can happen by collecting these events from the ESL, filtering them, and sending them to Homer with the proper formatting.
</div>
<div>
<br /></div>
<div>
This is what hepipe.js (<a href="https://github.com/sipcapture/hepipe.js">https://github.com/sipcapture/hepipe.js</a>) does. hepipe.js is a simple nodejs application that is able to:
</div>
<div>
- connect to FS via ESL
</div>
<div>
- subscribe to specific event categories
</div>
<div>
- format the events into HEP messages. HEP is a binary protocol used to transmit data to Homer.
</div>
<div>
<br /></div>
<div>
hepipe.js is easy to use:
</div>
<div>
- Clone it
</div>
<div>
- Run 'sudo npm install' to install the required dependencies
</div>
<div>
- Set configuration
</div>
<div>
- Run it ('sudo node hepipe.js', or 'sudo nodejs hepipe.js')
</div>
<div>
<br /></div>
<div>
The configuration is organized in "modules", and for this example you'll have to configure at a minimum the esl module and the hep module.
</div>
<div>
Edit a <i>config.js</i> file in the same folder as hepipe.js with something like:
</div>
<div>
<br /></div>
<div>
<span style="font-style: italic;">var config = {</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> hep_config: {</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> debug: true,</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> HEP_SERVER: '10.0.0.17',</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> HEP_PORT: 9060</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> },</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> esl_config: {</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> debug: true,</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> ESL_SERVER: '127.0.0.1',</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> ESL_PORT: 8021,</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> ESL_PASS: 'ClueCon',</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> HEP_PASS: 'multipass',</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> HEP_ID: 2222,</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> report_call_events: true,</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> report_rtcp_events: true,</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> report_qos_events: true</span></div>
<div style="text-align: start;">
<span style="font-style: italic;"> }</span></div>
<div style="text-align: start;">
<span style="font-style: italic;">};</span></div>
<div style="text-align: start;">
<br /></div>
<div>
<span style="font-style: italic;">module.exports = config;</span></div>
<div>
<br /></div>
<div>
This will configure the hep module to send data to a Homer instance listening on UDP, IP address 10.0.0.17, port 9060, and will try to connect to a FS' ESL on localhost, via TCP port 8021 and using the default password. See also other configuration examples in the examples/ folder.</div>
<div>
<br /></div>
<div>
Please note that the ESL requires at least two levels of authorization: a password and an ACL. You can check conf/autoload_config/event_socket.conf.xml in the FS configuration folder to ensure the ACL in use, if any, is compatible to the source IP address of hepipe.js when connecting to FS.
</div>
<div>
e.g.:
</div>
<div>
<i><param name="apply-inbound-acl" value="10.0.0.15/32" />
</i></div>
<div>
or
</div>
<div>
<i><param name="apply-inbound-acl" value="lan" />
</i></div>
<div>
<br /></div>
<div>
Once <i>config.js </i>will be ready, launch hepipe.js and look at the events being sent to Homer.
</div>
<div>
Note that you can filter out event types by setting to false some of these:
</div>
<div>
report_call_events: true,
</div>
<div style="text-align: start;">
report_rtcp_events: true,
</div>
<div>
report_qos_events: true
</div>
<div>
<br /></div>
<div>
Assuming FS is configured to send SIP signalling to the same Homer instance, you'll be able to see, associated to its SIP call flows, also the events captured by hepipe.js.
</div>
<div>
<br /></div>
<div>
See for example below log lines created by FS, sent to Homer, and then presented together with the SIP signalling in Homer:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSWzLY9pPrstmMUUCEGZbRj8gWLZ03O_3SG7zt39sxABdl6oaFbdNRh0Z1dpdmuS7VoYi9Vt13_7S3pixkuqtA1omBfDnoACfCiYm9GHL5KelcDYLplvva91Yvf9oH1XdyutyAkMvIFmc/s1600/homer_workshop_screenshot_fs_logs.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="202" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSWzLY9pPrstmMUUCEGZbRj8gWLZ03O_3SG7zt39sxABdl6oaFbdNRh0Z1dpdmuS7VoYi9Vt13_7S3pixkuqtA1omBfDnoACfCiYm9GHL5KelcDYLplvva91Yvf9oH1XdyutyAkMvIFmc/s400/homer_workshop_screenshot_fs_logs.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Enjoy!
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<!--?xml version="1.0" encoding="UTF-8"?-->
<br />
<div>
<br /></div>
Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-44423346652826063012017-01-15T21:19:00.001+01:002017-05-17T22:46:57.185+02:00Analysing Opus media from network tracesVoIP/RTC platforms have typically many elements processing audio. When an issue is reported it's important to be able to restrict the investigation field, to save time and resources.<br />
<br />
A typical scenario is bad or missing audio perceived on the client side. As I've done previously (<a href="http://www.giacomovacca.com/2016/02/extracting-opus-from-pcap-file-into.html" target="_blank">here for Opus</a> and <a href="http://www.giacomovacca.com/2017/01/voip-calls-encoded-with-silk-from-rtp.html" target="_blank">here for SILK</a>) I'd like to share some practical strategies to extract audio from a pcap trace (to verify the audio received/sent was "correct") and to "re-play" the call inside a test bed (to verify that the audio was good but also carried correctly by the RTP stream). Of course a lot can be inferred by indirect data, for example the summary of RTCP reports showing the number of packets exchanged, packets lost, the latency. But sometimes those metrics are perfect while the issue is still there.<br />
<br />
Focusing in this case on Opus audio, and starting from a pcap file with the network traces for a call under investigation, let's see how to decode the Opus frames carried by the RTP packets into an audible WAV file.<br />
<br />
You don't even need to have captured the signalling: it's sufficient to have the UDP packets carrying the RTP. If signalling is not visible by Wireshark it may not recognize that the UDP packets carry RTP, but you give it a hint by right-clicking on a frame and "Decode as..." and selecting "RTP".<br />
<br />
It's typically easy to find the relevant RTP stream in Wireshark ("Telephony -> RTP -> RTP Streams"), select it, and prepare a filter. Then you can Export the packets belonging to that stream into a dedicated pcap file ("File --> Export Specified Packets...").<br />
<br />
I've then modified <b>opusrtp</b> from <a href="https://github.com/giavac/opus-tools" target="_blank">a fork</a> of <a href="https://github.com/xiph/opus-tools" target="_blank">opus-tools</a> in order to be able to extract the payload from a given pcap, creating an Opus file. e.g.:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">./opusrtp --extract trace.pcap</span><br />
<br />
This will output a rtpdump.opus file, which can be converted into a WAV file directly with <b>opusdec</b>, still part of opus-tools:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">./opusdec --rate 8000 rtpdump.opus audio.wav</span><br />
<br />
You can listen to the wav file and verify whether at least the carried RTP payload was valid.<br />
<br />
The network trace with the RTP can also be used to re-play the call, injecting the same RTP as in the call under investigation. With the help of <a href="http://sipp.sourceforge.net/doc/reference.html" target="_blank">sipp</a> you can set up a rudimentary but very powerful test bed. Use the standard UAS scenario (e.g. in <a href="http://sipp.sourceforge.net/doc/uas.xml">uas.xml</a>), but with an additional part:<br />
<script src="https://gist.github.com/giavac/3fd179ca579f695d8cf94b9ee82b58df.js"></script>
<br />
right after the ACK is received. If you launch sipp with a command like:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">sipp -sf uas.xml -i MEDIA_IP_ADDRESS<media address="" ip=""></media></span><br />
<br />
you'll be able to call sipp. It will answer the call, as the scenario mandates, and will play the RTP contained in <i>rtp_opus.pcap</i>. The stream SSRC, timestamps, even Marker bits will be preserved. This will give you quite an accurate simulation of the stream received by the client in the original call.<br />
<br />
It should be straightforward to reach all these components. For opus-tools, on a debian-based machine, you can just:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">sudo apt-get install libogg-dev libpcap-dev</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">git clone https://github.com/giavac/opus-tools.git</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">cd opus-tools</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">./autogen.sh</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">./configure</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">make</span><br />
<br />
For sipp:<br />
<span style="font-family: "courier new" , "courier" , monospace;">sudo apt-get install sip-tester</span><br />
<br />
I hope this will save the reader some time in future investigations.<br />
<br />
<i>UPDATE: The fork of opus-tools was merged into the <a href="https://github.com/xiph/opus-tools">original repo</a>, so you don't need my repo.</i><br />
<i><br /></i>
<i>UPDATE 2: This only works if the opus payload in the RTP is not encrypted. Also it may need a patch when the extension header for volume indications are used (e.g. '</i><span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">urn:ietf:params:rtp-hdrext:</span><wbr style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;"></wbr><span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px;">ssrc-audio-level</span><i>', see RFC-6464). Don't forget that at the moment the payload type is harcoded to 120. You may need to rebuild opusrtp with the type your trace has, e.g. 96 (It should be easy to pass it as command line argument, something for a quiet moment).</i><br />
<br />
<br />
<br />Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com16tag:blogger.com,1999:blog-9031922773224056133.post-57916271276349307372017-01-13T14:55:00.000+01:002017-01-13T15:00:31.748+01:00VoIP calls encoded with SILK: from RTP to WAV, updateThree and a half years ago (which really sounds like a lot of time!) I was working with a VoIP infrastructure using <a href="https://en.wikipedia.org/wiki/SILK" target="_blank">SILK</a>. As it often happens to server-side developers/integrators, you have to prove whether the audio provided by a client or to a client is correctly encoded :-)<br />
<br />
Wireshark is able to decode, and play, G.711 streams, but not SILK (or <a href="http://opus-codec.org/" target="_blank">Opus</a> - more on this later). So I thought of having my own tool handy, to generate a WAV file for a PCAP with RTP carrying SILK frames.<br />
<br />
The first part requires extracting the SILK payload and writing it down into a bistream file. Then you have to decode the audio using the SILK SDK decoder, to get a raw audio file. From there to a WAV file it is very easy.<br />
<br />
<a href="http://www.giacomovacca.com/2013/06/voip-calls-encoded-with-silk-from-rtp.html" target="_blank">As I tried to describe in this previous post</a>, I had to reverse engineer the test files contained in the SDK, to see what a SILK file looked like.<br />
<br />
Since the SILK payload is not constant, all that was needed was to insert 2 Bytes with the length of the following SILK frame. At the beginning of the file you have to add a header containing "#!SILK_V3", and voilà.<br />
<br />
This is accomplished by <b>silk_rtp_to_bistream.c</b> (from <a href="https://github.com/giavac/silk_rtp_to_bitstream">https://github.com/giavac/silk_rtp_to_bitstream</a>), a small program based on <a href="http://www.tcpdump.org/" target="_blank">libpcap</a> that extracts the SILK payload from a PCAP and writes it properly into a bistream file.<br />
<br />
Build the binary with:<br />
<br />
<!--?xml version="1.0" encoding="UTF-8"?-->
<span style="font-family: "courier";">gcc silk_rtp_to_bitstream.c -lpcap -o silk_rtp_to_bitstream</span><br />
<br />
(you'll need libpcap-dev installed)<br />
<br />
Create the bistream with:<br />
<br />
<!--?xml version="1.0" encoding="UTF-8"?-->
<span style="font-family: "courier";">./silk_rtp_to_bitstream input.pcap silk.bit</span><br />
<br />
Now you can decode, using the SILK SDK, from bitstream into raw audio with:<br />
<br />
<!--?xml version="1.0" encoding="UTF-8"?-->
<span style="font-family: "courier";">$SILK_SDK/decoder silk.bit silk.raw</span><br />
<br />
Raw audio to WAV can be done with sox:<br />
<br />
<!--?xml version="1.0" encoding="UTF-8"?-->
<span style="font-family: "courier";">sox -V -t raw -b 16 -e signed-integer -r 24000 silk.raw silk.wav</span><br />
<br />
This works fine with single channel SILK at 8000 Hz.<br />
<br />
<br />
More to come: an update on how to accomplish the same but for Opus.<br />
<br />
<br />Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com1tag:blogger.com,1999:blog-9031922773224056133.post-15384422743200695612016-12-19T22:02:00.002+01:002016-12-19T22:03:36.644+01:00Thinking About Thinking<div>
I've just finished reading a surprising book, and I'd like to share some notes with you. I was initially looking for something to improve the reasoning and logical flow during conversations. I found <i>"Thinking, Fast and Slow"</i>, which is not really about that, but kept going for some pages and then couldn't stop.
</div>
<div>
<br /></div>
<div>
I'm not using any kind of referral promotion, so if any reader of this post will buy the book I won't get a cent. And I'm not expert on psychology, sociology or economy.</div>
<div>
<br /></div>
<div>
It's just that <a href="https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555" target="_blank">"Thinking, Fast and Slow"</a> has been a great reading experience. There are some insights on the way people think and decide that are worth analysing.
</div>
<div>
<br /></div>
<div>
First of all the authors present two separate ways of "thinking": the first is intuitive, voluntary. It derives from millennia of evolution, where as animals we needed to decide in a fraction of second whether a situation was a danger or an opportunity. This is effortless and always active.
</div>
<div>
<br /></div>
<div>
The second system is voluntary thinking. It requires an effort and it can't be always active during the day. This is strongly impacted by the first system: we may believe we are evaluating a problem coldly and rationally, but the intuitive system has already tagged it in a way or another, and energies need to be spent to fight that "first impression".
</div>
<div>
<br /></div>
<div>
This model explains why often we understand something without really thinking about it, or we have a very different point of view on something after some proper analysis. The book is dense of information, and I'll just mention some psychological effects that surprised me particularly.
</div>
<div>
<br /></div>
<div>
<i>Loss aversion</i>: People tent to evaluate losses more than gains. It seems the ratio is 2:1, i.e. losses weight double than gains. For example, to accept the risk of losing 100 euros most people would need to have an equal probability of gaining 200 euros. This is probably why people tend to leave things as they are, and defer changes: only rarely when comparing the current situation with a new one, gains seem double the losses.
</div>
<div>
<br /></div>
<div>
<i>Sunk-cost fallacy</i>: Once somebody has invested (money, time) on something, they tend to evaluate the status in a better way than reality. Also people refrain from abandoning a project or investment if a large cost has already been faced, even though the forecast is not positive.
</div>
<div>
<br /></div>
<div>
<i>Competition neglect</i> (and <i>Planning fallacy</i>): Once involved in an activity, our perception of competing factors tends to fade away. Similarly, when planning a task or project, we tend to be way more optimistic as we should given our own experience. I guess I saw this some times in software development...
</div>
<div>
<br /></div>
<div>
<i>Endowment effect</i> (and <i>Affect heuristic</i>): People associate a higher value to things that they own. This is perhaps why we accept to buy something at a price, but would refuse to sell it for the same or only slightly higher price. Once we own it, there is an emotional value attached to it.
</div>
<div>
<br /></div>
<div>
This book proves that humans are not very good in thinking in terms of probability, even, for example, when they are aware of the actual probability of an event. They'll tend to consider something more or less probable depending on various psychological factors.</div>
<div>
<br /></div>
<div>
Also we often believe that some events are a clear indication of a measurable reality, like the score of a football match, the performance of a company or a stockbroker, and like to ignore statistical fluctuations (which is called "<i>Regression to the mean</i>").
</div>
<div>
<br /></div>
<!--?xml version="1.0" encoding="UTF-8"?-->
<br />
<div>
All these elements, and many more, are in contrast with the ideal economical mind, where values, durations, students, even probability of occurrence are always correctly estimated. People are less rational that they like to believe, and being aware of these psychological factors impacting rationality may be useful in our lives.</div>
Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0tag:blogger.com,1999:blog-9031922773224056133.post-11325788522944979492016-11-30T10:32:00.001+01:002016-11-30T10:32:09.487+01:00Docker networking and a tricky behaviourThere's something about debugging: the more you have experienced finding the root cause of bugs in the past, the highest the hope and confidence you'll squash the one currently under the microscope.<br />
<br />
You see I didn't mention "fixing" a bug, because I think finding the root cause of any bug has value <i>per se</i>. Fixing them is a separate adventure.<br />
<br />
Anyway this week I was entirely bamboozled by this: a Docker container was re-deployed with a different networking configuration. In particular, with Compose, I was dedicating a network for that container (let's say 172.18.0.0/24) separate from the default Docker network (e.g. 172.17.0.0/24).<br />
<br />
In docker-compose.yml:<br />
<br />
<blockquote class="tr_bq">
networks:<br /> apps:<br /> driver: bridge<br /> ipam:<br /> driver: default<br /> config:<br /> - subnet: 172.18.0.0/24<br /> gateway: 172.18.0.1</blockquote>
<br />
<br />
In the "service" definition I just added this:<br />
<br />
<blockquote class="tr_bq">
services:<br /> THESERVICE:<br /> ..<br /> networks:<br /> - apps</blockquote>
<br />
No need to be more specific with names because Compose will add the project name to the network, so in the Docker network list you see something like: MYPROJECT_apps.<br />
<br />
This change worked perfectly on two previous deployments - in fact they were deployments on intermediate stages, in preparation for a deployment on a third stage. Everything went smoothly there, no big deal, Compose updating the networking as the new configuration was brought up.<br />
<br />
This particular container <i>exposes</i> a UDP port (say 8888) and it's expected to receive UDP packets from the VLAN via the host interface. Packets arrive at the host's UDP 8888 and are forwarded to the container's UDP 8888, where they are processed. Very simple.<br />
<br />
But when I did the deployment on the third environment, reasonably similar to the first two, things just didn't work. <i>tcpdump</i> was showing that the UDP packets were still received by the host but not forwarded to the container.<br />
<br />
The forwarding rules - inspected with <b>iptables</b> - were correct. The container had the expected IP Address. <i>ip route</i> was correct. Where are those packets going then?<br />
<br />
I ssh'd into one of the machines that were sending UDP traffic to the receiving host, opened a <b>netcat</b> connections to that host, port 8888, and sent some data. Those packets were correctly being received by the container! So some traffic with similar characteristics was received, some wasn't.<br />
<br />
Perhaps an issue with data length? Nope. I tried sending big UDP packets and they were correctly received too. It was a weak lead anyway, because the source traffic was of various lengths while not a single packet was forwarded.<br />
<br />
Errors logged somewhere? None.<br />
<br />
<i>iptables</i> stats were confirming the arrival of the packets, and that they were not routed to the container's virtual interface.<br />
<br />
At this point I was enough out of ideas to start tapping the shoulder of some colleague. "I'm observing an interesting behaviour!". "Have you tried...": "Yes". "How about... ": "Checked." "WTF?": "Exactly".<br />
<br />
It was indeed a WTF moment. Surely there were some differences in the list of virtual interfaces on the problematic host, but none seemed to have an impact.<br />
<br />
We looked deeper into the network traces. And there, glooming in their "light green on black" magnificence there were the <i>ICMP Destination Unreachable</i> responses from the receiving host to the producing hosts. A confirmation that packets were attempted to be sent to an unreachable destination. Why? The forwarding rules were not just right, but also easily testable. It was just the "official" traffic that wasn't forwarded...<br />
<br />
Then a colleague, one of the brightest I've ever had, started talking about tracked connections (<b>/proc/net/nf_conntrack</b>). We could see our netcat connections were tracked, as they were the ones from the other hosts producing data. But there was an important detail. The connections from the "official" producing hosts were associated with the old container IP address (e.g. 172.17.0.2 instead of the new one, 172.18.0.2)!<br />
<br />
The usage of <i>conntrack</i> is visible from iptables' output, in particular for the FORWARDING table:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: Courier;">-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT</span></blockquote>
<br />
Why didn't we observe this behaviour in the other environments? Simple: there the traffic was more sporadic, and so the tracked connections had the time to expire. In the failing environment the continuous flow of UDP packets kept refreshing the tracked connections, even though the forwarding was actually failing.<br />
<br />
Only at that point I had the right scenario described, and I could find this <a href="https://github.com/docker/docker/issues/8795" target="_blank">Docker issue</a>. From there:<br />
<blockquote class="tr_bq">
<span style="background-color: white; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px;">When you restart the container, container's ip has changed, so the DNAT rule, which will route to the new address. But the old connection's state in conntrack is not cleared. So when a packet arrives, it will not go through NAT table again, because it is not "the first" packet. So the solution is clearing the conntrack, [...].</span></blockquote>
<br />
It was a fun day :) I hope this may save some hours of head scratching to somebody in the future.<br />
<br />
<br />Giacomo Vaccahttp://www.blogger.com/profile/15806655752624396579noreply@blogger.com0