Monday, 5 February 2018

The interesting case of Lua in RTC world

An interesting pattern that caught my attention is the role that Lua is gaining in the RTC (Real-Time Communications) world.

Lua is a small-footprint programming language, powerful while keeping a simple syntax.

I’ve been using Lua to script dialplan actions for FreeSWITCH since about 2014. It has provided me with a way to define relatively complex logic and speed up the definition of FS’ behaviour.

Delegating this type of logic to a scripting language had several advantages, such as:
  • It’s easier to read and understand than native dialplans or native routing logic.
  • Makes unit testing of the dialplan possible/easier.
  • Allows changing some pieces of logic easily, in many cases preventing expensive reload of modules or restart of applications.

I’ve been using Lua for Kamailio as well. Kamailio is an open source programmable SIP Proxy. In a specific case, some bits of the routing logic required regex processing and was expecting to change often: an ideal case for an external script to do that work.
When the logic changes, it’s sufficient to instruct Kamailio to reload the script, and from that moment on the new requests being processed will use the new logic.

Recent versions of Kamailio though add a framework called KEMI. This opens up new possibilities, and also provides support for many other scripting languages, python being the most popular, JS, Squirrel. Still, Lua appears to provide the fastest implementations (with no observable performance degradation) while others have limitations. Python, as you can imagine, provides a rich set of functions and libraries, but it’s not as performant and the reload mechanism currently has some issues.

Wireshark, a tool to capture and analyse network activity, exposes a useful API for Lua. You can use the API to define your own Wireshark dissector (which you’ll need to install as a plugin). This has performance limitations in comparison with dissectors written in C - and so it’s recommended for prototyping only - but still can solve your problem perfectly. Out of need, I wrote a Wireshark dissector for HEP, a binary protocol used in the Homer environment. Homer is an open source framework for the monitoring and analysis of Real-Time Communications.

Last weekend a new interesting case was presented by Lorenzo Miniero at FOSDEM. The target application was Janus, an open source framework to build WebRTC gateways.

Janus allows to build applications by defining the transport and business logic as plugins, on top of the core that implements the WebRTC stack.
It’s written in C and so far users needed to write plugins with that language. The Janus developers have introduced the possibility to write plugins in Lua.

In his presentation Lorenzo explains also in detail what approach is best to use for a Real-Time application to interact with single threaded language like Lua in an asynchronous context.

Just a funny note: Lua uses a double dash for commenting out a line: '--'. Be careful when you watch diffs in a terminal because a removed comment will start with '- --‘ and may not the easiest thing to interpret (experiences may vary depending on the terminal, of course!).

Sunday, 21 January 2018

Cache busting when building Docker images

One of the handiest features of the docker build system is the caching system.

'docker build' tries to reuse the layers already built until something changes inside the Dockerfile. In this way, we can save several minutes when rebuilding an image if the changes happen further down the list in the Dockerfile.

Sometimes, though, we do want to invalidate the cache and ensure the next build won't use it.

To do this an option is to pass the '--no-cache' argument to 'docker build'.

When dealing with 'apt-get install' instructions though there are other tricks. I found this document on Dockerfile best practices very useful.

First of all an observation. If you have 'RUN apt-get update' as a single line of a Dockerfile, followed by the installation of a package, e.g.:

RUN apt-get update
RUN apt-get install -y nginx

then changing the list of packages and running again the build command won't trigger an 'apt-get update': that line hasn't changed so docker build reuses the cache. It might not be what you want.

To force cache invalidation for this specific case the recommendation is to use those commands in the form:

RUN apt-get update && apt-get install -y nginx

This will always install the latest version of the packages. It even has a name: "cache busting".

Another recommendation I like is to put each package on a single line, and have them in alphabetical order: this will ease visual inspection and prevent duplicates or other undesired conditions.

Of course, you can also specify exact versions for the packages as you would normally do with 'apt-get install'. That's "version pinning" and it invalidates the cache too.

You can find all this on the linked page on Dockerfile best practices; this is just my digested interpretation.

Just one more thing: a way to limit the size of a built image is to clean up the content of '/var/lib/apt/lists' in the same RUN command, e.g.:

RUN apt-get update && apt-get install -y \
    aufs-tools \
    automake \
    build-essential \
&& rm -rf /var/lib/apt/lists/* 

The command above will build an image layer that doesn't contain the apt cache.

If you had instead used this:

RUN apt-get update && apt-get install -y \
    aufs-tools \
    automake \
RUN rm -rf /var/lib/apt/lists/*

you would have had not only a larger layer, containing the apt cache, but also an additional layer generated by the second RUN command.

Saturday, 13 January 2018

SIP - ACK loose routing

If you've ever worked with SIP, you must have stumbled upon a trace with 200 OK to INVITE being retransmitted for about 30" and then the call just being set up fail.

The ACK was never received.

Then comes the interesting part: discovering why.

Here are some notes about what should happen, in particular when there are multiple proxies along the path, and with a little additional complexity of one of the proxies with two network interfaces. All this assuming loose routing everywhere. The main reference here of course is RFC 3261.

Isn't an image worth a thousand words? Then here's a sequence diagram:

All Record-Route headers are assumed to carry loose routing URIs (they have the ;lr attribute).

B, C and D, working as proxies that want to stay in the path, record route themselves. For this reason E, the UAS and "callee", receives an INVITE with a list of Record-Route headers with B, C and D.
In particular for B there will be two Record-Route headers, since B is using two separate interfaces, one facing A and the other facing C.

In typical cases the two interfaces represent the interaction with the public Internet on a side and a private infrastructure on the other. But it's not important for this discussion.

Omitting provisional responses for simplicity, let's assume E responds immediately with a 200 OK. This response will have the same list of R-R headers, in the same order, as received by E.
E will also add its URI in the Contact header of the 200 OK.

In this loose routing context the IP address in E's Contact's URI will be relevant only for D in the future.

D, C and B don't modify the list of Record-Route headers, and A receives it as sent by E.

Apart from the operations related to the media session set up, A will send the ACK to the 200 OK.
This ACK will have a Request URI with E's Contact URI (stripped of anything that can't se inside a Request URI), and a Route header list which is basically the received Record-Route header list inverted (see images).

A is saying: "Route this ACK to E, routing it via this list of hops".

When B receives that ACK, it must recognise that the topmost Route headers are B itself, remove them from the Route list, and pick b2 as the interface to deliver the ACK to C.

C and D will have an easier task to remove a single Route header, the one representing them, and deliver the ACK to the next route.

For D, the next route will be in fact E, and the ACK will be routed using only the Request URI, as the Route headers have all been eliminated. This is the only step where the IP address that E has set in the Contact of its 200 OK response needs to be visible by another entity, namely D.

ACK routing as explained in RFC 3665

To further reiterate this concept, let's look at a somewhat simpler example in RFC 3665.

The ACK part is:

F15 ACK Alice -> Proxy 1

  ACK SIP/2.0
  Via: SIP/2.0/TCP;branch=z9hG4bK74b76
  Max-Forwards: 70
  Route: ,
  From: Alice ;tag=9fxced76sl
  To: Bob ;tag=314159
  CSeq: 2 ACK
  Content-Length: 0

  F16 ACK Proxy 1 -> Proxy 2

  ACK SIP/2.0
  Via: SIP/2.0/TCP;branch=z9hG4bK2d4790.1
  Via: SIP/2.0/TCP;branch=z9hG4bK74b76
  Max-Forwards: 69
  From: Alice ;tag=9fxced76sl
  To: Bob ;tag=314159
  CSeq: 2 ACK
  Content-Length: 0

  F17 ACK Proxy 2 -> Bob

  ACK SIP/2.0
  Via: SIP/2.0/TCP;branch=z9hG4bK721e4.1

You can see there's no requirement for Proxy 1 to be able to reach the UAC contact ( might be completely unreachable from Proxy 1).
It's Proxy 2's responsibility to route the ACK in the last hop towards Bob.
Proxy 1 must leave the R-URI as is (see below for more details on proxy behaviour), strip itself from the list of Routes and route the ACK to the new topmost Route (Proxy 2).
Proxy 2 will strip itself from the Route list, being the topmost Route, and forward the ACK to Bob. There are no more Route headers.
Only at the last hop Bob's contact reachability is relevant, and it is for Proxy 2 only.

More about the behaviour of the proxies to corroborate the ACK routing

From RFC 3261, 16.4:

“  If the first value in the Route header field indicates this proxy,
  the proxy MUST remove that value from the request.”

From RFC 3261, 16.5:

“     A proxy can only change the Request-URI of a request during

      forwarding if it is responsible for that URI.”

APPENDIX - Why is the ACK to 200 OK to INVITE a separate transaction?

From RFC 3261, ch. 17:

     The reason for this separation is rooted in the importance of
      delivering all 200 (OK) responses to an INVITE to the UAC.  To
      deliver them all to the UAC, the UAS alone takes responsibility
      for retransmitting them (see Section, and the UAC alone
      takes responsibility for acknowledging them with ACK (see Section  Since this ACK is retransmitted only by the UAC, it is
      effectively considered its own transaction.

Tuesday, 9 January 2018

Copying a file from a Docker container to the host

Often things are more convoluted than you expect. Sometimes they are easier than you fear.

Here's an example: you're generating files inside a Docker container (with no volumes configured) and you realise you need some of those files available in the host.

A simple solution is to use 'docker cp', with a format like


Official documentation.
A Stackoverflow question with more discussions.

Thursday, 30 November 2017

Around sipp with pcap and authentication support

This is just a practical guide to have latest sipp on debian, to benefit from features that are not available in the stock package (sip-tester - version 3.2.1). For example support for playing pcap files or computing authentication hashes.

Build sipp with pcap support

apt-get install autoconf libncurses5-dev libpcap-dev g++
cd /usr/local/src/
git clone
cd sipp/
./ --with-pcap

./sipp is the built binary. You can see the version and capabilities with './sipp -v', e.g.:

$ ./sipp -v
SIPp v3.6-dev-149-gb95f98f-PCAP-RTPSTREAM.

This version will be able to use actions like

exec rtp_stream="file.wav"


exec play_pcap_audio="pcap/g711a.pcap"

(see details in current documentation).

A little caveat for 'rtp_stream' and WAV files. As the documentation says sipp expects a WAV file encoded with PCAM ('A-law'). You can also loop that audio for as long as you need, by adding ',-1' after the file name (and within the double quotes).
But sipp will also send the WAV header in the RTP payload. If this is not acceptable for you, then you can strip the WAV header with something like:

sox -t WAV -r 8000 -c 1 -e a-law audio.wav audio.raw

(where sox is this tool).

play_pcap_audio is useful when you want full control of the RTP produced by sipp, e.g. the exact RTP SSRC, missing packets, Mark set bits, etc. Very powerful.

Add authentication support

Same dependencies as above, but add:
apt-get install libssl-dev

./ --with-pcap --with-openssl

./sipp is the built binary. You can see the version and capabilities with './sipp -v', e.g.:

$ ./sipp -v
SIPp v3.6-dev-149-gb95f98f-TLS-PCAP-RTPSTREAM.

See the authentication features in current documentation.

More to come around these topics soon.

Sunday, 7 May 2017

Monitoring FreeSWITCH with Homer - adding non-SIP events with hepipe.js

FreeSWITCH (from now on FS) provides a very powerful tool to interact with it: the Event Socket (ESL), made available via the mod_event_socket module (

ESL is a TCP socket where applications can connect to, and perform two types of action:
1. Send commands.
2. Subscribe to events.

The applications subscribing to events will receive the expected notifications through the same TCP connection.
A simple protocol and transport made it possible for various libraries in various languages to be written.

Events from FS can serve multiple purposes. In this article I'm interested in monitoring and event correlation.

Homer ( is a widely used, open source tool to monitor RTC infrastructures. It has a multitude of features, but the core is the ability to collect SIP signalling and other events from RTC applications, and perform a form of correlation. In particular, it's able to correlate the SIP signalling involved in a call with other events like RTCP reports or log lines associated to the same call.

While FS, through the sofia module, has native support for transmitting SIP signalling to Homer, the acquisition of other events can happen by collecting these events from the ESL, filtering them, and sending them to Homer with the proper formatting.

This is what hepipe.js ( does. hepipe.js is a simple nodejs application that is able to:
- connect to FS via ESL
- subscribe to specific event categories
- format the events into HEP messages. HEP is a binary protocol used to transmit data to Homer.

hepipe.js is easy to use:
- Clone it
- Run 'sudo npm install' to install the required dependencies
- Set configuration
- Run it ('sudo node hepipe.js', or 'sudo nodejs hepipe.js')

The configuration is organized in "modules", and for this example you'll have to configure at a minimum the esl module and the hep module.
Edit a config.js file in the same folder as hepipe.js with something like:

var config = {
  hep_config: {
    debug: true,
    HEP_SERVER: '',
    HEP_PORT: 9060
  esl_config: {
    debug: true,
    ESL_SERVER: '',
    ESL_PORT: 8021,
    ESL_PASS: 'ClueCon',
    HEP_PASS: 'multipass',
    HEP_ID: 2222,
    report_call_events: true,
    report_rtcp_events: true,
    report_qos_events: true

module.exports = config;

This will configure the hep module to send data to a Homer instance listening on UDP, IP address, port 9060, and will try to connect to a FS' ESL on localhost, via TCP port 8021 and using the default password. See also other configuration examples in the examples/ folder.

Please note that the ESL requires at least two levels of authorization: a password and an ACL. You can check conf/autoload_config/event_socket.conf.xml in the FS configuration folder to ensure the ACL in use, if any, is compatible to the source IP address of hepipe.js when connecting to FS.

Once config.js will be ready, launch hepipe.js and look at the events being sent to Homer.
Note that you can filter out event types by setting to false some of these:
    report_call_events: true,
    report_rtcp_events: true,
    report_qos_events: true

Assuming FS is configured to send SIP signalling to the same Homer instance, you'll be able to see, associated to its SIP call flows, also the events captured by hepipe.js.

See for example below log lines created by FS, sent to Homer, and then presented together with the SIP signalling in Homer:


Sunday, 15 January 2017

Analysing Opus media from network traces

VoIP/RTC platforms have typically many elements processing audio. When an issue is reported it's important to be able to restrict the investigation field, to save time and resources.

A typical scenario is bad or missing audio perceived on the client side. As I've done previously (here for Opus and here for SILK) I'd like to share some practical strategies to extract audio from a pcap trace (to verify the audio received/sent was "correct") and to "re-play" the call inside a test bed (to verify that the audio was good but also carried correctly by the RTP stream). Of course a lot can be inferred by indirect data, for example the summary of RTCP reports showing the number of packets exchanged, packets lost, the latency. But sometimes those metrics are perfect while the issue is still there.

Focusing in this case on Opus audio, and starting from a pcap file with the network traces for a call under investigation, let's see how to decode the Opus frames carried by the RTP packets into an audible WAV file.

You don't even need to have captured the signalling: it's sufficient to have the UDP packets carrying the RTP. If signalling is not visible by Wireshark it may not recognize that the UDP packets carry RTP, but you give it a hint by right-clicking on a frame and "Decode as..." and selecting "RTP".

It's typically easy to find the relevant RTP stream in Wireshark ("Telephony -> RTP -> RTP Streams"), select it, and prepare a filter. Then you can Export the packets belonging to that stream into a dedicated pcap file ("File --> Export Specified Packets...").

I've then modified opusrtp from a fork of opus-tools in order to be able to extract the payload from a given pcap, creating an Opus file. e.g.:

./opusrtp --extract trace.pcap

This will output a rtpdump.opus file, which can be converted into a WAV file directly with opusdec, still part of opus-tools:

./opusdec --rate 8000 rtpdump.opus audio.wav

You can listen to the wav file and verify whether at least the carried RTP payload was valid.

The network trace with the RTP can also be used to re-play the call, injecting the same RTP as in the call under investigation. With the help of sipp you can set up a rudimentary but very powerful test bed. Use the standard UAS scenario (e.g. in uas.xml), but with an additional part:

right after the ACK is received. If you launch sipp with a command like:

sipp -sf uas.xml -i MEDIA_IP_ADDRESS

you'll be able to call sipp. It will answer the call, as the scenario mandates, and will play the RTP contained in rtp_opus.pcap. The stream SSRC, timestamps, even Marker bits will be preserved. This will give you quite an accurate simulation of the stream received by the client in the original call.

It should be straightforward to reach all these components. For opus-tools, on a debian-based machine, you can just:

sudo apt-get install libogg-dev libpcap-dev
git clone
cd opus-tools

For sipp:
sudo apt-get install sip-tester

I hope this will save the reader some time in future investigations.

UPDATE: The fork of opus-tools was merged into the original repo, so you don't need my repo.

UPDATE 2: This only works if the opus payload in the RTP is not encrypted. Also it may need a patch when the extension header for volume indications are used (e.g. 'urn:ietf:params:rtp-hdrext:ssrc-audio-level', see RFC-6464). Don't forget that at the moment the payload type is harcoded to 120. You may need to rebuild opusrtp with the type your trace has, e.g. 96 (It should be easy to pass it as command line argument, something for a quiet moment).