Friday, 22 November 2019

kamailio-tests, a testing framework for Kamailio developers

Kamailio is an open source VoIP server, widely used in the VoIP industry for its performance and feature set.

kamailio-tests is a project that aims to provide a level of automated testing for developers.

The main idea is that simple things like loading a module or calling a core or module function can be tested without building an entire infrastructure around Kamailio.

Also, we want to be able to perform the tests against various versions of Kamailio, and on various OS distributions. In this way a backward incompatible function or a regression can be discovered by a developer even though their system uses one single version and OS combination.

This includes also third party libraries, e.g. you may think of testing the HTTP clients with different curl libraries, while Kamailio and OS stay at the same version.

kamailio-test doesn't perform tests at function level (but it requires the application to run) or "integration testing" (because in order to remain generic and compact it doesn't cover complex architectures), but aims to test Kamailio as an isolated application.

In order to achieve this, kamailio-tests relies on Docker to allow developers to experiment and perform their tests in perfect isolation. For example, once you've built the base Docker images you can add and modify tests while travelling on a plane.

Most VoIP platforms have extensive and complex test scenarios that involve setting up accounts, other components, specific logging systems, etc. We can't provide a compact, generic solution that would replace that, but instead we can focus on self-contained (pun intended), simple tests that give us key feedback on issues (and also the opportunity to debug them in that same infrastructure).

The internal structure of the project and how to use it is described in its README, but here I'd like to reiterate how tests are run and how new tests can be added.

Of course you'll need Docker, and it can be a Desktop or server version.

Currently CentOS 7, Debian 9 and Debian 10 are supported via dedicated Dockerfiles.

What version of Kamailio to test can be chosen by simply checking out the desired git branch or tag from Kamailio github repo.

The installation process is described in the README, but let's see comment it here too:

First create a directory where to store the resources and go to it, e.g.:

mkdir kamailio-testing
cd kamailio-testing

Clone the kamailio-tests git repository inside that work directory:

git clone

Clone the kamailio git repository inside that work directory:

git clone

Here you can git checkout the branch or tag you want to test, e.g.

cd kamailio
git checkout 5.3.1
cd ../

Copy the desired Dockerfile from kamailio-tests in the current folder, based on the OS distribution you want, e.g. for Debian Stretch:

cp kamailio-tests/docker/Dockerfile.debian9 Dockerfile

Build the Docker image. Give it the name you want: you'll just have to refer to it when launching the tests. e.g.:

docker build -t kamailio-tests-deb9x .

Run the tests. If you're happy with the default behaviour you can just:

docker run kamailio-tests-deb9x

and all not explicitly excluded tests will be executed.

Some tests may be excluded from execution by listing them in the etc/excludeunits.txt.DISTRIBUTION file, e.g. etc/excludeunits.txt.centos7.

How to add a test?

Create a new folder in units/, with a name starting with 't', followed by a string indicating the module, like 'geoip', and four digits, e.g. 'tgeoip0001'.

Inside this folder the minimum amount of data is a shell script that the test framework will execute and a kamailio.cfg file. See for example this test for geoip module.

The script must start with:

. ../../etc/config
. ../../libs/utils

to prepare configuration and utility commands.

Then you can launch Kamailio with the configuration required, e.g.:

${KAMBIN} -P ${KAMPID} -w ${KAMRUN} -Y ${KAMRUN} -f ./kamailio-tgeoip0001.cfg -a no -ddd -E 2>&1 | tee /tmp/kamailio-tgeoip0001.log &

You can see that logs are being sent to a local file; this is typically how the test result can be verified. Of course you can use a different approach.

Inside the container you have sipsak available, so you can launch a test call to trigger call processing in Kamailio with something like:

sipsak -s sip:alice@
To help triggering the tests also tools like sipp and pjsua can be considered, but are not currently installed in the docker images.

After waiting for the necessary time, you can just kill Kamailio with

kill_pidfile ${KAMPID}

and check the test result by parsing the log file, e.g.:

grep "ip address is registered in US" /tmp/kamailio-tgeoip0001.log
if [ ! "$ret" -eq 0 ] ; then
    exit 1

'exit 1' will make the test fail.

End the test script with

exit 0

to make it pass.

Adding a inside that same folder is highly recommended.

You can add one or more tests and run them before committing the changes.

Once happy please fork the kamailio-tests repo with your changes and raise a pull request.

Thanks for reading; feedback is welcome.

Sunday, 17 November 2019

My notes on Kamailio Developer Meeting - November 2019

The Kamailio Developers Meeting is a two-day event held in Dusseldorf, currently at the second edition. 

As described in, 
"The purpose of the event is to support the interaction between developers and to offer a great environment to work together on relevant topics related to the Kamailio project. It is intended for participants that want to write code for Kamailio and its tools or improve the documentation. There will be no formal presentations, only open discussions, coding or documentation writing sessions."

The sipgate offices offered a very welcoming environment. Noticeable to have a kitchen with chefs for breakfast and lunch, and a private pub, where the social event was held. I noticed the presence of art works and learned that those offices are also part of an art itinerary in Dusseldorf.

We started listing the topics that we wanted to tackle during the event, then discussed a plan to go through them.

The first important activity was the release of kamailio version 5.3.1 (minor bug fix). The release process includes running the tests in kamilio-tests repo (more on this later in a dedicated post), and an issue was found and resolved during the release.

Then it was the time to discuss RPM packages building: Sergey Safarov has made an incredible amount of work to ensure that RPM packages for kamailio are available, and now they are available at Work is in progress to move all the building phases to infrastructure belonging to the kamailio projects and avoiding personal accounts to access cloud services as much as possible.

A documentation review was carried out: it was reported that often the return values of module functions are not documented, and work is encouraged in that direction, including a review of documentation of function parameters. Modules may make available several pseudovariables, and not always they are documented. There's also a dedicated wiki page on pseudovariables that's useful as reference.

Daniel made a briefing on the Kemi framework, explaining how to structure the wrapper functions (two steps, one the parameters manipulation for the standard cfg file, and then encapsulate the code into a separate function. Also in this way executing a function from Kemi doesn't require executing all the typical wrapper manipulations required by cfg functions, impacting positively on performance). Developers need to ensure that when you develop a new function the body of the function is separate from the code that manipulates the parameters.

An open topic focused on the best process to handle "dialog failover". This may be necessary if a kamailio-based component disappears during the dialog lifetime, or if the architecture allows for in-dialog messages to be processed by different entities during a call, even in the typical case where record-routing applies. Currently it's possible to load dialog information on the fly from a DB, see dlg_db_load_callid(), but when an instance "takes over" the dialog information there's typically the need to manipulate the record-routing, which is complex and not exactly "standard" behaviour.

Modules like evapi and http_async_client allow for the management of asynchronous events by spawning dedicated workers which will execute portions of the routing logic when defined events happen. An open discussion is about the best method for managing asynchronous events and at the same time return execution to the main workers when an event is triggered.

Debian packaging; a PGP signed key was created, to benefit from the physical presence of various holders of PGP signatures. An open point is related to the ability to keep earlier versions of the kamailio packages every time a new version is released. This is a known behaviour that may limit the options in some environments, and I've experienced it directly. This is caused by reprepro and what seems to be the right way to go is moving from reprepro and use aptly instead. We'll update on this.

Use of TLSv1.3: the only limitation appears to be in libssl library. It's possible to allow the usage of version 1.3, but only by choosing tlsv1.2+, which admits using v1.2, and so doesn't enforce v1.3 only. When this will be allowed, kamailio administrators will just need to update the configuration and reload the tls module. Remember that only configurations in tls.cfg can be reloaded, while modparam declarations require a restart.

Finally, we know that a common pattern of usage of kamailio involves querying APIs and basing call processing on the API responses. A simple but powerful solution sees the use of http_async_client in collaboration with rtjson; see for example this article from wazo developers: We are discussing whether extending rtjson may be generally helpful for scenarios like this; in that case I'll post an update.

For me this event was a great experience, both from a friendship point of view and as a learning experience. I'm very happy about how the kamailio project is managed, decisions are taken and information is shared.

Here's my report; for obvious reasons this can only be a limited account of what happened during the event, and the information provided is based on my recollection and notes. I'm sure I'm omitting other important material, and I'll be happy to integrate with another post. 

Tuesday, 27 November 2018

You need to slow down

This blog has been historically focused on technical topics in areas related to Real Time Communications, but I'm taking the liberty to digress a little.

I've been reading recently about the dynamics of performance in running.

I'm just an amateur runner, and I always train (and race) alone, so I felt the need to make it more interesting than just reading training tables. I've been studying what is it that limits performance. "Train more", unfortunately, not only doesn't always work, but needs to take into account injuries, and overtraining in general.

One of the first concepts that struck me is that it's been proven that fatigue, and the consequential slow down, does not mean that the body is unable to continue with that effort. What's behind slowing down is a sort of protective mechanism in our nervous system, which wants to prevent the body to reach exhaustion.

Our nervous system is constantly getting feedback signals from the body, including the perception of adverse weather conditions, and computing for how longer the current effort can be sustained. Even knowing how long is left to run is a form of external feedback.

When this computation detects that the effort is too high, the nervous system takes control and doesn't fire as many muscle fibres as the athlete wants to. The athlete thinks the muscles can't continue to work at that level, but in reality they are entering a protective status. Without that, people could literally run until body exhaustion or even death.

I find this fascinating. Evolution has given us a sophisticated algorithm aimed to prevent body exhaustion by generating fatigue symptoms and consequently reduce the actual effort.

What's also fascinating is that it seems this system can be "tricked". One way of doing is by training. Training is a way of educating your nervous system that a certain effort is OK. "There's no need to shut me down, brain, I know what I'm doing. I did that thousands of times in my training sessions."
So while the body adapts to the stress of training, the nervous system too becomes more familiar with that stress, and gives up a little.

Another way of tricking that system is by providing false external feedback. It seems that if you get false information about elements like the air temperature, or even the remaining time in your training session or race, then the nervous system acts accordingly. If it believes it's not as hot as it is, it will not intervene to shut the muscle fibre activation.

Similarly, if the athlete thinks the race is close to the end, the nervous system will allow a prolonged effort. This is why elite marathoners, who clearly haven't underperformed for the first 40 km, can run the last 2 km even faster.

Of course, all these tricks have limits, and the improvements that can be tricked are in the order of small percentages. But still this shows the importance of external feedback.

This is properly explained by Steve Magness in his "Science of running" book. I then read another book from this author, "Peak performance". To be perfectly honest, I was expecting something strictly related to endurance sport, but in this second case the concept of performance was wider.

There seems though to be an analogy between the shutting down of muscle fibres from the nervous system during running, and something that happens behind a desk and is more widely known: mental burnout. From this point of view, mental burnout can be seen as a way of saying "You can't keep going at this (perceived) level of effort. You need to sleep, hydrate, rest, but you keep working. I'm going to take control and shut you down.".

As I'm making my own little experiments, in the future I'd like to write more about this, and in particular about the relationship between effort and rest. 

Monday, 19 November 2018

Docker from scratch

Some time ago I prepared an introductory seminar on Docker, which I called "Docker from scratch".

The audience was a local group of heterogeneous developers. As it typically happens, preparing that material was a great opportunity to understand better some of the aspects.

I then published the slides in Slideshare. I notice now that they got a decent number of views and downloads, so I'm linking those slides here as well for reference.

One big change in respect to 2016 is represented by the choice of abandoning Docker Toolbox in favour of running Docker directly on macOS. I have to say I liked the sandboxing that came with Docker Toolbox, where the docker engine ran inside VirtualBox, now only available for older versions.

Tuesday, 29 May 2018

On Kamailio World 2018, part II

In the first part of my brain dump about this year's edition of Kamailio World I focused mainly on testing. Core developers and application designers want to be able to test the behaviour of Kamailio-based architectures with minimal effort and fast feedback.

A different dimension to testing, that I haven't mentioned in my previous post, was related to Fuzz testing. There were two presentations focused on this: Sandro Gauci's (The easiest way to understand who Sandro is: listen on port 5060 on the public Internet and wait a couple of minutes. You'll see a SIP request from a tool called sipvicious (aka friendly-scanner), a penetration testing tool Sandro wrote (and others misuse)) and Henning Westerholt, historical member of the Kamailio community.

Sandro's presentation focused around fuzzing approaches for RTC in general (slides), while Henning was more specifically focused on Kamailio.

Fuzzing is a sophisticated technique to verify the robustness of a software application, by sending input that can vary greatly from the typical or expected usage. The objective is to find weaknesses that can lead to crashes or other malfunctions, so that they can be fixed. Of course testing a server like Kamailio is even trickier than testing an application that can read from a file. It is a fascinating topic.

Kamailio proved to be very robust: Henning reported an average of  about 1 message every 44 million required to make Kamailio misbehave. The video of Henning's presentation is here (by the way, Pascom have done a great work this year too, providing a flawless video streaming and recording. It feels like we are a little spoiled, because we give it for granted and barely notice all the work behind it).

In terms of learning opportunities for architects and administrators of Kamailio-based infrastructure, I found very valuable Daniel's presentations around high-level scripting (with KEMI) to build the routing logic (Video and slides).

Remember that Lua may not be the most popular - apparently - but it's the one estimated to give you performances closest to the native routing language.

Another valuable presentation was around the Least Cost Routing techniques that the Kamailio environment makes available. (Video, and slides). Some solutions use out of the box modules (like lcr, carrierroute, drouting), some are more indirect (pdt, mtree, dialplan, prefix_route), and others are a combination of them. Must-see if you're working in that area.

Another learning goldmine has been Lorenzo Miniero's (author of Janus, a WebRTC conferencing framework (this definition is mine)) lecture about Privacy, Security and Authentication for WebRTC. (Video and slides) Lorenzo does talk fast, but no word is spoken in vain. Worst case, you can watch the video at 0.5 speed (smile). Interesting the case of double encryption for media.

I guess there's enough for a part III in the near future! To be continued. 

Thursday, 24 May 2018

On Kamailio World 2018, part I

This was my fifth time in a row attending Kamailio World in Berlin. The weather was warmer and sunnier than usual.

Apart from the obvious focus on Kamailio, as usual the RTC ecosystem was well represented (with Janus, Asterisk, FreeSWITCH, Homer, RTPEngine, and many others).

Attendance from the other side of the Atlantic Ocean gave stronger emphasis to the "World" term in the title.

My personal mission this year was to talk about a framework for testing Kamailio as a tool for developers and maintainers of the project: kamailio-tests. The main concept was that early tests that are not focused on a specific business logic (as we all have in our projects) and can be automated will be beneficial to Kamailio's reliability. We want to defer end-to-end testing to later stages, because they are expensive.

To provide a uniform infrastructure where to run the tests, without requiring permanent test environments, we use Docker for this. This is, of course, not the only possible approach, e.g. you could dynamically spawn VMs, AWS EC2s, etc. But Docker can run on your laptop as well as on a full-fledged CI environment, and this makes it easier to use for the developers.

Please take a look at the slides for more details. The feedback has been great so far, and this proved various points:

1. Conferences for developers are not paid holidays for IT guys, but opportunities for knowledge sharing and collaboration (I would say, in particular if Open Source is in the equation).

2. "Functional" or "component" testing is needed by many, but we haven't a mature solution yet.

3. Docker in RTC is less a fancy technology borrowed by other IT areas and more an everyday tool.

Some have already volunteered to help me improve kamailio-tests, and their point of view will be very useful. More on this project in the future.

Around the topic of testing, in this case not Kamailio itself but more the business logic built around it, there have been interesting insights from Sebastian Damm (sipgate) and Alex Sosic (evosip). 

Sebastian presented an approach that benefits from moving the Kamailio routing logic from the native language to KEMI with Lua ( Alex presented a way to verify the routing logic is going through the expected paths, again with Docker, and sipp.

KEMI is an extension of Kamailio that allows developers to write the routing logic in high level languages, like Lua, Python, JS and others. Anedoctical experience made me think Lua was the most popular, while apparently Python is. For what concerns Lua in the RTC world, I wrote a few notes in February:

The advantages of working with a high level language are obvious: easier to read and maintain, it's easier to test the functions in isolation, and also easier to involve developers without specific knowledge in Kamailio's routing logic script. They will still need to understand how Kamailio works though, and the underlying protocols, so unless you're doing something extremely basic, it's not a complete abstraction from how Kamailio manages its role as "programmable SIP Proxy".

I have tons of notes from Kamailio World, but if I wait to go through all of them before writing something here, there will be the 2019 edition to talk about. So here's at least a part I.

Monday, 5 February 2018

The interesting case of Lua in RTC world

An interesting pattern that caught my attention is the role that Lua is gaining in the RTC (Real-Time Communications) world.

Lua is a small-footprint programming language, powerful while keeping a simple syntax.

I’ve been using Lua to script dialplan actions for FreeSWITCH since about 2014. It has provided me with a way to define relatively complex logic and speed up the definition of FS’ behaviour.

Delegating this type of logic to a scripting language had several advantages, such as:
  • It’s easier to read and understand than native dialplans or native routing logic.
  • Makes unit testing of the dialplan possible/easier.
  • Allows changing some pieces of logic easily, in many cases preventing expensive reload of modules or restart of applications.

I’ve been using Lua for Kamailio as well. Kamailio is an open source programmable SIP Proxy. In a specific case, some bits of the routing logic required regex processing and was expecting to change often: an ideal case for an external script to do that work.
When the logic changes, it’s sufficient to instruct Kamailio to reload the script, and from that moment on the new requests being processed will use the new logic.

Recent versions of Kamailio though add a framework called KEMI. This opens up new possibilities, and also provides support for many other scripting languages, python being the most popular, JS, Squirrel. Still, Lua appears to provide the fastest implementations (with no observable performance degradation) while others have limitations. Python, as you can imagine, provides a rich set of functions and libraries, but it’s not as performant and the reload mechanism currently has some issues.

Wireshark, a tool to capture and analyse network activity, exposes a useful API for Lua. You can use the API to define your own Wireshark dissector (which you’ll need to install as a plugin). This has performance limitations in comparison with dissectors written in C - and so it’s recommended for prototyping only - but still can solve your problem perfectly. Out of need, I wrote a Wireshark dissector for HEP, a binary protocol used in the Homer environment. Homer is an open source framework for the monitoring and analysis of Real-Time Communications.

Last weekend a new interesting case was presented by Lorenzo Miniero at FOSDEM. The target application was Janus, an open source framework to build WebRTC gateways.

Janus allows to build applications by defining the transport and business logic as plugins, on top of the core that implements the WebRTC stack.
It’s written in C and so far users needed to write plugins with that language. The Janus developers have introduced the possibility to write plugins in Lua.

In his presentation Lorenzo explains also in detail what approach is best to use for a Real-Time application to interact with single threaded language like Lua in an asynchronous context.

Just a funny note: Lua uses a double dash for commenting out a line: '--'. Be careful when you watch diffs in a terminal because a removed comment will start with '- --‘ and may not the easiest thing to interpret (experiences may vary depending on the terminal, of course!).