Friday 22 November 2019

kamailio-tests, a testing framework for Kamailio developers

Kamailio is an open source VoIP server, widely used in the VoIP industry for its performance and feature set.

kamailio-tests is a project that aims to provide a level of automated testing for developers.

The main idea is that simple things like loading a module or calling a core or module function can be tested without building an entire infrastructure around Kamailio.

Also, we want to be able to perform the tests against various versions of Kamailio, and on various OS distributions. In this way a backward incompatible function or a regression can be discovered by a developer even though their system uses one single version and OS combination.

This includes also third party libraries, e.g. you may think of testing the HTTP clients with different curl libraries, while Kamailio and OS stay at the same version.

kamailio-test doesn't perform tests at function level (but it requires the application to run) or "integration testing" (because in order to remain generic and compact it doesn't cover complex architectures), but aims to test Kamailio as an isolated application.

In order to achieve this, kamailio-tests relies on Docker to allow developers to experiment and perform their tests in perfect isolation. For example, once you've built the base Docker images you can add and modify tests while travelling on a plane.

Most VoIP platforms have extensive and complex test scenarios that involve setting up accounts, other components, specific logging systems, etc. We can't provide a compact, generic solution that would replace that, but instead we can focus on self-contained (pun intended), simple tests that give us key feedback on issues (and also the opportunity to debug them in that same infrastructure).

The internal structure of the project and how to use it is described in its README, but here I'd like to reiterate how tests are run and how new tests can be added.

Of course you'll need Docker, and it can be a Desktop or server version.

Currently CentOS 7, Debian 9 and Debian 10 are supported via dedicated Dockerfiles.

What version of Kamailio to test can be chosen by simply checking out the desired git branch or tag from Kamailio github repo.

The installation process is described in the README, but let's see comment it here too:

First create a directory where to store the resources and go to it, e.g.:

mkdir kamailio-testing
cd kamailio-testing

Clone the kamailio-tests git repository inside that work directory:

git clone

Clone the kamailio git repository inside that work directory:

git clone

Here you can git checkout the branch or tag you want to test, e.g.

cd kamailio
git checkout 5.3.1
cd ../

Copy the desired Dockerfile from kamailio-tests in the current folder, based on the OS distribution you want, e.g. for Debian Stretch:

cp kamailio-tests/docker/Dockerfile.debian9 Dockerfile

Build the Docker image. Give it the name you want: you'll just have to refer to it when launching the tests. e.g.:

docker build -t kamailio-tests-deb9x .

Run the tests. If you're happy with the default behaviour you can just:

docker run kamailio-tests-deb9x

and all not explicitly excluded tests will be executed.

Some tests may be excluded from execution by listing them in the etc/excludeunits.txt.DISTRIBUTION file, e.g. etc/excludeunits.txt.centos7.

How to add a test?

Create a new folder in units/, with a name starting with 't', followed by a string indicating the module, like 'geoip', and four digits, e.g. 'tgeoip0001'.

Inside this folder the minimum amount of data is a shell script that the test framework will execute and a kamailio.cfg file. See for example this test for geoip module.

The script must start with:

. ../../etc/config
. ../../libs/utils

to prepare configuration and utility commands.

Then you can launch Kamailio with the configuration required, e.g.:

${KAMBIN} -P ${KAMPID} -w ${KAMRUN} -Y ${KAMRUN} -f ./kamailio-tgeoip0001.cfg -a no -ddd -E 2>&1 | tee /tmp/kamailio-tgeoip0001.log &

You can see that logs are being sent to a local file; this is typically how the test result can be verified. Of course you can use a different approach.

Inside the container you have sipsak available, so you can launch a test call to trigger call processing in Kamailio with something like:

sipsak -s sip:alice@
To help triggering the tests also tools like sipp and pjsua can be considered, but are not currently installed in the docker images.

After waiting for the necessary time, you can just kill Kamailio with

kill_pidfile ${KAMPID}

and check the test result by parsing the log file, e.g.:

grep "ip address is registered in US" /tmp/kamailio-tgeoip0001.log
if [ ! "$ret" -eq 0 ] ; then
    exit 1

'exit 1' will make the test fail.

End the test script with

exit 0

to make it pass.

Adding a inside that same folder is highly recommended.

You can add one or more tests and run them before committing the changes.

Once happy please fork the kamailio-tests repo with your changes and raise a pull request.

Thanks for reading; feedback is welcome.

Sunday 17 November 2019

My notes on Kamailio Developer Meeting - November 2019

The Kamailio Developers Meeting is a two-day event held in Dusseldorf, currently at the second edition. 

As described in, 
"The purpose of the event is to support the interaction between developers and to offer a great environment to work together on relevant topics related to the Kamailio project. It is intended for participants that want to write code for Kamailio and its tools or improve the documentation. There will be no formal presentations, only open discussions, coding or documentation writing sessions."

The sipgate offices offered a very welcoming environment. Noticeable to have a kitchen with chefs for breakfast and lunch, and a private pub, where the social event was held. I noticed the presence of art works and learned that those offices are also part of an art itinerary in Dusseldorf.

We started listing the topics that we wanted to tackle during the event, then discussed a plan to go through them.

The first important activity was the release of kamailio version 5.3.1 (minor bug fix). The release process includes running the tests in kamilio-tests repo (more on this later in a dedicated post), and an issue was found and resolved during the release.

Then it was the time to discuss RPM packages building: Sergey Safarov has made an incredible amount of work to ensure that RPM packages for kamailio are available, and now they are available at Work is in progress to move all the building phases to infrastructure belonging to the kamailio projects and avoiding personal accounts to access cloud services as much as possible.

A documentation review was carried out: it was reported that often the return values of module functions are not documented, and work is encouraged in that direction, including a review of documentation of function parameters. Modules may make available several pseudovariables, and not always they are documented. There's also a dedicated wiki page on pseudovariables that's useful as reference.

Daniel made a briefing on the Kemi framework, explaining how to structure the wrapper functions (two steps, one the parameters manipulation for the standard cfg file, and then encapsulate the code into a separate function. Also in this way executing a function from Kemi doesn't require executing all the typical wrapper manipulations required by cfg functions, impacting positively on performance). Developers need to ensure that when you develop a new function the body of the function is separate from the code that manipulates the parameters.

An open topic focused on the best process to handle "dialog failover". This may be necessary if a kamailio-based component disappears during the dialog lifetime, or if the architecture allows for in-dialog messages to be processed by different entities during a call, even in the typical case where record-routing applies. Currently it's possible to load dialog information on the fly from a DB, see dlg_db_load_callid(), but when an instance "takes over" the dialog information there's typically the need to manipulate the record-routing, which is complex and not exactly "standard" behaviour.

Modules like evapi and http_async_client allow for the management of asynchronous events by spawning dedicated workers which will execute portions of the routing logic when defined events happen. An open discussion is about the best method for managing asynchronous events and at the same time return execution to the main workers when an event is triggered.

Debian packaging; a PGP signed key was created, to benefit from the physical presence of various holders of PGP signatures. An open point is related to the ability to keep earlier versions of the kamailio packages every time a new version is released. This is a known behaviour that may limit the options in some environments, and I've experienced it directly. This is caused by reprepro and what seems to be the right way to go is moving from reprepro and use aptly instead. We'll update on this.

Use of TLSv1.3: the only limitation appears to be in libssl library. It's possible to allow the usage of version 1.3, but only by choosing tlsv1.2+, which admits using v1.2, and so doesn't enforce v1.3 only. When this will be allowed, kamailio administrators will just need to update the configuration and reload the tls module. Remember that only configurations in tls.cfg can be reloaded, while modparam declarations require a restart.

Finally, we know that a common pattern of usage of kamailio involves querying APIs and basing call processing on the API responses. A simple but powerful solution sees the use of http_async_client in collaboration with rtjson; see for example this article from wazo developers: We are discussing whether extending rtjson may be generally helpful for scenarios like this; in that case I'll post an update.

For me this event was a great experience, both from a friendship point of view and as a learning experience. I'm very happy about how the kamailio project is managed, decisions are taken and information is shared.

Here's my report; for obvious reasons this can only be a limited account of what happened during the event, and the information provided is based on my recollection and notes. I'm sure I'm omitting other important material, and I'll be happy to integrate with another post. 

Wireshark setting to interpret UDP as RTP automatically

 Before I forget again, a Wireshark setting that can help saving time by trying to interpret any UDP as RTP, if possible: Analyze --> Ena...