kamailio-tests, a testing framework for Kamailio developers

Kamailio is an open source VoIP server, widely used in the VoIP industry for its performance and feature set.

kamailio-tests is a project that aims to provide a level of automated testing for developers.

The main idea is that simple things like loading a module or calling a core or module function can be tested without building an entire infrastructure around Kamailio.

Also, we want to be able to perform the tests against various versions of Kamailio, and on various OS distributions. In this way a backward incompatible function or a regression can be discovered by a developer even though their system uses one single version and OS combination.

This includes also third party libraries, e.g. you may think of testing the HTTP clients with different curl libraries, while Kamailio and OS stay at the same version.

kamailio-test doesn't perform tests at function level (but it requires the application to run) or "integration testing" (because in order to remain generic and compact it doesn't cover complex architectures), but aims to test Kamailio as an isolated application.

In order to achieve this, kamailio-tests relies on Docker to allow developers to experiment and perform their tests in perfect isolation. For example, once you've built the base Docker images you can add and modify tests while travelling on a plane.

Most VoIP platforms have extensive and complex test scenarios that involve setting up accounts, other components, specific logging systems, etc. We can't provide a compact, generic solution that would replace that, but instead we can focus on self-contained (pun intended), simple tests that give us key feedback on issues (and also the opportunity to debug them in that same infrastructure).

The internal structure of the project and how to use it is described in its README, but here I'd like to reiterate how tests are run and how new tests can be added.

Of course you'll need Docker, and it can be a Desktop or server version.

Currently CentOS 7, Debian 9 and Debian 10 are supported via dedicated Dockerfiles.

What version of Kamailio to test can be chosen by simply checking out the desired git branch or tag from Kamailio github repo.

The installation process is described in the README, but let's see comment it here too:

First create a directory where to store the resources and go to it, e.g.:

mkdir kamailio-testing
cd kamailio-testing

Clone the kamailio-tests git repository inside that work directory:

git clone https://github.com/kamailio/kamailio-tests

Clone the kamailio git repository inside that work directory:

git clone https://github.com/kamailio/kamailio

Here you can git checkout the branch or tag you want to test, e.g.

cd kamailio
git checkout 5.3.1
cd ../

Copy the desired Dockerfile from kamailio-tests in the current folder, based on the OS distribution you want, e.g. for Debian Stretch:

cp kamailio-tests/docker/Dockerfile.debian9 Dockerfile

Build the Docker image. Give it the name you want: you'll just have to refer to it when launching the tests. e.g.:

docker build -t kamailio-tests-deb9x .

Run the tests. If you're happy with the default behaviour you can just:

docker run kamailio-tests-deb9x

and all not explicitly excluded tests will be executed.

Some tests may be excluded from execution by listing them in the etc/excludeunits.txt.DISTRIBUTION file, e.g. etc/excludeunits.txt.centos7.

How to add a test?

Create a new folder in units/, with a name starting with 't', followed by a string indicating the module, like 'geoip', and four digits, e.g. 'tgeoip0001'.

Inside this folder the minimum amount of data is a shell script that the test framework will execute and a kamailio.cfg file. See for example this test for geoip module.

The script must start with:

#!/bin/bash
. ../../etc/config
. ../../libs/utils

to prepare configuration and utility commands.

Then you can launch Kamailio with the configuration required, e.g.:

${KAMBIN} -P ${KAMPID} -w ${KAMRUN} -Y ${KAMRUN} -f ./kamailio-tgeoip0001.cfg -a no -ddd -E 2>&1 | tee /tmp/kamailio-tgeoip0001.log &

You can see that logs are being sent to a local file; this is typically how the test result can be verified. Of course you can use a different approach.

Inside the container you have sipsak available, so you can launch a test call to trigger call processing in Kamailio with something like:

sipsak -s sip:alice@127.0.0.1

To help triggering the tests also tools like sipp and pjsua can be considered, but are not currently installed in the docker images.

After waiting for the necessary time, you can just kill Kamailio with

kill_pidfile ${KAMPID}

and check the test result by parsing the log file, e.g.:

grep "ip address is registered in US" /tmp/kamailio-tgeoip0001.log
ret=$?
if [ ! "$ret" -eq 0 ] ; then
    exit 1
fi

'exit 1' will make the test fail.

End the test script with

exit 0

to make it pass.

Adding a README.md inside that same folder is highly recommended.

You can add one or more tests and run them before committing the changes.

Once happy please fork the kamailio-tests repo with your changes and raise a pull request.

Thanks for reading; feedback is welcome.

My notes on Kamailio Developer Meeting - November 2019

The Kamailio Developers Meeting is a two-day event held in Dusseldorf, currently at the second edition. 

As described in https://www.kamailio.org/w/developers-meeting/, 

"The purpose of the event is to support the interaction between developers and to offer a great environment to work together on relevant topics related to the Kamailio project. It is intended for participants that want to write code for Kamailio and its tools or improve the documentation. There will be no formal presentations, only open discussions, coding or documentation writing sessions."

The sipgate offices offered a very welcoming environment. Noticeable to have a kitchen with chefs for breakfast and lunch, and a private pub, where the social event was held. I noticed the presence of art works and learned that those offices are also part of an art itinerary in Dusseldorf.

We started listing the topics that we wanted to tackle during the event, then discussed a plan to go through them.

The first important activity was the release of kamailio version 5.3.1 (minor bug fix). The release process includes running the tests in kamilio-tests repo (more on this later in a dedicated post), and an issue was found and resolved during the release.

Then it was the time to discuss RPM packages building: Sergey Safarov has made an incredible amount of work to ensure that RPM packages for kamailio are available, and now they are available at https://rpm.kamailio.org/. Work is in progress to move all the building phases to infrastructure belonging to the kamailio projects and avoiding personal accounts to access cloud services as much as possible.

A documentation review was carried out: it was reported that often the return values of module functions are not documented, and work is encouraged in that direction, including a review of documentation of function parameters. Modules may make available several pseudovariables, and not always they are documented. There's also a dedicated wiki page on pseudovariables that's useful as reference.

Daniel made a briefing on the Kemi framework, explaining how to structure the wrapper functions (two steps, one the parameters manipulation for the standard cfg file, and then encapsulate the code into a separate function. Also in this way executing a function from Kemi doesn't require executing all the typical wrapper manipulations required by cfg functions, impacting positively on performance). Developers need to ensure that when you develop a new function the body of the function is separate from the code that manipulates the parameters.

An open topic focused on the best process to handle "dialog failover". This may be necessary if a kamailio-based component disappears during the dialog lifetime, or if the architecture allows for in-dialog messages to be processed by different entities during a call, even in the typical case where record-routing applies. Currently it's possible to load dialog information on the fly from a DB, see dlg_db_load_callid(), but when an instance "takes over" the dialog information there's typically the need to manipulate the record-routing, which is complex and not exactly "standard" behaviour.

Modules like evapi and http_async_client allow for the management of asynchronous events by spawning dedicated workers which will execute portions of the routing logic when defined events happen. An open discussion is about the best method for managing asynchronous events and at the same time return execution to the main workers when an event is triggered.

Debian packaging; a PGP signed key was created, to benefit from the physical presence of various holders of PGP signatures. An open point is related to the ability to keep earlier versions of the kamailio packages every time a new version is released. This is a known behaviour that may limit the options in some environments, and I've experienced it directly. This is caused by reprepro and what seems to be the right way to go is moving from reprepro and use aptly instead. We'll update on this.

Use of TLSv1.3: the only limitation appears to be in libssl library. It's possible to allow the usage of version 1.3, but only by choosing tlsv1.2+, which admits using v1.2, and so doesn't enforce v1.3 only. When this will be allowed, kamailio administrators will just need to update the configuration and reload the tls module. Remember that only configurations in tls.cfg can be reloaded, while modparam declarations require a restart.

Finally, we know that a common pattern of usage of kamailio involves querying APIs and basing call processing on the API responses. A simple but powerful solution sees the use of http_async_client in collaboration with rtjson; see for example this article from wazo developers: https://wazo-platform.org/blog/kamailio-routing-with-rtjson-and-http-async-client We are discussing whether extending rtjson may be generally helpful for scenarios like this; in that case I'll post an update.

For me this event was a great experience, both from a friendship point of view and as a learning experience. I'm very happy about how the kamailio project is managed, decisions are taken and information is shared.

Here's my report; for obvious reasons this can only be a limited account of what happened during the event, and the information provided is based on my recollection and notes. I'm sure I'm omitting other important material, and I'll be happy to integrate with another post. 

On Kamailio World 2018, part II

In the first part of my brain dump about this year's edition of Kamailio World I focused mainly on testing. Core developers and application designers want to be able to test the behaviour of Kamailio-based architectures with minimal effort and fast feedback.

A different dimension to testing, that I haven't mentioned in my previous post, was related to Fuzz testing. There were two presentations focused on this: Sandro Gauci's (The easiest way to understand who Sandro is: listen on port 5060 on the public Internet and wait a couple of minutes. You'll see a SIP request from a tool called sipvicious (aka friendly-scanner), a penetration testing tool Sandro wrote (and others misuse)) and Henning Westerholt, historical member of the Kamailio community.

Sandro's presentation focused around fuzzing approaches for RTC in general (slides), while Henning was more specifically focused on Kamailio.

Fuzzing is a sophisticated technique to verify the robustness of a software application, by sending input that can vary greatly from the typical or expected usage. The objective is to find weaknesses that can lead to crashes or other malfunctions, so that they can be fixed. Of course testing a server like Kamailio is even trickier than testing an application that can read from a file. It is a fascinating topic.

Kamailio proved to be very robust: Henning reported an average of  about 1 message every 44 million required to make Kamailio misbehave. The video of Henning's presentation is here (by the way, Pascom have done a great work this year too, providing a flawless video streaming and recording. It feels like we are a little spoiled, because we give it for granted and barely notice all the work behind it).

In terms of learning opportunities for architects and administrators of Kamailio-based infrastructure, I found very valuable Daniel's presentations around high-level scripting (with KEMI) to build the routing logic (Video and slides).

Remember that Lua may not be the most popular - apparently - but it's the one estimated to give you performances closest to the native routing language.

Another valuable presentation was around the Least Cost Routing techniques that the Kamailio environment makes available. (Video, and slides). Some solutions use out of the box modules (like lcr, carrierroute, drouting), some are more indirect (pdt, mtree, dialplan, prefix_route), and others are a combination of them. Must-see if you're working in that area.

Another learning goldmine has been Lorenzo Miniero's (author of Janus, a WebRTC conferencing framework (this definition is mine)) lecture about Privacy, Security and Authentication for WebRTC. (Video and slides) Lorenzo does talk fast, but no word is spoken in vain. Worst case, you can watch the video at 0.5 speed (smile). Interesting the case of double encryption for media.

I guess there's enough for a part III in the near future! To be continued. 

On Kamailio World 2018, part I

This was my fifth time in a row attending Kamailio World in Berlin. The weather was warmer and sunnier than usual.

Apart from the obvious focus on Kamailio, as usual the RTC ecosystem was well represented (with Janus, Asterisk, FreeSWITCH, Homer, RTPEngine, and many others).

Attendance from the other side of the Atlantic Ocean gave stronger emphasis to the "World" term in the title.

My personal mission this year was to talk about a framework for testing Kamailio as a tool for developers and maintainers of the project: kamailio-tests. The main concept was that early tests that are not focused on a specific business logic (as we all have in our projects) and can be automated will be beneficial to Kamailio's reliability. We want to defer end-to-end testing to later stages, because they are expensive.

To provide a uniform infrastructure where to run the tests, without requiring permanent test environments, we use Docker for this. This is, of course, not the only possible approach, e.g. you could dynamically spawn VMs, AWS EC2s, etc. But Docker can run on your laptop as well as on a full-fledged CI environment, and this makes it easier to use for the developers.

Please take a look at the slides for more details. The feedback has been great so far, and this proved various points:

1. Conferences for developers are not paid holidays for IT guys, but opportunities for knowledge sharing and collaboration (I would say, in particular if Open Source is in the equation).

2. "Functional" or "component" testing is needed by many, but we haven't a mature solution yet.

3. Docker in RTC is less a fancy technology borrowed by other IT areas and more an everyday tool.

Some have already volunteered to help me improve kamailio-tests, and their point of view will be very useful. More on this project in the future.

Around the topic of testing, in this case not Kamailio itself but more the business logic built around it, there have been interesting insights from Sebastian Damm (sipgate) and Alex Sosic (evosip). 

Sebastian presented an approach that benefits from moving the Kamailio routing logic from the native language to KEMI with Lua (https://github.com/sipgate/lua-kamailio). Alex presented a way to verify the routing logic is going through the expected paths, again with Docker, and sipp.

KEMI is an extension of Kamailio that allows developers to write the routing logic in high level languages, like Lua, Python, JS and others. Anedoctical experience made me think Lua was the most popular, while apparently Python is. For what concerns Lua in the RTC world, I wrote a few notes in February: http://www.giacomovacca.com/2018/02/the-interesting-case-of-lua-in-rtc-world.html

The advantages of working with a high level language are obvious: easier to read and maintain, it's easier to test the functions in isolation, and also easier to involve developers without specific knowledge in Kamailio's routing logic script. They will still need to understand how Kamailio works though, and the underlying protocols, so unless you're doing something extremely basic, it's not a complete abstraction from how Kamailio manages its role as "programmable SIP Proxy".

I have tons of notes from Kamailio World, but if I wait to go through all of them before writing something here, there will be the 2019 edition to talk about. So here's at least a part I.

The interesting case of Lua in RTC world

An interesting pattern that caught my attention is the role that Lua is gaining in the RTC (Real-Time Communications) world.

Lua is a small-footprint programming language, powerful while keeping a simple syntax.

I’ve been using Lua to script dialplan actions for FreeSWITCH since about 2014. It has provided me with a way to define relatively complex logic and speed up the definition of FS’ behaviour.

Delegating this type of logic to a scripting language had several advantages, such as:

• It’s easier to read and understand than native dialplans or native routing logic.
• Makes unit testing of the dialplan possible/easier.
• Allows changing some pieces of logic easily, in many cases preventing expensive reload of modules or restart of applications.

I’ve been using Lua for Kamailio as well. Kamailio is an open source programmable SIP Proxy. In a specific case, some bits of the routing logic required regex processing and was expecting to change often: an ideal case for an external script to do that work.

When the logic changes, it’s sufficient to instruct Kamailio to reload the script, and from that moment on the new requests being processed will use the new logic.

Recent versions of Kamailio though add a framework called KEMI. This opens up new possibilities, and also provides support for many other scripting languages, python being the most popular, JS, Squirrel. Still, Lua appears to provide the fastest implementations (with no observable performance degradation) while others have limitations. Python, as you can imagine, provides a rich set of functions and libraries, but it’s not as performant and the reload mechanism currently has some issues.

Wireshark, a tool to capture and analyse network activity, exposes a useful API for Lua. You can use the API to define your own Wireshark dissector (which you’ll need to install as a plugin). This has performance limitations in comparison with dissectors written in C - and so it’s recommended for prototyping only - but still can solve your problem perfectly. Out of need, I wrote a Wireshark dissector for HEP, a binary protocol used in the Homer environment. Homer is an open source framework for the monitoring and analysis of Real-Time Communications.

Last weekend a new interesting case was presented by Lorenzo Miniero at FOSDEM. The target application was Janus, an open source framework to build WebRTC gateways.

Janus allows to build applications by defining the transport and business logic as plugins, on top of the core that implements the WebRTC stack.

It’s written in C and so far users needed to write plugins with that language. The Janus developers have introduced the possibility to write plugins in Lua.

In his presentation Lorenzo explains also in detail what approach is best to use for a Real-Time application to interact with single threaded language like Lua in an asynchronous context.

Just a funny note: Lua uses a double dash for commenting out a line: '--'. Be careful when you watch diffs in a terminal because a removed comment will start with '- --‘ and may not the easiest thing to interpret (experiences may vary depending on the terminal, of course!).

Continuous Integration and Kamailio

I've presented a workshop at Kamailio World 2016. It focused on tools to help automating the build, deployment and test of Kamailio-based applications using Jenkins, Docker and a few other technologies.

It's been also an opportunity to show a sample usage of the new http_async_client module, designed to perform non-blocking HTTP queries from Kamailio.

The interested reader can find the slides here:

Continuous Integration and Kamailio from Giacomo Vacca

And if you have an hour to spare, here's the full video:




Any feedback or question you may have, please get in touch. I have a post on the event in progress, but there are so many things to highlight that it will require some more time.

Many thanks to Daniel and Elena-Ramona (more info here), event hosts, and Pascom.net for video streaming, recording and editing.

Deploying Kamailio with Puppet

Almost three years ago I started automating the deployment of all server-side applications with Puppet.

One of the key applications in Truphone's RTC platform (and arguably in most of VoIP/WebRTC platforms today) is Kamailio, so with some proper encapsulation it’s been possible to build a Puppet module dedicated to it.

This is now publicly available on PuppetForge. Puppet Forge is a public repository of Puppet modules. We use many of them, and they can be incredibly useful to get you up to speed and solve common problems. You can also fork it on github, if you want.

You can use this module in different ways:

• To quickly deploy a kamailio instance, with default configuration files.
• As the basis for a custom configuration, where you use the official debian packages but with your own configuration details and business logic.

The first approach is very simple. Imagine you start from and empty VM or base docker container: all you have to do is install puppet, download the trulabs-kamailio module from Puppet Forge and apply the test manifest included in the module.

$ apt-get update
$ apt-get install -y puppet
$ puppet module install trulabs-kamailio

  

Dry-run the deployment with ‘noop’ first, and see what Puppet would do:

$ puppet apply -v /etc/puppet/modules/kamailio/tests/init.pp --noop

And then run it for real, by removing ‘noop’:

$ puppet apply -v /etc/puppet/modules/kamailio/tests/init.pp

Now you can verify that Kamailio is running, listening on the default interfaces, and at the latest stable version (this was tested on debian wheezy, and it’s expected to work on Ubuntu Precise and Trusty too).

The second approach (using trulabs-kamailio as a basis for your custom configuration) requires a little bit more work.

Here’s an example, where we want to deploy kamailio as WebSocket proxy. What changes in respect to the test installation is:

1. We need to install also ‘kamailio-websocket-modules’ and ‘kamailio-tls-modules’ (as I’m assuming the WebSockets will be Secure).
2. Take care of the kamailio cfg files directly, and not use the vanilla ones (trulabs-kamailio contains the “sample cfg files” already available with the official debian packages).

A sensible approach is to create a new Puppet module, e.g. ‘kamailio_ws’, and inside its init.pp instantiate the ‘kamailio’ class with these options:

class kamailio_ws () {
  class { '::kamailio':
    service_manage  => true,
    service_enable  => true,
    service_ensure  => 'running',
    manage_repo     => true,
    with_tls        => true,
    with_websockets => true,
    manage_config   => false,
  }
}

(This is similar to the concept of Composition in OOP.)

You can see that ‘manage_config’ is set to false, which is the way of telling the kamailio module not to install the vanilla cfg files, and that those will be managed elsewhere (inside the kamailio_ws module in this case).

To install your cfg files it’s sufficient to add something like this to the init.pp manifest:

  file { '/etc/kamailio/kamailio.cfg':
    ensure => present,
    source => 'puppet:///modules/kamailio_ws/kamailio.cfg',
    notify => Exec['warning-restart-needed'],
  }

In this way you’re declaring that the kamailio.cfg file (to be installed in /etc/kamailio/kamailio.cfg) has to be taken from the files/ folder inside the kamailio_ws module.

As an additional action, every time that file changes, Puppet should notify the user that a restart is necessary (and then leave to the sysadmin to choose the right moment to do so – the Exec resource is just expected to print something on std out).

If you want to restart kamailio immediately after kamailio.cfg changes, then you can use:

  file { '/etc/kamailio/kamailio.cfg':
    ensure => present,
    source => 'puppet:///modules/kamailio_ws/kamailio.cfg',
    notify => Service[‘kamailio’],
  }

Note that if the file doesn't change, for example because you’re just running Puppet to verify the configuration, or updating another file, then Puppet won’t try any action associated to the ‘notify’ keyword.

You can then add all the other files you want to manage (e.g. you may have a defines.cfg file that kamailio.cfg includes) by instantiating other File resources.

A common practice for modules is to have the main class (in this case kamailio_ws) inside the init.pp manifest, and the configuration part (e.g. kamailio.cfg) inside a config.pp manifests included by init.pp.

What do you think? Would you like to see any particular aspect of automating Kamailio configuration treated here? Will you try this module?

Your opinion is welcome.

My presentation at Astricon 2014

Automatic Configuration Management For Kamailio And Asterisk from Giacomo Vacca