On Kamailio World 2018, part II

In the first part of my brain dump about this year's edition of Kamailio World I focused mainly on testing. Core developers and application designers want to be able to test the behaviour of Kamailio-based architectures with minimal effort and fast feedback.

A different dimension to testing, that I haven't mentioned in my previous post, was related to Fuzz testing. There were two presentations focused on this: Sandro Gauci's (The easiest way to understand who Sandro is: listen on port 5060 on the public Internet and wait a couple of minutes. You'll see a SIP request from a tool called sipvicious (aka friendly-scanner), a penetration testing tool Sandro wrote (and others misuse)) and Henning Westerholt, historical member of the Kamailio community.

Sandro's presentation focused around fuzzing approaches for RTC in general (slides), while Henning was more specifically focused on Kamailio.

Fuzzing is a sophisticated technique to verify the robustness of a software application, by sending input that can vary greatly from the typical or expected usage. The objective is to find weaknesses that can lead to crashes or other malfunctions, so that they can be fixed. Of course testing a server like Kamailio is even trickier than testing an application that can read from a file. It is a fascinating topic.

Kamailio proved to be very robust: Henning reported an average of  about 1 message every 44 million required to make Kamailio misbehave. The video of Henning's presentation is here (by the way, Pascom have done a great work this year too, providing a flawless video streaming and recording. It feels like we are a little spoiled, because we give it for granted and barely notice all the work behind it).

In terms of learning opportunities for architects and administrators of Kamailio-based infrastructure, I found very valuable Daniel's presentations around high-level scripting (with KEMI) to build the routing logic (Video and slides).

Remember that Lua may not be the most popular - apparently - but it's the one estimated to give you performances closest to the native routing language.

Another valuable presentation was around the Least Cost Routing techniques that the Kamailio environment makes available. (Video, and slides). Some solutions use out of the box modules (like lcr, carrierroute, drouting), some are more indirect (pdt, mtree, dialplan, prefix_route), and others are a combination of them. Must-see if you're working in that area.

Another learning goldmine has been Lorenzo Miniero's (author of Janus, a WebRTC conferencing framework (this definition is mine)) lecture about Privacy, Security and Authentication for WebRTC. (Video and slides) Lorenzo does talk fast, but no word is spoken in vain. Worst case, you can watch the video at 0.5 speed (smile). Interesting the case of double encryption for media.

I guess there's enough for a part III in the near future! To be continued. 

On Kamailio World 2018, part I

This was my fifth time in a row attending Kamailio World in Berlin. The weather was warmer and sunnier than usual.

Apart from the obvious focus on Kamailio, as usual the RTC ecosystem was well represented (with Janus, Asterisk, FreeSWITCH, Homer, RTPEngine, and many others).

Attendance from the other side of the Atlantic Ocean gave stronger emphasis to the "World" term in the title.

My personal mission this year was to talk about a framework for testing Kamailio as a tool for developers and maintainers of the project: kamailio-tests. The main concept was that early tests that are not focused on a specific business logic (as we all have in our projects) and can be automated will be beneficial to Kamailio's reliability. We want to defer end-to-end testing to later stages, because they are expensive.

To provide a uniform infrastructure where to run the tests, without requiring permanent test environments, we use Docker for this. This is, of course, not the only possible approach, e.g. you could dynamically spawn VMs, AWS EC2s, etc. But Docker can run on your laptop as well as on a full-fledged CI environment, and this makes it easier to use for the developers.

Please take a look at the slides for more details. The feedback has been great so far, and this proved various points:

1. Conferences for developers are not paid holidays for IT guys, but opportunities for knowledge sharing and collaboration (I would say, in particular if Open Source is in the equation).

2. "Functional" or "component" testing is needed by many, but we haven't a mature solution yet.

3. Docker in RTC is less a fancy technology borrowed by other IT areas and more an everyday tool.

Some have already volunteered to help me improve kamailio-tests, and their point of view will be very useful. More on this project in the future.

Around the topic of testing, in this case not Kamailio itself but more the business logic built around it, there have been interesting insights from Sebastian Damm (sipgate) and Alex Sosic (evosip). 

Sebastian presented an approach that benefits from moving the Kamailio routing logic from the native language to KEMI with Lua (https://github.com/sipgate/lua-kamailio). Alex presented a way to verify the routing logic is going through the expected paths, again with Docker, and sipp.

KEMI is an extension of Kamailio that allows developers to write the routing logic in high level languages, like Lua, Python, JS and others. Anedoctical experience made me think Lua was the most popular, while apparently Python is. For what concerns Lua in the RTC world, I wrote a few notes in February: http://www.giacomovacca.com/2018/02/the-interesting-case-of-lua-in-rtc-world.html

The advantages of working with a high level language are obvious: easier to read and maintain, it's easier to test the functions in isolation, and also easier to involve developers without specific knowledge in Kamailio's routing logic script. They will still need to understand how Kamailio works though, and the underlying protocols, so unless you're doing something extremely basic, it's not a complete abstraction from how Kamailio manages its role as "programmable SIP Proxy".

I have tons of notes from Kamailio World, but if I wait to go through all of them before writing something here, there will be the 2019 edition to talk about. So here's at least a part I.

The interesting case of Lua in RTC world

An interesting pattern that caught my attention is the role that Lua is gaining in the RTC (Real-Time Communications) world.

Lua is a small-footprint programming language, powerful while keeping a simple syntax.

I’ve been using Lua to script dialplan actions for FreeSWITCH since about 2014. It has provided me with a way to define relatively complex logic and speed up the definition of FS’ behaviour.

Delegating this type of logic to a scripting language had several advantages, such as:

• It’s easier to read and understand than native dialplans or native routing logic.
• Makes unit testing of the dialplan possible/easier.
• Allows changing some pieces of logic easily, in many cases preventing expensive reload of modules or restart of applications.

I’ve been using Lua for Kamailio as well. Kamailio is an open source programmable SIP Proxy. In a specific case, some bits of the routing logic required regex processing and was expecting to change often: an ideal case for an external script to do that work.

When the logic changes, it’s sufficient to instruct Kamailio to reload the script, and from that moment on the new requests being processed will use the new logic.

Recent versions of Kamailio though add a framework called KEMI. This opens up new possibilities, and also provides support for many other scripting languages, python being the most popular, JS, Squirrel. Still, Lua appears to provide the fastest implementations (with no observable performance degradation) while others have limitations. Python, as you can imagine, provides a rich set of functions and libraries, but it’s not as performant and the reload mechanism currently has some issues.

Wireshark, a tool to capture and analyse network activity, exposes a useful API for Lua. You can use the API to define your own Wireshark dissector (which you’ll need to install as a plugin). This has performance limitations in comparison with dissectors written in C - and so it’s recommended for prototyping only - but still can solve your problem perfectly. Out of need, I wrote a Wireshark dissector for HEP, a binary protocol used in the Homer environment. Homer is an open source framework for the monitoring and analysis of Real-Time Communications.

Last weekend a new interesting case was presented by Lorenzo Miniero at FOSDEM. The target application was Janus, an open source framework to build WebRTC gateways.

Janus allows to build applications by defining the transport and business logic as plugins, on top of the core that implements the WebRTC stack.

It’s written in C and so far users needed to write plugins with that language. The Janus developers have introduced the possibility to write plugins in Lua.

In his presentation Lorenzo explains also in detail what approach is best to use for a Real-Time application to interact with single threaded language like Lua in an asynchronous context.

Just a funny note: Lua uses a double dash for commenting out a line: '--'. Be careful when you watch diffs in a terminal because a removed comment will start with '- --‘ and may not the easiest thing to interpret (experiences may vary depending on the terminal, of course!).