NGINX Dynamic Modules: How They Work

This post is adapted from a presentation delivered at nginx.conf 2016 by NGINX, Inc. developer Maxim Dounin. You can view a recording of the presentation on YouTube.

	Introduction
0:17	The Basics
1:42	Are Dynamic Modules Really Needed?
3:00	Main Goals of Dynamic Modules
3:32	Mini How-To
4:07	Implementation Details
4:45	Initial Trivial Solution
5:29	Subtle Parts
5:46	Module Config Scripts
7:28	List of Modules: ngx_modules
8:34	Module Indexes
9:48	Order of Modules
11:01	Signature Checking
12:36	Version Checking
13:14	Filter Chains
16:00	What We Ended Up With
16:41	Conversion
17:15	ngx_count_modules()
17:41	ngx_modules to cycle->modules
17:57	Config Script
18:51	Compatible Config Version
19:20	Parameters of auto/module
20:39	Complex Modules
21:06	How Loading Works
21:20	The load_module Directive
21:35	Configuration Reload
22:36	How to Upgrade a Module?
23:02	Further Improvements
23:14	Compatibility Between Builds
24:27	Compatibility with NGINX Plus
24:54	Thank You

Introduction

Maxim Dounin: My name is Maxim Dounin, and I am an NGINX developer. Today, I would like to talk about dynamic modules, which we introduced last winter.

0:17 The Basics

First, let’s review some basics about modules in general, and dynamic modules in particular.

As you probably know, modules in NGINX have existed since its inception. In fact, NGINX is actually a collection of modules. Even very basic functions like HTTP, or serving static files inside HTTP – these basic things are modules.

You can extend NGINX by adding your own modules. The modular architecture allows you to easily modify NGINX. I think it’s a very important part of NGINX. It’s what makes it so good, and so successful.

Last winter, we introduced dynamic modules. You can now load modules into a compiled NGINX binary, without recompiling NGINX itself. Dynamic modules were the result of a joint effort by myself and Ruslan Ermilov.

1:42 Are Dynamic Modules Really Needed?

Why are dynamic modules needed?

Normal static modules have existed for ages, and they work fine. Moreover, NGINX has a very useful and cool feature. You can change the binary in place, without dropping any requests. This feature is normally used for upgrades, but you can also recompile NGINX with a different set of modules and just switch to a different binary. Changing sets of modules is not a problem.

Packaging is a problem, though. It’s quite difficult to package NGINX with different modules, especially when there are external dependencies. You may have to ship NGINX without modules, or you may have to ship single‑loaded packages. It’s awkward, and doesn’t always work very well.

3:00 Main Goals of Dynamic Modules

A main goal of creating dynamic modules was to simplify package management. Another goal of dynamic modules was to simplify debugging.

Users can just comment out third‑party modules to check whether a bug they are experiencing was introduced by some third‑party module that they compiled in accidentally.

3:32 Mini How‑To

Here’s a short how‑to for using dynamic modules.

You can compile NGINX with several standard modules dynamically loaded, by using the =dynamic suffix. You can compile third‑party modules dynamically by using the --add-dynamic-module option to configure. And then in NGINX configuration, you use the load_module directive to load a module.

4:07 Implementation Details

Let’s now take a look at some implementation details. We’ll look at some of the more interesting things we encountered while implementing dynamic modules.

First of all, we decided that implementation should be simple.

There are a lot of high‑value third‑party modules and we can’t just force all developers to change their modules and make life hard for them. So, it was important to minimize required changes to modules.

4:45 Initial Trivial Solution

Our initial trivial solution was to just load all modules that exist in a certain directory when NGINX starts up. This requires no changes to module code, but it still required changes to module config scripts.

So, then we implemented something a bit more complex with the load_module directive in the configuration. Now you could load and unload modules from the configuration, and you didn’t need to rename files, or move files around.

5:29 Subtle Parts

These are some problems we were working on while implementing dynamic loading.

Let’s look at them in order.

5:46 Module Config Scripts

First, module config scripts. This was probably the most complex part from an implementation point of view.

There are many existing modules and compatibility is very important. You don’t want to break things. So, we had to provide compatibility with all modules.

Module config scripts are essentially shell code. In these config scripts, modules just set some global variables. NGINX configuration then recognizes these variables and uses them. We’ve left everything as is, so the same variables are still used by NGINX configuration.

We’ve introduced the auto/module script to assign these variables. So, now there’s an interface to assign these variables for static compilation, and it also knows how to assign variables for dynamic compilation. By converting your module config script to use the auto/module script, you make it clearer and also add dynamic compilation.

A bit later we’ll take a look at how to do this for a sample module. It’s quite trivial.

7:28 List of Modules: ngx_modules

The second problem regards a list of modules inside NGINX. It’s set with a global variable, ngx_modules.

The problem is that each configuration needs its own list of modules. Therefore, we have to replace the global variable ngx_modules with cycle->modules inside a list of modules, inside a particular configuration.

The ngx_modules global variable is still there. If a module doesn’t need to support dynamic loading, it can continue to use ngx_modules as is, but if a module is a complex one that needs to be converted to understand dynamic loading, it needs to be changed to use cycle->modules instead.

8:34 Module Indexes

Another problem is module indexes.

Each module is described by a global structure, ngx_module_t. This means that indexes that are stored within the structure cannot be changed. If the index is assigned, you can’t change it later.

Moreover, there must be no conflicts with previous configurations. If you’ve loaded a module in a previous configuration, and in the new configuration you haven’t loaded this module yet, you still can’t reuse the index of this module, because this module might be loaded later.

This is something the NGINX core takes care of while loading modules, and we’ve introduced the ngx_count_modules() function to assign ctx_indexes for complex modules.

9:48 Order of Modules

The next issue is the order of modules.

The order is important in some cases, especially if you write a filter. For example, gzip must happen before writing the data to a client. Both these tasks are done by filters, so you have to specify the appropriate order for the filters to run in.

During static compilation, this is done via the configuration order. But when you’re loading a module, there is no [predetermined] order, so we have to store a configuration-imposed order inside the module itself. Then, when NGINX loads the module, it takes a look at the order recorded in the module, and puts the module in the appropriate place in the module list.

11:01 Signature Checking

The next problem is that NGINX uses conditional compilation on various structures.

NGINX doesn’t compile some fields of the structures which are not used in a particular compilation. And there’s no API layer between these structures and the modules. The structures are essentially a part of our API. So, if you try to load a module that was compiled with a different set of configuration options, it will likely segfault because it assumes a different structural layout.

To prevent this, we’ve introduced signature checking.

NGINX checks if the module was compiled with the same options. If it was compiled with a different set of options, NGINX refuses to load the module. We have managed to implement this in an existing macro, so no changes to modules are required at all. It just seems to work like magic.

12:36 Version Checking

A similar issue is version checking.

The same problem with structures happens when you switch from version to version: segfaults happen when loading a module compiled with a different NGINX version.

Therefore, while loading modules, NGINX checks the version the module was compiled with.

13:14 Filter Chains

The last problem is filter chains.

Filter chains are global like the list of modules. They’re initialized during post‑configuration callbacks, so most configuration problems (like syntax errors, etc.) happen before this, and in that case the filter chains are not initialized.

But some errors can still happen after the filter chains are constructed. If a configuration fails after the filter chains are built, NGINX rolls back to the old configuration, and unloads the modules that were just loaded. As a result, the NGINX master process is in an inconsistent state. It does not have a module loaded, but the module is linked in the filter chain.

This is not fatal, as a master process doesn’t call filters itself. Normally, it will just change the configuration, build another filter chain, and will spawn new workers with correct configuration and filter chains.

But there’s one small case when this can be a problem. If a worker process dies, NGINX tries to respawn the worker with the current master state. This state is inconsistent, so the worker dies too because it is referencing a module that’s not loaded. This problem requires reimplementing filter chains in a way that is specific to each configuration.

Because this happens only in some very specific cases, we’ve left things like this; fixing it would require too many changes to modules. If you do have a good solution, please let us know.

16:00 What We Ended Up With

What we ended up with is the auto/module configure script, the ngx_modules global variable replaced with cycle->modules, the ngx_count_modules() function to assign context indexes for complex modules, and the load_module directive.

All previous modules continue to work fine, and most modules can be easily converted to dynamic loading with just a few simple changes.

16:41 Conversion

Let’s look at what’s needed to convert a module for dynamic loading. There are actually just three simple steps for all modules. And in most cases, only one is required.

In all cases, you have to rewrite your config script to use auto/module. In complex cases, you also have to use ngx_count_modules() and you have to replace ngx_modules with cycle->modules.

17:15 ngx_count_modules()

Let’s start with the most complex case.

Here’s an example from NGINX itself: how to count modules. It basically replaces a loop in the code with a function call.

17:41 ngx_modules to cycle->modules

And here is how to replace ngx_modules with cycle->modules. Again, we just replaced the global references with cycle->modules.

17:57 Config Script

In most cases, you have to rewrite your config script. In most cases, this is actually the only thing you have to do.

Here is an example for a simple module. The original version uses global variables. It specifies a domain, HTTP modules, and add‑on sources.

The new version uses the auto/module script. It basically does the same thing as the original version but uses the auto/module script, which allows it to selectively specify the module type, name, included directories, dependencies, sources, and libraries.

That’s all you need to do in this case.

18:51 Compatible Config Version

In some cases, you may want to provide compatibility with previous versions of NGINX.

This can be easily done by testing for the ngx_module_link variable. If it’s set, you use the new version. If it’s not set, you use global variables as in previous versions.

19:20 Parameters of auto/module

Here’s a short list of parameters for the auto/module script. You can specify module type, usually HTTP or HTTP_AUX_FILTER, the name of the module, include directories, dependencies to check when building, sources, and libraries.

Some special values are recognized for libraries NGINX knows how to build. You can also just provide a link of flags like NGINX’s own test C++ module.

There are a couple more variables I haven’t showed yet. One is ngx_module_link, which specifies linking types; do set it directly. NGINX expects to set it itself before calling the config script.

The order of modules to be used is normally set based on module type. You don’t need to worry about it unless you are doing something specific.

20:39 Complex Modules

It is also possible to compile many modules into a single, shared object file.

You specify multiple module names and multiple sources with the --add-dynamic-module parameter to the configure script [for dynamic modules] or with the --add-module parameter [for the complete NGINX binary].

21:06 How Loading Works

Now, let’s take a closer look at how module loading works, how it happens, and the results.

21:20 The load_module Directive

The load_module directive calls dlopen(). Later, when the configuration is freed, dlclose() is called.

21:35 Configuration Reload

What happens when you reload the configuration?

The new configuration calls dlopen(). The previous configuration is freed, and it calls dlclose(). If the module is in both the previous and the new configuration, it remains loaded.

This is a good thing, as it means you can even remove the module file itself, and NGINX doesn’t notice. You can still reload the configuration when needed.

It allows you to replace the NGINX binary and NGINX modules in any order. You can still reload the configuration if needed.

22:36 How to Upgrade a Module?

If you want to upgrade a module without upgrading NGINX, you actually do the same thing you do when upgrading NGINX itself.

You use the USR2 signal, or on most systems, service nginx upgrade which sends the USR2 signal for you.

23:02 Further Improvements

The last topic I wanted to talk about today is further improvements. These are things we are still working on and would like to share with you.

23:14 Compatibility Between Builds

First, compatibility between builds.

This is actually related to signature checking. It’s a pain to compile dynamic modules if you do it outside of NGINX itself. You have to use the same configure options, or signature checking prevents you from loading the module.

We are currently working to make this easier. With a special option to configure, perhaps something like --with-compat, NGINX will just compile in all the structure fields it knows about, so that modules will be compatible even with different configure options. You will be able to load them even if you compile NGINX and the module with a different set of configure options.

[Editor – The --with-compat option was introduced in NGINX Plus R11 and NGINX 1.11.5.]

24:27 Compatibility with NGINX Plus

Closely related to that is compatibility with NGINX Plus.

Right now, it is not possible to load a module compiled for NGINX into NGINX Plus, because of signature checking, and the incompatibilities between structures.

We are working to make that possible.

[Editor – Binary compatibility for NGINX Plus was introduced in NGINX Plus R11 and NGINX 1.11.5.]

24:54 Thank You

Thank you for your attention. If you have any further questions about dynamic modules, feel free to contact me.