Auto-Load Jinja2 Macros

Last night I was up hacking away on a side project for work. Some of my favorite “quick wins” involve automating and standardizing our operational infrastructure. This project involved generating HAProxy configs from Jinja templates instead of painstakingly managing each server’s (or server group’s) configuration by hand.

In the latest iteration, I was trying to automatically load a common macro in every template without requiring every template to specify it. This is equivalent to adding

{% from 'macros.txt' import macro1, macro2 with context %}

at the top of every template file. But of course I don’t want to repeat myself everywhere.

This article shows two techniques for achieving this goal, depending on what you need — specifically whether you need access to the rendering context from within your macro.

The Global Module Technique

Jinja2 has the concept of a global namespace where you can add functions. This seemed like a great place, except for the small detail that I want to add a macro (defined in a Jinja2 template) instead of a python function. After searching the interwebs and asking on IRC to no avail, I started poking around the Jinja2 source. I finally arrived at a fairly simple (partial) solution.

The Jinja2 Template object has a module property that looks interesting. The docs note

This is used for imports in the template runtime but is also useful if one wants to access exported template variables from the Python layer

Sounds promising, right?! Let’s make our first test case.

We’ll create a Jinja DictLoader, which is handy for these test purposes, and an Environment which we can use for compiling and retrieving Templates in Jinja2. Then we can create a Template from the macro, transform it into a module, and add the individual macros from the module to the Jinja global namespace. It goes like this:

import jinja2
macro = ("{% macro some_macro() %}"
         "templates"
         "{% endmacro %}")
use_macro = "I eat {{ some_macro() }} for breakfast"
loader = jinja2.DictLoader({'template': use_macro})
env = jinja2.Environment(loader=loader)
macro_template = env.from_string(macro)
env.globals['some_macro'] = macro_template.module.some_macro
template = env.get_template('template')
rendered = template.render()
print rendered
# I eat templates for breakfast

Yuss! This works like a charm! You can use this technique to automatically load macros with a few caveats which we’ll discuss below.

Although this technique worked for my first couple of test cases, I was actually writing a file-loading macro that looks like this:

{% macro include_backend(component) -%}
  {% set template = 'backends/' + component + '.txt' -%}
  {% include [template, 'backends/default.txt'] %}
{%- endmacro %}

(Don’t worry about the particulars here. I’m still iterating on this project but hope to be able to share more of it with you soon-ish! :))

Most of the backend templates, including the default one, rely on data passed into the Jinja rendering context. For example, the default backend looks something like this:

backend {{ component }}
        balance roundrobin
        option httpchk GET /healthcheck
        {%- for server in vips[component].servers %}
        server {{ server['name'] }} {{ server.details['private ip'] }}:{{ vips[component].backend_port }} check
        {%- endfor %}

Notice that vips variable? That’s not passed from the macro like component is. Its passed into the rendering context as in

print get_template('some_template').render(vips=vips)

We can modify our small example to test this type of usage. We’ll use a variable, food, from the rendering context in our macro.

import jinja2
macro = ("{% macro some_macro() %}"
         "{{ food }}"
         "{% endmacro %}")
use_macro = "I eat {{ some_macro() }} for breakfast"
loader = jinja2.DictLoader({'template': use_macro})
env = jinja2.Environment(loader=loader)
env.globals['some_macro'] = env.from_string(macro).module.some_macro
template = env.get_template('template')
rendered = template.render(food='cheerios')
print rendered
# I eat  for breakfast

Uh-oh! Now we see the problem. Because we added the macros as globals before the other templates were rendered, the context wasn’t yet available for inclusion and can’t be rendered. Shoot.

The caveat mentioned above is this: we can automatically load macros so long as they don’t need access to anything from the rendering context. Said another way, we’ve effectively left out the with context from the Jinja2 import.

{% from 'macros.txt' import macro1, macro2 with context %}

Sooo, where’s this leave us? I had to change tack.

The Prepending Loader Technique

If you recall, the original goal was explained as automatically adding an import line before every template. What if we try taking this approach more literally? After failing to find a way to support context in globals, I started reading the code for Jinja’s loaders. Is there a way to inject the macro’s source code before the template code is compiled? Luckily, this turned out to be pretty straightforward using the Loader API!

The Loader API provides a get_source method for implementers to override. In this case, we want to use the Composite Pattern to delegate the actual source lookup for both the template and the macro (or prepend_template as its called here) to an underlying loader. For example, we may want to prepend a template to one loaded from the filesystem or a dictionary.

The logic is basically to call get_source for both the main template and the prepended template. This’ll give us the source code for both which can simply concatenate together!

The method requires returning a filename; we can use the name of the main template file. There is a known issue here that the line numbers in exceptions will be offset by the number of lines in the prepended macro. The API also requires returning a callable which is used to instruct the built-in cache whether the source needs invalidated; the returned source code is up to date only if both the prepended source and main source are up to date.

class PrependingLoader(BaseLoader):
 
    def __init__(self, delegate, prepend_template):
        self.delegate = delegate
        self.prepend_template = prepend_template
 
    def get_source(self, environment, template):
        prepend_source, _, prepend_uptodate = self.delegate.get_source(environment, self.prepend_template)
        main_source, main_filename, main_uptodate = self.delegate.get_source(environment, template)
        uptodate = lambda: prepend_uptodate() and main_uptodate()
        return prepend_source + main_source, main_filename, uptodate
 
    def list_templates(self):
        return self.delegate.list_templates()

Let’s try this out with our example.

macro = ("{% macro some_macro() %}"
         "{{ food }}"
         "{% endmacro %}")
use_macro = "I eat {{ some_macro() }} for breakfast"
base_loader = jinja2.DictLoader({'template': use_macro, 'macro': macro})
loader = PrependingLoader(base_loader, 'macro')
env = jinja2.Environment(loader=loader)
template = env.get_template('template')
rendered = template.render(food="cheerios")
print rendered
# I eat cheerios for breakfast

That’s more like it!

One thing to be aware is that directly including the macro source this way can cause spacing issues if you have newlines at the end of the file or other whitespace controls in place. If this becomes an issue, you can either adjust the whitespace or include the import line instead of the actual macros.

So there we have two ways to automatically load macros in Jinja2 templating system!