(iterate think thoughts)

Introducing Selmer

30 Jul, 2013

Introducing Selmer

Rationale

There are a number of templating engines available in Clojure. Some of the popular ones include Hiccup, Enlive, Laser, Stencil, mustache.clj and Clabango.

As I've mentioned previously, my personal preference is for Clabango syntax. In my opinion it provides the right balance between simplicity and flexibility. Being modeled on Django template syntax it's also very accessible to those who are new to Clojure web development.

However, one major downside to Clabango is that it's slow. On TechEmpower fortunes benchmark Luminus is crawling behind the Compojure results. Yes, you read that right, it's nearly 20 times slower for Clabango to render the results. The difference being that the Compojure benchmark is using Hiccup for rendering the results while Luminus is using Clabango.

The core problem is that Clabango always parses the source files when rendering a template. This is very expensive as it involves disk access and scanning each character in the source file each time a page is served. Dan states that performance has not been a priority.

On top of that, some of the existing behaviours put limitations on how much the performance can ultimately be improved. For example, the child templates aren't required to put their content inside blocks. Clabango parses the templates and creates the AST that's then evaluated. This means that you can put blocks inside the if tags and decide at runtime whether they will be included. If inheritance resolution is pushed to compile time this becomes a problem.

After having some discussions with bitemyapp and ceaserbp we decided that it would be worth writing a fresh impelementation with pefromance as its primary goal. Another reason is that I would like to be able to ensure that the templating engine in Luminus isn't a compromise between speed and convenience. Owning the implementation is the best way to achieve that.

Enter Selmer

All this resulted in Selmer named after the guitar favored by Django Reinhardt whom in turn Django is named after. Selmer aims to be a near drop in replacement for Clabango. The current version is already quite fast keeping pace with Stencil which is one of the faster engines around.

In order to minimize the work that's done at runtime Selmer splits the process into three distinct steps. These steps include preprocessing, compilation and rendering.

First, Selmer will resolve the inheritance hierarchy and generate the definitive template source to be compiled. The extends and include tags will be handled at this time.

The compilation step then produces a vector of text nodes and runtime transformer functions.

The renderer uses these compiled templates to generate its output. The text gets rendered without further manipulation while the transformers use the context map to generate their output at runtime.

Using this approach we minimize the amount of logic that needs to be executed during each request as well as avoiding any disk access in the process.

In order not to have to restart the application when the source templates are changed the renderer checks the last updated timestamp of the template. When the timestamp is changed a recompile is triggered.

Performance Tricks

To our chagrin the first run of the parser ran no better than Clabango. This was rather disappointing considering we took pains to be mindful of the performance issues. However, this mystery was quickly solved by profiling the parser.

Sure enough majority of time was spent in reflection calls. One major problem was that the renderer had to check whether each node was text or a function:


(defn render [template params]  
  (let [buf (StringBuilder.)]
    (doseq [element template] 
      (.append buf (if (string? element) element (element params))))
    (.toString buf)))

Protocols offer an elegant solution to this problem. With their help we can move this work to compile time as follows:


(defprotocol INode
  (render-node [this context-map] "Renders the context"))

(deftype FunctionNode [handler]
  INode
  (render-node ^String [this context-map]
    (handler context-map)))

(deftype TextNode [text]
  INode
  (render-node ^String [this context-map]
    text))

Now our parser can happily run along and call render-node on each element:


(defn render-template [template context-map]
  """ vector of ^selmer.node.INodes and a context map."""
  (let [buf (StringBuilder.)]
    (doseq [^selmer.node.INode element template]
        (if-let [value (.render-node element context-map)]
          (.append buf value)))
    (.toString buf)))

With this change and a few type annotations the performance improved dramatically. Running clojure-template-benchmarks the results are comparable to Stencil. Here are the results from benchmarking on my machine:

Clabango

  • Simple Data Injection
    • Execution time mean : 657.530826 µs
    • Execution time std-deviation : 2.118301 µs
  • Small List (50 items)
    • Execution time mean : 2.446739 ms
    • Execution time std-deviation : 17.448003 µs
  • Big List (1000 items)
    • Execution time mean : 28.230365 ms
    • Execution time std-deviation : 173.518425 µs

Selmer

  • Simple Data Injection
    • Execution time mean : 42.444958 µs
    • Execution time std-deviation : 235.652171 ns
  • Small List (50 items)
    • Execution time mean : 209.158509 µs
    • Execution time std-deviation : 4.045131 µs
  • Big List (1000 items)
    • Execution time mean : 3.223797 ms
    • Execution time std-deviation : 55.511322 µs

Stencil

  • Simple Data Injection
    • Execution time mean : 92.317566 µs
    • Execution time std-deviation : 213.253353 ns
  • Small List (50 items)
    • Execution time mean : 290.403204 µs
    • Execution time std-deviation : 1.801479 µs
  • Big List (1000 items)
    • Execution time mean : 1.223634 ms
    • Execution time std-deviation : 4.264979 µs

As you can see Selmer is showing a large improvement over Clabango and has no trouble keeping up with Stencil.

Obviously, this benchmark is fairly simplistic so you can take it with a grain of salt. If anybody would like to put together a more comprehensive suite that would be great. :)

Current status

The library implements all the functionality offered by Clabango and passes the Clabango test sutie. There are a few minor deviations, but overall it should work as a drop in replacement without the need to change your existing HTML templates.

We also have a few new features such as the Django {{block.super}} tag support and ability to use filters in if statements. In Selmer you can write things like:


(selmer.filters/add-filter! :empty? empty?)

(render 
  "{% if files|empty? %}
   no files available 
   {% else %} 
       {% for file in files %}{{file}}{% endfor %} 
   {% endif %}"
  {:files []})

Switching to Selmer involves swapping the [clabango "0.5"] dependency for [selmer "0.5.3"] and referencing selmer.parser instead of clabango.parser. Selmer provides the same API for rendering templates using the selmer.parser/render and selmer.parser/render-file functions.

One major area of difference is in how custom tags and filters are defined. Defining a filter is done by calling selmer.filters/add-filter! with the id of the filter and the filter function:


(use 'selmer.filters)

(add-filter! :embiginate #(.toUpperCase %))

(render "{{shout|embiginate}}" {:shout "hello"})
=>"HELLO"

Defining custom tags is equally simple using the selmer.parser/add-tag! macro:


(use 'selmer.parser)

(add-tag! :foo
  (fn [args context-map]
    (str "foo " (first args))))

(render "{% foo quux %} {% foo baz %}" {})
=>"foo quux foo baz"

tags can also contain content and intermediate tags:


(add-tag! :foo
  (fn [args context-map content]
    (str content))
  :bar :endfoo)

(render "{% foo %} some text {% bar %} some more text {% endfoo %}" {})
=>"{:foo {:args nil, :content \" some text \"}, :bar {:args nil, :content \" some more text \"}}"

Selmer also supports overriding the default tag characters using :tag-open, :tag-close, :filter-open, :filter-close and :tag-second keys:


(render "[% for ele in foo %]<<[{ele}]>>[%endfor%]"
                 {:foo [1 2 3]}
                 {:tag-open \[
                  :tag-close \]})

This makes it much easier to use it in conjunction with client-side frameworks such as AngularJs.

One limitation Selmer has is the way it handles inheritance. Since the inheritance block hierarchy is compiled before the parsing step, any content in child templates must be encapsulated in block tags. Free-floating tags and text will simply be ignored by the parser. This is in line with Django behavior.

So there you have it. If you like Django template syntax or just want a fast templating engine then give Selmer a try.

As it is a new project there may be bugs and oddities so don't hesitate to open an issue on the project page if you find any. So far I haven't found any problems in switching my application from Clabango to Selmer and the test coverage is fairly extensive at this point.



tags clojureluminusselmer

comments


30 Jul, 2013 - Manuel

I've just swapped Clabango with Selmer in a small application of mine and it worked flawlessly (and faster). Great work!

30 Jul, 2013 - bhauer

This looks fantastic, Yogthos! I am a tad envious of Clojure developers and would like to use this on my Java projects. I've had mixed feelings since we adopted Mustache as our default template language. The strictness of Mustache at times drives me a little nuts. I long for a filters feature like Selmer's.

I'd really like to see this incorporated into the Luminus test for our Round 7. Do you think you'll be able to submit a pull request to include Selmer in the next week or two (we don't have an ETA yet, so there is not a big rush)?

30 Jul, 2013 - Yogthos

Good news is you can easily use Selmer from Java, I made a wrapper here, which I'll push out to Clojars tonight. I'll update the Luminus test to use Selmer for round 7. Very excited to see the results. :)

If you do end up trying it from Java, let me know how well it works performance wise.

30 Jul, 2013 - anonymous

Aside: Looks like you accidentally put in a Python-style "triple-quote" in one of your example code blocks above.

30 Jul, 2013 - Yogthos

Apparently that's an actual thing, bitemyapp added these in the source, possibly for Marginalia?

30 Jul, 2013 - bhauer

Ooooh, a Java wrapper. I will check this out! Any chance of getting the remainder of the API exposed at some point down the line?

Thanks again for this and for making performance a first-class priority. I love finding components developed with that mindset.

30 Jul, 2013 - bitemyapp

Yeah the triple quote was me. I'm still cleaning up and adding to the docs for Marginalia (for which there is a link in the README).

30 Jul, 2013 - Yogthos

@bhauer, I could look at exposing the API if that's something that would be useful. I've also realized the wrapper needs to be a bit smarter to handle nested structures.

30 Jul, 2013 - Yogthos

@bhauer the wrapper should work with nested elements now

30 Jul, 2013 - bhauer

Awesome.

Django-style templates developed with a performance and quality-oriented focus, that I can use from Java code? Yes please! But I should clarify that while a full Java API to Selmer would be fun, by no means do I want to impose.

30 Jul, 2013 - Yogthos

Let me know how the Java API works out and if it's useful and you get decent performance with it I could certainly look into exposing the rest of the API. I would expect that the first invocation would be quite slow as it has to spin up the Clojure runtime, but after that it should be fast.

31 Jul, 2013 - Yogthos

@bhauer

I updated the Java API to support filters and inline tags and it's available on clojars repo. Let me know if you get a chance to play with it.

31 Jul, 2013 - bhauer

Excellent! I will check it out. Thanks so much for building this!

01 Aug, 2013 - anonymous

Looks like deftag has already been changed to add-tag! ? Also, where are the tags defined in the source? Any more comprehensive examples?

01 Aug, 2013 - Yogthos

Yes, I decided to change it to be inline with add-filter! before it gets too much usage. The tags are defined in the selmer.tags namespace. The basic idea is that the tag handler will receive the args supplied to the tag, the context map, and optionally the content if it's a block tag.

01 Aug, 2013 - anonymous

Gotcha. I guess I didn't express myself well enough and didn't look hard enough: I was looking to see where the if/for blocks were added since I didn't see a call to add-tag! for them. Found them being added to the expr-tags atom in parser.clj.

I'm excited that you're actively working on this. Is there any desire for any kind of filter/tag parity with Django? Are you accepting filter/tag contributions?

01 Aug, 2013 - Yogthos

We'd like to get as much coverage as possible for filters and tags and contributions are absolutely accepted. :)

New filters can be added here and we're still getting around putting all the existing ones in the docs. There are a few more than what's listed on the project page.

09 Nov, 2013 - anonymous

Swapped in Selmer and replaced fleet. Worked flawlessly and completed in about 4 hours. This is exactly what I was looking for. Great work.




help

*italics*italics
**bold**bold
~~foo~~strikethrough
[link](http://http://example.net/)link
super^scriptsuperscript
>quoted text
4 spaces indented code4 spaces indented code

preview

submit