I had originally included a section on macros in my introductory Enlive tutorial but in the end thought better of it. My Enlive tutorial turned out to be far more popular than I had imagined it would be. I think the notion of keeping your HTML and your code completely separate struck a nerve with web developers. Now that the dust has settled I realize there is no better introduction to Lisp macros then the topic of HTML templating and I’m writing this more advanced tutorial to explain Lisp macros to the uninitiated as well illustrate how you can use the features of Enlive to build higher level HTML templating abstractions.
I strongly recommend that you go through the first six tutorials if you aren’t already familiar with Clojure. If you are familiar with Clojure you can skim this one as well.
Coding web pages is largely about taking one textual representation and transforming it into another based on some data you are given. Every web developer knows this. Well, it turns out that many Lisps, including Clojure, allow you to do the very same thing with Lisp code itself.
So while Enlive gives you the full power of Clojure to generate your HTML, Clojure gives you the full power of Clojure to generate your code! It’s one of those features of Lisp that is often talked about but rarely well described. This why I think HTML templating is a great introduction to Lisp macros, if you’re a web developer, HTML templating is an activity where the concept of transforming structures is already internalized. If you understand HTML templating you can understand Lisp macros.
Imagine that you’re working with Enlive and you find yourself writing the following snippet:
(defsnippet book-widget “resources/widgets.html” [:.book]
[{:keys [title author publisher date edition ISBN] :as ctxt}]
[:.title] (content title)
[:.author] (content author)
[:.publisher] (content publisher)
[:.date] (content date)
[:.edition] (content edition)
[:.ISBN] (content ISBN))
In my experience most programmers consider this standard fare- an unavoidable bit of tediousness. However a Lisp programmer looks at this and thinks “I know I’m going to be writing stuff like this all the time, I want this to be generated automatically for me.”
What is the pattern here that can be abstracted away? We have a HTML file of widgets handed to us by the designer. The first CSS class name of all the elements in that file can be used for templating in values. If there’s a lot of widgets in that file you’re going to have a lot of widgets in your code that look exactly the same as the above. There’s no reason to type this over and over again. We have a real pattern that can be abstracted away. I am going to show you how to do this.
By the end of this tutorial you will be able to write the following:
(quick-template book-widget "resources/widgets.html" [:.book])
And it will automatically generate the verbose code above.
Now if your favorite programming language is not a competent Lisp there’s not much you can do except approach the templating library authors and ask them to make an enhancement. What would the requested enhancement be?
“Could you pretty please make it so that I can quickly generate a template from the CSS classes that are already in the markup?”
Imagine taking this kind of request to, say, the Django templating library maintainers, they probably wouldn’t take you too seriously and honestly, rightly so. In languages which don’t support macros often the approach is to design a Domain Specific Langauge (DSL) to tackle the problem. The problem with DSLs is that they are generally even more restrictive then the language they are written in. In Lisp writing a DSL is often just writing some macros. The beauty of macros is that you can write macros that use other macros; at no point is there is a decrease in the amount of expressive power available to you even as your syntax becomes simpler and more domain specific.
The power of Lisp is that when we are confronted with what we believe to be an improper level of expression we can just write a macro; we don’t need to wait around for the library maintainers.
This is powerful stuff.
I assume that you have already installed Leiningen. You clone this repo in the usual way. Once you have it checked out you should run the following from your clone at the command line.
lein deps
This will grab all of the dependencies required to go through this tutorial.
Before we get into macros let’s run the following code. Start up a REPL with “lein repl” in the repository:
user=> (load "tutorial/books") nil user=> (in-ns 'tutorial.books) nil tutorial.books=> (start-app) ... output ...
Point your favorite browser at http://localhost:8080/. You should see a list of books. If you remember the previous tutorials the following should shock you. This is how much templating code we have:
(m/quick-snippet book "resources/widgets.html" [:.book]) (html/deftemplate index "resources/index.html" [ctxt] [:.books] (html/content (map book (:books ctxt))))
Intrigued yet? :)
Let’s start up the Clojure REPL in the usual way:
user=> (load "tutorial/syntax") nil user=> (in-ns 'tutorial.syntax) nil
The first thing you need to understand is that when you template code, you are working with lists.
tutorial/syntax=> () ()
This is a list. But it’s also useful to think of it as being something very similar to, say, a DOM node ;)
tutorial/syntax=> (a b) (a b) java.lang.Exception: Unable to resolve symbol: a in this context ...
We’ve tried to evaluate a list which includes two symbols which have never been defined. Thus the error.
tutorial/syntax=> '(a b) (a b)
Interesting eh? No error. This is a quoted list. A quoted listed will not attempt to resolve it’s contents. The following looks remarkably similar.
tutorial/syntax=> `(a b) (tutorial.syntax/a tutorial.syntax/b)
Interesting. ` is the syntax-quote. But what is this good for?
tutorial/syntax=> (let [a 1 b 2] `(~a ~b)) (1 2)
Say hello to your first Lisp “template”. syntax-quote allows us to template in values with ~ (called “unquote”). Again if you have done some HTML templating this should not seem so alien to you.
But this is not a macro. We’re just creating lists here. What we want is a way to transform a piece of code and replace it with the transformation. This is the whole point of defmacro. We’ll get to this in a moment. Let’s show another useful feature before we move on.
tutorial.syntax=> (let [f + v [1 2 3]] `(~f ~@v)) (#<core$_PLUS ...> 1 2 3)
Wow. ~@ lets us splice in a sequence of values! ~@ is called splicing unquote. When transforming code this is a particularly useful operation.
Okay it’s time to talk about defmacro. Be warned the following is completely contrived, it’s only for the purposes of demonstration. We’ll get to the real world macro soon. Before we look at the code let’s run the following:
tutorial.syntax=> (slow-add 1 2) (slow-add 1 2) "Elapsed time: 93.175 msecs" "Elapsed time: 87.948 msecs" "Elapsed time: 82.617 msecs" "Elapsed time: 95.717 msecs" "Elapsed time: 81.954 msecs" nil
Then run the following:
tutorial.syntax=> (fast-add 1 2) (fast-add 1 2) "Elapsed time: 51.211 msecs" "Elapsed time: 35.186 msecs" "Elapsed time: 32.346 msecs" "Elapsed time: 29.503 msecs" "Elapsed time: 38.929 msecs" nil
What’s the difference here? If we look at the code in syntax.clj we see the following:
(defmacro int-add [a b]
`(+ (int ~a) (int ~b)))
(defn slow-add [a b]
(dotimes [_ 5]
(time
(dotimes [_ 5000000]
(+ a b)))))
(defn fast-add [a b]
(dotimes [_ 5]
(time
(dotimes [_ 5000000]
(int-add a b)))))
In order to get the best arithmetic performance out of Clojure, you can provide type-hints to the compiler:
(+ (int 4) (int 5))
With this extra bit of information the JVM can call the correct implementation of + and avoid losing time in reflection. Again this is not something you should concern yourself with, and clearly the timing numbers above show that operations on boxed numbers is more than passable if you’re not doing serious number crunching.
Type the following into the REPL:
tutorial.syntax=> (macroexpand '(int-add 1 2)) (clojure.core/+ (clojure.core/int 1) (clojure.core/int 2))
macroexpand is a convenience that outputs what the code would expand to. Oftentime people complain that macros would somehow decrease the maintainability of a codebase. This is classic Fear, Uncertainity, Doubt (FUD). It’s critical to be able to debug and inspect macros and the tools for doing so are a part of the language.
Note that here we quote the code so that the symbols in the expression don’t get resolved. We’ve converted code into data. You’ve probably heard the expression that in Lisp “code is data”. With a simple quote we’ve converted code into data that we can operate on. defmacro is how we convert transformed data back into working code.
Before your code gets compiled the reader will replace every occurence of int-add with it’s macroexpansion. If you’re familiar with C/C++ preprocessing you should be used to this idea. But if you conclude from this example that Lisp macros really don’t add much beyond that you would be very, very wrong. Macros have full access to the programming language. In the following section we will load markup from an HTML file, analyze it with Enlive and use that information to generate our code.
Just to drive the point home, this int-add example is a trivial, unrealistic, and fairly pointless exercise only for the purposes of illustration. Now let’s solve some real world problems.
Now I won’t lie, as simple as the previous parts were the following is going to be pretty rough going but I hope you stick with it. There is not much code involved in this transformation, but what’s there packs a punch.
Start your REPL if it’s not already up and running and enter the following:
user=> (load "tutorial/books") nil user=> (in-ns 'tutorial.books) nil
We’ve created a macro called quick-snippet which loads and analyzes a widget so it can produce the right code. With quick-snippet instead of writing out the whole template you can write the following:
(m/quick-snippet book "resources/widgets.html" [:.book]))
(In case you’ve forgotten, we often want to include the functionality from a different library. In this case we’re loading the library macroology and aliasing to m. Thus m/quick-snippet).
It’s important to be able to expand macros in order to debug them. Try the following at your REPL:
tutorial.books=> (pprint (macroexpand '(m/quick-snippet book "resources/widgets.html" [:.book]))) (def book (net.cgrand.enlive-html/snippet "resources/widgets.html" [:.book] [ctxt] [:span.title] (net.cgrand.enlive-html/content (:title ctxt)) [:span.author] (net.cgrand.enlive-html/content (:author ctxt)) [:span.publisher] (net.cgrand.enlive-html/content (:publisher ctxt)) [:span.date] (net.cgrand.enlive-html/content (:date ctxt)) [:span.edition] (net.cgrand.enlive-html/content (:edition ctxt)) [:span.ISBN] (net.cgrand.enlive-html/content (:ISBN ctxt)))) nil
Wow. Without changing a line of Enlive, here’s a whole new level of functionality on top of Enlive. pprint is just a part of clojure-contrib and lets us print formatted Clojure code.
Let’s dive into the code. Open the file macroology.clj. It’s a tiny file. This is all the code that it takes to augment Enlive.
(ns tutorial.macroology
(:use [tutorial.utils :only [render]])
(:require [net.cgrand.enlive-html :as html]
[clojure.contrib.str-utils2 :as s]))
(def *child-nodes-with-class* (html/selector [html/root [:* (html/attr? :class)]]))
(defn selector-for-node [{tag :tag, attrs :attrs}]
(let [css-class (s/trim (first (s/split (:class attrs) #" ")))]
`([~(keyword (str (name tag) "." css-class))] (html/content (~(keyword css-class) ~'ctxt)))))
(defmacro quick-snippet [name rsrc selector]
(let [nodes (-> (html/html-resource (eval rsrc))
(html/select (eval `(html/selector ~selector)))
(html/select *child-nodes-with-class*))]
`(html/defsnippet ~name ~rsrc ~selector
[~'ctxt]
~@(reduce concat (map selector-for-node nodes)))))
Let’s break it down. First:
(defn selector-for-node [{tag :tag, attrs :attrs}]
(let [css-class (s/trim (first (s/split (:class attrs) #" ")))]
`([~(keyword (str (name tag) "." css-class))] (html/content (~(keyword css-class) ~'ctxt)))))
This function takes a node and automatically generates the selector/function pair that Enlive templates and snippets use to transform HTML. The first line simply extracts the first CSS class of a node.
You should be familiar with syntax quote now. Note that we can run regular code before we template in the final value.
To get a better sense of what’s going on lets give this function a spin at the REPL:
tutorial.books=> (m/selector-for-node {:tag :div :attrs {:class "foo bar"}})
([:div.foo] (net.cgrand.enlive-html/content (:foo ctxt)))
Since we know how Enlive works it should be quite clear what this function is for. Given an HTML node it creates the matching selector/function pair based on the node’s first CSS class.
The next function is not so simple:
(defmacro quick-snippet [name rsrc selector]
(let [nodes (-> (html/html-resource (eval rsrc))
(html/select (eval `(html/selector ~selector)))
(html/select *child-nodes-with-class*))]
`(html/defsnippet ~name ~rsrc ~selector
[~'ctxt]
~@(reduce concat (map selector-for-node nodes)))))
The first thing we do is use eval. This is generally not necessary for macros and in fact is discouraged. However for what we mean to accomplish there is no other way that doesn’t generate more code than we’d like. We need information from the HTML file in order to generate our code.
IMPORTANT: When a macro is executed you are not handed “values” you’re simply handled whatever symbols or expressions are present in the code. This is a common error to think you are being handed values. Macros manipulate code as data, there are no values yet. But there are ways to get at those values if you really need them, eval is one way to get at them.
So we evaluate rsrc so that we can load an HTML resource. We then also need to evaluate the selector so that we can match only a specific part of the document. Finally we select only the child nodes of that part of the document that actually have CSS classes.
We then emit the defsnippet inserting name, rsrc, and selector unchanged. ~'ctxt is there so that the macro doesn’t try to resolve this symbol, it’s the argument to our macro-defined Enlive template. Finally we take all the matching nodes, map over them with selector-for-node and splice in the selector/fn pairs.
This is definitely a doozy. And programming macros is not a simple task by any means. But clearly the dividends are huge in this case. We now have a way to automatically generate templates for many, many kinds of widgets in one line of code.
I’ve glossed over few things for the sake of simplicity. Once you start writing macros you’ll have to make sure you don’t accidentally capture variables. Clojure has a very simple gensym syntax to avoid this. I’m not going to illustrate them here, but Clojure macros are very, very similar to Common Lisp macros, so I whole-heartedly recommend looking at Paul Graham’s On Lisp which is available for free online for a more in-depth treatment.
On a final note, as fantastic as Lisp macros are they aren’t the answer to everything. For example they are not functions and cannot be used where functions are very useful. The following would not work at all:
(map int-add vector-of-ints-1 vector-of-ints-2)
Still in real world Lisp applications they allow you to avoid boilerplate in way that few other languages can, at least not without an undue amount of work on your part.