(iterate think thoughts)

Managing Complexity with Mycelium

Wed, 25 Feb 2026 00:00:00 +0000

Software architecture is at root a creature of human frailty. The sacred cow of clean code and the holy grail of design patterns are understood to be, at least in practical terms, little more than tricks to help people keep their sanity along the way. Human cognitive capacity is strictly limited, and we're still figuring out ways to reliably build machines significantly more complex than can be held in a single mind. Current attempts to offload coding tasks to language models are hitting the same wall. These models can be brilliant, but only up to a point. They’ll effortlessly compose a flawless function, but when challenged to manage a project with a thousand such moving parts, they quickly lose the plot. This problem is commonly known as context rot, but it might equally well be called a coding architecture failure.

If one hands an LLM a big pile of mutable state and loosely defined relationships, a solution that doesn’t actually work will be hallucinated inevitably, and when it does work, it will do so largely by accident. This comes about because there are just too many moving parts, and the ground truth cannot be kept track of. Humans are known to have exactly the same problem. Enormous amounts of time are spent chasing down bugs that exist because some distant part of the app decided to tweak a variable it didn’t own. Shared mutable state quickly leads to an overwhelming information flow, generating invisible threads between components that make it nearly impossible to know the scope of the change being made. Seemingly innocuous code changes end up mutating state in unintended ways that lead to unexpected consequences.

We solve difficult problems by dividing them into smaller, more manageable ones, and then composing them together. Successful architecture requires separable components, each with a function that is easy to grasp, and creating interfaces between them to abstract over their internal details and present to the outside world only their functional aspects. We must discover the boundaries between components, separate external effects from internal implementation details, and arrange for each component to control its own context. Then, and only then, can we be sure that we will always be working in a clear context that we can fit in our heads.

We naturally long for layers of organization. Hierarchies permit us to construct separate, self-contained units which can then be connected together to make larger structures. It's a powerful kind of architecture, one that facilitates writing large projects by abstracting over the complexity of the constituent parts. In working on a given component, we have to know all its details. But in using it, we only need to know what it does, which is entirely reflected in its API surface. The internal complexity is encapsulated within the API boundary. These are building blocks that give us a stable base upon which to build higher-level abstractions. Several such components can be assembled into a bigger block, where the connections between subcomponents become its internal complexity. The composite component becomes a new layer, providing its own API, which can be used by still higher-level abstractions. If what I'm describing sounds familiar, it is because that's exactly the way in which software libraries work. A library is simply a model of a class of problems, and when we encounter these types of problems, we can use it as an off-the-shelf building block.

A complex system has to be resilient and adaptable. But if every component is hooked directly into every other, there’s no way you can anticipate what a change in any one place will do. There are no boundaries to stop the chaos. Anything that works at scale relies on stable subassemblies. Herbert Simon described it long ago in a parable of two watchmakers. The first built his watches one piece at a time. He had to keep the entire complexity of the watch in his mind as he did his work, for any interruption would undo his progress and send the component pieces clattering to the floor. He worked on a flat plane of complexity, having to think through the workings of the entire watch just to insert a single gear. The other watchmaker did well because he first constructed small, stable modules which he was able to click together. So if he was interrupted, only a small amount of work was lost. He never had to think through more than a few components at a time. The system was resilient because it was built as a hierarchy of subunits, each able to stand on its own merit, and so the watchmaker had to consider a small context rather than the stupefying complexity of the whole.

Practically any large system can so be divided into smaller, nested subsystems, which communicate with one another as they go about their business. The intimate workings of other subsystems need hardly be known, which permits these modules to form. Individual components are not encumbered by what’s going on at other places where events are transpiring at different paces and according to a different set of circumstances. This is how hierarchies help in managing the complexity of a system. Each subsystem can evolve on its own. A malfunction in one region can be quarantined; it need not bring down the entire enterprise. A useful way to think of hierarchies is to treat them as connective tissue between the various subsystems of the program, as a principal means of control, providing the architecture and the infrastructure that coordinates the internal functioning of the individual parts of the system.

In contrast, in software development, practitioners are often confronted by the opposite scenario. Large software projects degenerate into a tangled web in which every function depends on a global state or a shared database connection. Humans have difficulty working with such systems because they can’t reason about their pieces independently. If a feature requires more information than can be kept in one’s head at once, guesses and assumptions begin to proliferate.

Just as humans need clear boundaries to maintain sanity, coding agents are even more vulnerable to cognitive saturation. There’s no intuition in a large language model that can tune out the noise. Everything is placed into its context window. If a piece of code is a labyrinth of obscure dependencies, the agent has to parse through it all to make a single modification. Eventually, the context becomes so cluttered that the purpose of the task gets lost in the entropy. The model starts hallucinating because of its inability to distinguish between the logic it’s supposed to refactor and the three hundred other things it’s presented with.

Where We Are At

To understand how we might solve this for both humans and machines, let's first examine the tools we've already developed for managing complexity. The software industry has spent decades working out how to decompose code into manageable pieces. A very powerful set of tools for this purpose is already available. At the inter-process scale, there are microservices, which provide a physical boundary between programs. And at the intra-program level, there are techniques such as message passing, pure functions, and immutable data.

The functional style is particularly good at taming global state. Here, functions are the smallest building blocks, and pure functions can be considered on their own. The idea here is to build systems from separate, single-purpose parts, by using isolation, composition, and clear contracts inside the application's own logic. Code becomes a pipeline focused on the data flow. A program takes one piece of data as input, and produces another as output, pumping it through a sequence of pure functions. Since data is immutable, it is transparent and inert, having no hidden states or secret side effects. This is how stable contracts are formed. Once the transformations have been specified, the interface is a contract you can trust.

When you design applications in this way, the overall architecture looks like a network of railway lines, with the input data package needing to get from point A to point B. The package might pass through many different stations on the way to its destination, each one inspecting the package to decide where to route it next. An HTTP handler takes the payload, parses the request, determines the content type, and forwards it on toward an authentication handler. The authentication handler might inspect permissions in payload metadata to determine where to send it next, and so on. Eventually, the package arrives at its intended destination where the data gets serialized, stored, or presented to the user.

But even with all these great functional tools, we still tend to tangle two rather different kinds of code together. We mix code that cares what the data means and the code that cares how it travels from one component to another. Traditional software design structures embed the routing implicitly in the function call graph. For example, our authentication handler will have the logic to select the next function to invoke within its implementation. The control logic and its internal implementation details, thus, end up being intertwined in an ad hoc manner, resulting in a significant coupling problem. If you wish to alter the flow of your application, you must sift through voluminous amounts of incidental code describing the internal minutiae of component implementations.

An effective solution to this problem is to use inversion of control by removing routing logic from the functions and elevating it to first-class citizenship in the design. Why is a state machine the natural fit for this, rather than an event bus or dependency injection? Event buses scatter routing logic to the winds with components shouting into the void, making the overall flow impossible to trace. Implicit callbacks hide the flow inside the implementation. State machines, on the other hand, make routing declarative and visible in one place. They force the separation of what to do from how to do it.

Introducing Mycelium

I’ve spent a great deal of time considering how to construct a system where distinct components are clearly separated by design, with clear boundaries between semantics of the code and the implementation details. This is the basic conceptual orientation of Mycelium, which treats the program as a recursive ecosystem of workflows, solving the very routing problem described above.

Clojure provides the tools for writing pure functions, but falls short of giving us guidance on how to orchestrate them when writing large applications. I initially developed Maestro to provide a clear organizational framework, separating side-effectful concerns from pure calculation, and structuring workflows as graphs where the nodes represent computations of state, and the edges represent transitions between them. The nodes are distinct, context-free blocks, responsible for specific tasks, linked by a thin coordinating layer controlling the flow of data between them. The state itself is represented as a map that's passed from one node to another.

The business logic for each node lives inside a Mycelium component called a cell. Since cells are completely unaware of one another, they are inherently isolated. Each one can be viewed as a miniature self-contained application. It knows how to do its specific job and adheres to a strict lifecycle. It takes a state map, loads the data, runs the logic, and computes a new state as its output. All they can access are the IO resources, and a map containing the input state.

Maestro is responsible for arranging these transitions, so that the decisions are pure, and their effects are encapsulated. When a component needs to move itself from one state to another, it does not simply reach in and take what it needs. Instead, it updates the state map, and delegates to Maestro to orchestrate the transition. Again, this keeps the code responsible for deciding what will happen next separate and distinct from the private parts of the individual cells.

Each cell is additionally wrapped in a Malli schema, which gives the cell a protocol to abide by. You can’t simply hope that the LLM will understand your intentions when they’re expressed in plain English. What you need is a formal contract to determine whether the implementation is correct. Malli enables us to specify precisely what a cell is entitled to receive and what it can produce as its output. It's a flexible way to encode deep, structural invariants representing the interface of the cell.

An agent tasked with constructing a handler for a particular node operates within the constraints of a contract, enforced by the schema, both during development and at runtime. Crucially, the agent doesn't need to scour the codebase to discover the relevant cell; the orchestration layer (acting as a Conductor) assigns the specific cell ID and its schema directly to the agent. It operates within a tightly bounded context provided to it. If the code produced by the agent does not adhere to the contract, if the output is even slightly off, the system will reject it providing meaningful feedback on what went wrong.

Think, for a minute, about what this does for the scope creep problem. The schema defines the boundary of the cell letting the agent know exactly what keys are in the map, what the data types are, and what the constraints are. Since the components do not interact directly, the agent has a well-defined, perfectly sized context that it needs to understand.

We now have a self-correcting loop. The primary agent, the Conductor, designs the workflow in EDN. A fleet of smaller, specialized agents do the individual tasks. If one of them makes an error, it is detected by the Malli contract before it can propagate forward to the next node in the graph. The specific validation failure is known at the point where it arose, and its scope is limited to the node that produced it.

The State Machine Graph as a Contract

Treating an application's high-level behavior as a state machine graph provides us with a master blueprint. It allows us to determine what the intent of a particular workflow is by reading a declarative schema describing the states and the transitions between the cells. A human can review and approve a data flow diagram, which specifies the semantics of each cell, and the rules guiding the flow of data across them. The details of how each cell functions are abstracted behind its API, and managed by the agent responsible for implementing its functionality. Hence, the orchestrator only needs to concern itself with the routing aspect of the application and ensuring that the schemas of nodes sharing an edge are compatible.

The orchestration layer is in charge of directing the work, and executing the branching logic. Its sole concern is to examine the results from each node to decide on the next branch to take according to the EDN specification.

Because the intercellular connections form a directed graph, and since motion is governed by payload state, with the routing logic separated from the cell code, you can, in principle, determine the entire decision tree of the application just by examining the EDN spec. Instead of having to dig through conditional branches buried in thousands of lines of implementation code, you have a declarative map of possibilities.

How This Works in Practice

You can glimpse the way in which Mycelium binds these ideas together by examining a snippet from the user-onboarding demo. The workflow definition is the point of departure. We start by defining a cell and its strict contract:

;; A cell contract for session validation
{:id       :auth/validate-session
   :doc      "Check credentials against the session store"
   :schema   {:input  [:map
                        [:user-id :string]
                        [:auth-token :string]]
              :output {:authorized   [:map
                                       [:session-valid :boolean]
                                       [:user-id :string]]
                       :unauthorized [:map
                                       [:session-valid :boolean]
                                       [:error-type :keyword]
                                       [:error-message :string]]}}
   :requires [:db]}

This map represents a stable building block. There is no ambiguity in a declarative specification. A cell is defined by the shape of its input and output, along with its resource requirements. The routing logic is extracted entirely into separate :edges and :dispatches keys in the workflow.

 :edges
 {:validate-session {:authorized   :fetch-profile
                     :unauthorized :error}}

 :dispatches
 {:validate-session [[:authorized   (fn [data] (:session-valid data))]
                     [:unauthorized (fn [data] (not (:session-valid data)))]]}

The edges shown in the snippet represent the possible transitions. The dispatches constitute a list of node identifiers, each with an associated decision function that examines the state and determines whether it should be processed by the identifier in question. Dispatches are processed on a first come, first served basis; that is, the state will be routed to the first matching identifier found.

Next, we have the cell associated with the spec, which is responsible for performing the actual work. Following Integrant philosophy, the cells are defined as a collection of multimethods.

(defmethod cell/cell-spec :user/fetch-profile [_]
  {:id      :user/fetch-profile
   :doc     "Fetch user profile from database"
   :handler (fn [{:keys [db]} data]
              (if-let [user (db/get-user db (:user-id data))]                
                (assoc data :profile (select-keys user [:name :email]))
                (assoc data
                       :error-type    :not-found
                       :error-message (str "User not found: " (:user-id data)))))})

Note how the :user/fetch-profile handler doesn’t need to know where the data came from, nor does it decide where to send it. All it receives is the current state. The cell does its work and then returns an updated map. The orchestration layer evaluates the dispatches against this new data to select the next edge.

Long before a user ever makes a request, the workflow goes through a rigorous validation phase at compile time. During startup, the engine verifies that every cell exists in the registry, that every transition has a valid destination, and that all the input and output schemas chain together with no discontinuities. This is the moment where the blueprint becomes an active, executable process.

When a request (like POST /api/onboarding) actually arrives, the HTTP routing library recognizes the endpoint and shoves it onward to an onboarding handler. This handler summons the pre-compiled workflow engine, giving it the database connection and the raw request. As the state machine proceeds through its transitions, the Malli schemas are serving as sentinels.

Reliability, Debugging, and Testing

The State Map acts as the single source of truth throughout this lifecycle. Every transformation is explicit in the return value, and there are no side channels modifying the state. Like a messenger, it travels through the system, carrying on its person all the data that has been gathered up to this point, as well as associated metadata.

Because the state map keeps a :mycelium/trace of every transition, you get unparalleled observability. If a workflow fails, you don’t just get a stack trace telling you where in the codebase the crash happened; you get the full history at the moment of failure. You can see the inputs, the previous steps, and the exact data that caused the routing logic to stumble. For the coding agent, it’s as if there’s a black box flight recorder on every single run.

Such level of observability fundamentally transforms how we test applications. Testing in often seen as a thoroughly distasteful chore, so much so that you will often find people spending more time with mocks and dependency injection than actually writing tests.

Mycelium treats each fragment of logic as a pure update of a data structure. Testing reduces to a straightforward exercise in data juggling. You don’t have to mock up a database to test how a particular system handles a User Not Found scenario; you simply feed the component a state map lacking the :user key and see what output it produces.

Because every workflow node is contractually bound by its Malli schema, you can take the :validate-user-data handler, feed it a map of bad data, and check that it sets an :invalid key on the state map. You’re not testing the whole onboarding flow; you’re testing one specific cell.

In Mycelium, logical integration tests can be performed trivially, simply by executing the workflow with a mock resource map. Resources like the database are passed in separately, so a real Postgres connection can be exchanged for a mock in the test suite. The difference is entirely irrelevant to the workflow.

;; A logical integration test
(deftest onboarding-workflow-test
  (let [ds       (create-test-db!)
        compiled (wf/compile-workflow onboarding-manifest)
        result   (fsm/run compiled
                         {:db ds}
                         {:data {:http-request
                                 {:body {"email" "test@example.com"}}}})]
    (is (some? (:profile result)))
    (is (= "test@example.com" (get-in result [:profile :email])))))

Because of the trace history, your test assertions become incredibly descriptive. You aren't just checking if the final result is 200 OK. You are able to check that the system moved from :start to :validate-session to :fetch-profile in the exact order you expected. You get the confidence of a full system test by simply passing mock resources to your workflow, and verifying that transitions happen in correct order.

Layered Abstraction and Infinite Scale

The entire design is naturally recursive. A complete system implemented as a network of cells can itself be viewed as a single cell, which offers up an interface and can then be slotted into a yet-larger state machine network. You might have a simple network handling user login, which becomes a component in a medium-scale network managing the payment process, which itself becomes a component in a large-scale network implementing a complete online emporium. Scaling becomes a matter of arranging components on a graph rather than increased coupling within the codebase.

But of course, there must be clear boundaries set for such a system. The most promising way to define components and graphs draws on functional programming and formal methods. For example, Malli-driven schemas provide a means of establishing checkpoints that the LLM agent cannot bypass. The agent must fulfill the contract by adhering to all the constraints and requirements.

Once the high-level design is in place, these contracts can be issued to the agents in charge of the nodes in the graph. You no longer need an exceptionally capable agent that's able to comprehend a massive context and keep track of the interconnections across a sprawling application. The job description of the agent in charge of the flow of control is likewise dramatically circumscribed. The internal details of the cells can be ignored, with attention paid only to the graph itself. If the graph grows too complex, it, too, can be divided into separate, independent subgraphs. In this architecture, the context never needs to become unmanageably large.

The Agent Synergy

Historically, this kind of design has been hard to sell. Workflow engines and state-graph systems are not without their proponents, but they haven’t exactly swept the world. The basic problem is that they require a lot of additional ceremony. While wiring functions together by hand permits the programmer to forge ahead in a straight line, forcing yourself to step back, design a state graph, worry about the transition logic among disparate files, and code the glue just feels too onerous. Most programmers would much rather just bang out an if statement and keep going.

But the picture changes completely when we introduce coding agents. A large language model lacks ego and has no difficulty writing boilerplate code. It does not find itself bored by ceremony, nor frustrated by the need for upfront structural planning. In fact, language models thrive on it. What is a tedious tax for a human developer serves as an explicit, unambiguous map for an agent. By embracing this structure, the agent secures the very boundaries it needs to stay coherent.

Divorcing data flow from data transformation resolves the problem of LLM context overload in an elegant and general way. A strategic agent, in the form of the Conductor, coordinates the flow in the orchestration layer, plotting the tracks. Meanwhile, individual handler agents are responsible for writing, refining, and documenting their particular domain which is just a station on the railway. They do so within a safe, bounded context, avoiding the thicket of intractable problems posed by runaway cognitive saturation.

Imagine a future in which coding agents assemble software systems by integrating and adapting existing workflows that have been approved by human oversight. We no longer have to suffer the frustration of constructing complex machines built out of opaque and unreliable components. We can rely on standardized and tested building blocks, following a clear and verifiable assembly plan, so that the resulting system is guaranteed by its very design to be correct, adaptable, and observable.

Stop Round-Tripping Your Codebase: How to Cut LLM Token Usage by 80% Using Recursive Document Analysis

Fri, 16 Jan 2026 00:00:00 +0000

When you employ AI agents such as Claude, there’s a significant volume problem for document study. Reading one file of 1000 lines consumes about 10,000 tokens. Token consumption incurs costs and time penalties. Codebases with dozens or hundreds of files, a common case for real world projects, can easily exceed 100,000 tokens in size when the whole thing must be considered. The agent must read and comprehend, and be able to determine the interrelationships among these files. And, particularly, when the task requires multiple passes over the same documents, perhaps one pass to divine the structure and one to mine the details, costs multiply rapidly.

Matryoshka is a tool for document analysis that achieves over 80% token savings while enabling interactive and exploratory analysis. The key insight of the tool is to save tokens by caching past analysis results, and reusing them, so you do not have to process the same document lines again. These ideas come from recent research, and retrieval-augmented generation, with a focus on efficiency. We'll see how Matryoshka unifies these ideas into one system that maintains a persistent analytical state. Finally, we'll take a look at some real-world results analyzing the anki-connect codebase.

The Problem: Context Rot and Token Costs

A common task is to analyze a codebase to answers a question such as “What is the API surface of this project?” Such work includes identifying and cataloguing all the entry points exposed by the codebase.

Traditional approach:

1. Read all source files into context (~95,000 tokens for a medium project)
2. The LLM analyzes the entire codebase’s structure and component relationships
3. For follow-up questions, the full context is round-tripped every turn

This creates two problems:

Token Costs Compound

Every time, the entire context has to go to the API. In a 10-turn conversation about a codebase of 7,000 lines, almost a million tokens might be processed by the system. Most of those tokens are the same document contents being dutifully resent, over and over. The same core code is sent with every new question. This redundant transaction is a massive waste. It forces the model to process the same blocks of text repeatedly, rather than concentrating its capabilities on what’s actually novel.

Context Rot Degrades Quality

As described in the Recursive Language Models paper, even the most capable models exhibit a phenomenon called context degradation, in which their performance declines with increasing input length. This deterioration is task-dependent. It’s connected to task complexity. In information-dense contexts, where the correct output requires the synthesis of facts presented in widely dispersed locations in the prompt, this degradation may take an especially precipitous form. Such a steep decline can occur even for relatively modest context lengths, and is understood to reflect a failure of the model to maintain the threads of connection between large numbers of informational fragments long before it reaches its maximum token capacity.

The authors argue that we should not be inserting prompts into the models, since this clutters their memory and compromises their performance. Instead, documents should be considered as external environments with which the LLM can interact by querying, navigating through structured sections, and retrieving specific information on an as-needed basis. This approach treats the document as a separate knowledge base, an arrangement that frees up the model from having to know everything.

Prior Work: Two Key Insights

Matryoshka builds on two research directions:

Recursive Language Models (RLM)

The RLM paper introduces a new methodology that treats documents as external state to which step-by-step queries can be issued, without the necessity of loading them entirely. Symbolic operations, search, filter, aggregate, are actively issued against this state, and only the specific, relevant results are returned, maintaining a small context window while permitting analysis of arbitrarily large documents.

Key point is that the documents stay outside the model, and only the search results enter the context. This separation of concerns ensures that the model never sees complete files, instead, a search is initiated to retrieve the information.

Barliman: Synthesis from Examples

Barliman, a tool developed by William Byrd and Greg Rosenblatt, shows that it is possible to use program synthesis without asking for precise code specifications. Instead, input/output examples are used, and a solver engine is used as a relational programming system in the spirit of miniKanren. Barliman uses such a system to synthesize functions that satisfy the constraints specified. The system interprets the examples as if they were relational rules, and the synthesis engine tries to satisfy them. This approach makes it possible to describe what is desired for concrete test cases.

The approach is to simply show examples of the kind of behavior one wishes the system to exhibit, letting it derive the implmentation on its own. Thus, the emphasis shifts from writing long and detailed step-by-step recipes for behavior to simply portraying, in a declarative fashion, what the desired goal is.

Matryoshka: Combining the Insights

Matryoshka incorporates these insights into a functioning system for LLM agents. A practical tool is provided that enables agents to decompose challenging tasks into a sequence of smaller and more manageable objectives.

1. Nucleus: A Declarative Query Language

Instead of issuing commands, the LLM describes what it wants, using Nucleus, a simple S-expression query language. This changes the focus from describing each step to specifying the desired outcome.

(grep "class ")           ; Find all class definitions
(count RESULTS)           ; Count them
(map RESULTS (lambda x    ; Extract class names
  (match x "class (\\w+)" 1)))

We observe that the declarative interface retains its robustness even when the LLM employs different vocabulary or sentence structures. This robustness originates from the system’s commitment to elucidating the underlying intent of a request, independent of superficial linguistic variations.

2. Pointer-Based State

The key new insight is that we can separate the results from the context. Results are now stored in the REPL state, rather than in the context.

When Claude runs (grep "def ") and gets 150 matches:

Traditional tools: All 150 lines are fed into context, and round-tripped every turn
Matryoshka: Binds matches to RESULTS in the REPL, returning only "Found 150 results"

The variable RESULTS is bound to the actual value in the REPL. This binding acts as a pointer, revealing the location of the data within the server's memory. Subsequent operations, queries, for example, or updates, use this reference to access the data. But the data itself never actually enters the conversation:

Turn 1: (grep "def ")         → Server stores 150 matches as RESULTS
                              → Context gets: "Found 150 results"

Turn 2: (count RESULTS)       → Server counts its local RESULTS
                              → Context gets: "150"

Turn 3: (filter RESULTS ...)  → Server filters locally
                              → Context gets: "Filtered to 42 results"

The LLM never sees the 150 function definitions, just the aggregated answers from these functions.

3. Synthesis from Examples

When queries need custom parsing, Matryoshka synthesizes functions from examples:

(synthesize_extractor
  "$1,250.00" 1250.00
  "€500" 500
  "$89.99" 89.99)

The synthesizer learns the pattern directly from examples, obtaining numerical values straight from the currency strings and entirely circumventing the need to construct manual regex.

The Lifecycle

A typical Matryoshka session:

1. Load Document

(load "./plugin/__init__.py")
→ "Loaded: 2,244 lines, 71.5 KB"

The document is parsed and stored server-side. Only metadata enters the context.

2. Query Incrementally

(grep "@util.api")
→ "Found 122 results, bound to RESULTS"
   [402] @util.api()
   [407] @util.api()
   ... (showing first 20)

Each query returns a preview plus the count. Full data stays on server.

3. Chain Operations

(count RESULTS)           → 122
(filter RESULTS ...)      → "Filtered to 45 results"
(map RESULTS ...)         → Transforms bound to RESULTS

Operations chain through the RESULTS binding. Each step refines without re-querying.

4. Close Session

(close)
→ "Session closed, memory freed"

Sessions auto-expire after 10 minutes of inactivity.

How Agents Discover and Use Matryoshka

Matryoshka integrates with LLM agents via the Model Context Protocol (MCP).

Tool Discovery

When Claude Code starts, it launches Matryoshka as an MCP server and receives a tool manifest:

{
  "tools": [
    {
      "name": "lattice_load",
      "description": "Load a document for analysis..."
    },
    {
      "name": "lattice_query",
      "description": "Execute a Nucleus query..."
    },
    {
      "name": "lattice_help",
      "description": "Get Nucleus command reference..."
    }
  ]
}

Claude sees the available tools and their descriptions. When a user asks to analyze a file, Claude decides which tools to use based on the task.

Guided Discovery

The lattice_help tool returns a command reference, teaching the LLM the query language on-demand:

; Search commands
(grep "pattern")              ; Regex search
(fuzzy_search "query" 10)     ; Fuzzy match, top N
(lines 10 20)                 ; Get line range

; Aggregation
(count RESULTS)               ; Count items
(sum RESULTS)                 ; Sum numeric values

; Transformation
(map RESULTS fn)              ; Transform each item
(filter RESULTS pred)         ; Keep matching items

The agent learns capabilities incrementally rather than needing upfront training.

Session Flow

User: "How many API endpoints does anki-connect have?"

Claude: [Calls lattice_load("plugin/__init__.py")]
        → "Loaded: 2,244 lines"

Claude: [Calls lattice_query('(grep "@util.api")')]
        → "Found 122 results"

Claude: [Calls lattice_query('(count RESULTS)')]
        → "122"

Claude: "The anki-connect plugin exposes 122 API endpoints,
         decorated with @util.api()."

Each tool invocation maintains its own state within the conversation. So, for example, when a document is loaded, that content is retained in memory. Similarly, the results of any query that is executed are saved and available for later use.

Real-World Example: Analyzing anki-connect

Let's walk through a complete analysis of the anki-connect Anki plugin. Here we have a real-world codebase with 7,770 lines across 17 files.

The Task

"Analyze the anki-connect codebase: find all classes, count API endpoints, extract configuration defaults, and document the architecture."

The Workflow

The agent uses Matryoshka's prompt hints to accomplish the following workflow:

1. Discover files with Glob
2. Read small files directly (<300 lines)
3. Use Matryoshka for large files (>500 lines)
4. Aggregate across all files

Step 1: File Discovery

Glob **/*.py → 15 Python files
Glob **/*.md → 2 markdown files

File sizes:
  plugin/__init__.py    2,244 lines  → Matryoshka
  plugin/edit.py          458 lines  → Read directly
  plugin/web.py           301 lines  → Read directly
  plugin/util.py          107 lines  → Read directly
  README.md             4,660 lines  → Matryoshka
  tests/*.py           11 files      → Skip (tests)

Step 2: Read Small Files

Reading util.py (107 lines) reveals configuration defaults:

DEFAULT_CONFIG = {
    'apiKey': None,
    'apiLogPath': None,
    'apiPollInterval': 25,
    'apiVersion': 6,
    'webBacklog': 5,
    'webBindAddress': '127.0.0.1',
    'webBindPort': 8765,
    'webCorsOrigin': None,
    'webCorsOriginList': ['http://localhost'],
    'ignoreOriginList': [],
    'webTimeout': 10000,
}

Reading web.py (301 lines) reveals the server architecture:

- Classes: WebRequest, WebClient, WebServer
- JSON-RPC style API with jsonschema validation
- CORS support with configurable origins

Step 3: Query Large Files with Matryoshka

; Load the main plugin file
(load "plugin/__init__.py")
→ "Loaded: 2,244 lines, 71.5 KB"

; Find all classes
(grep "^class ")
→ "Found 1 result: [65] class AnkiConnect:"

; Count methods
(grep "def \\w+\\(self")
→ "Found 148 results"

; Count API endpoints
(grep "@util.api")
→ "Found 122 results"

; Load README for documentation
(load "README.md")
→ "Loaded: 4,660 lines, 107.2 KB"

; Find documented action categories
(grep "^### ")
→ "Found 13 sections"
   [176] ### Card Actions
   [784] ### Deck Actions
   [1231] ### Graphical Actions
   ...

Complete Findings

Metric	Value
Total files	17 (15 .py + 2 .md)
Total lines	7,770
Classes	8 (1 main + 3 web + 4 edit)
Instance methods	148
API endpoints	122
Config settings	11
Imports	48
Documentation sections	8 categories, 120 endpoints

Token Usage Comparison

Approach	Lines Processed	Tokens Used	Coverage
Read everything	7,770	~95,000	100%
Matryoshka only	6,904	~6,500	65%
Hybrid	7,770	~17,000	100%

The hybrid method achieves a 82% savings in tokens while retaining 100% of the original coverage. This approach combines two different strategies, one for compressing redundant information and one for preserving unique insights.

The pure Matryoshka approach ends up missing details from small files (configuration defaults, web server classes), because Claude only uses the tool to query large ones. The hybrid workflow does direct, full-content reads on small files, while leveraging Matryoshka to analyze bigger files, in a kind of divide-and-conquer strategy. All that's needed is to provide the agent an explicit hint on the strategy to use.

Why Hybrid Works

Small files (<300 lines) contain critical details:

- util.py: All configuration defaults, the API decorator implementation
- web.py: Server architecture, CORS handling, request schema

These fit comfortably in context, and there's no need to do anything different. Matryoshka adds value for:

- __init__.py (2,244 lines): Query specific patterns without loading everything
- README.md (4,660 lines): Search documentation sections on demand

Architecture

┌─────────────────────────────────────────────────────────┐
│                     Adapters                            │
│  ┌──────────┐  ┌──────────┐  ┌───────────────────────┐  │
│  │   Pipe   │  │   HTTP   │  │   MCP Server          │  │
│  └────┬─────┘  └────┬─────┘  └───────────┬───────────┘  │
│       │             │                    │              │
│       └─────────────┴────────────────────┘               │
│                          │                              │
│                ┌─────────┴─────────┐                    │
│                │   LatticeTool     │                    │
│                │   (Stateful)      │                    │
│                │   • Document      │                    │
│                │   • Bindings      │                    │
│                │   • Session       │                    │
│                └─────────┬─────────┘                    │
│                          │                              │
│                ┌─────────┴─────────┐                    │
│                │  NucleusEngine    │                    │
│                │  • Parser         │                    │
│                │  • Type Checker   │                    │
│                │  • Evaluator      │                    │
│                └─────────┬─────────┘                    │
│                          │                              │
│                ┌─────────┴─────────┐                    │
│                │    Synthesis      │                    │
│                │  • Regex          │                    │
│                │  • Extractors     │                    │
│                │  • miniKanren     │                    │
│                └───────────────────┘                    │
└─────────────────────────────────────────────────────────┘

Getting Started

Install from npm:

npm install matryoshka-rlm

As MCP Server (Claude Code / Claude Desktop)

Add to your Claude configuration:

{
  "mcpServers": {
    "lattice": {
      "command": "npx",
      "args": ["lattice-mcp"]
    }
  }
}

Programmatic Use

import { NucleusEngine } from "matryoshka-rlm";

const engine = new NucleusEngine();
await engine.loadFile("./document.txt");

const result = engine.execute('(grep "pattern")');
console.log(result.value); // Array of matches

Interactive REPL

npx lattice-repl
lattice> :load ./data.txt
lattice> (grep "ERROR")
lattice> (count RESULTS)

Conclusion

Matryoshka embodies the principle, emerging from RLM research, that documents are to be treated as external environments rather than as contexts to be parsed. This principle alters the fundamental character of the model’s engagement, no longer a passive reader but an active agent, navigating through and interrogating a document to extract specific information, somewhat as a programmer would browse through code. Combined with Barliman-style synthesis, in which a solution is built up in a series of small, well-defined steps, and pointer-based state management, it achieves:

- 82% token savings on real-world codebase analysis
- 100% coverage when combined with direct reads for small files
- Incremental exploration where each query builds on previous results
- No context rot because documents stay outside the model

We observe that variable bindings such as RESULTS refer to REPL state rather than holding data directly in model context. As we formulate and submit queries, what is sent to the server are mere pointers, placeholders indicating where the actual computation should occur. It is the server that executes the substantive computational tasks, returning only the distilled results.

The tool is open source: https://github.com/yogthos/Matryoshka

Grounding LLMs with Recursive Code Execution

Mon, 12 Jan 2026 00:00:00 +0000

Despite context windows expanding to millions of tokens, LLMs still struggle with the fundamental task of precision. When you ask an LLM to "analyze this report," it often glances at the text and simply hallucinates a plausible-sounding answer based on probability.

A good example of the problem can be seen when asking a model to sum sales figures from a financial report. Left to its own devices, it will likely not bother reading the whole document and simply give you a made-up answer. This is especially a problem with smaller models that you can run locally.

The standard approach to dealing with this problem is to use Retrieval Augmented Generation (RAG), which relies on semantic similarity (embeddings). If you ask for "sales figures," a Vector DB retrieves chunks of text that sound like sales figures. However, semantic similarity is fuzzy and limited in functionality. Embeddings can't count, so you can't ask questions like "count the number of times X happens." They also can't handle information scattered across a bunch of unrelated lines in a document. Furthermore, they don't distinguish between concepts like "Projected Sales" and "Actual Sales" when they appear in similar contexts.

It would be nice to have a system that treats text as a dataset to be queried rather than a prompt to be completed. This is where the Recursive Language Model paper comes in. The core idea here is that instead of having the model operate directly on the document, it uses a programmatic interface to interact with it via a REPL. The model acts as a programmer writing code to explore the document, interpreting execution results, and only then formulating an answer based on them.

The core insight is that code execution provides grounding for the model. When an LLM guesses a number by trying to understand the document, it might be right, or it might be wrong. It has no way to know. When it writes regex.match() and the computer returns ['$2,340,000'], that result is a hard fact. What the model needs to understand is how to form a query—a general task it's likely good at—instead of trying to solve a domain-specific problem it has no direct training on.

Allowing an LLM to write and run code directly on your system would obviously be a security nightmare, so the implementation uses isolated-vm to create a secure sandbox for it to play in. The model cannot hallucinate rm -rf / or curl a random URL. Having a sandbox also prevents infinite loops or memory leaks. And since the document is immutable, the model can read it but cannot alter the source truth.

The process works as follows:

1. The document is loaded into a secure, isolated Node.js environment as a read-only context variable.
2. The model is given exploration tools: text_stats(), fuzzy_search(), and slice().
3. The Loop:
- • The model writes TypeScript to probe the text.
- • The Sandbox executes it and returns the output.
- • The model reads the result and refines its next step.
4. The loop iterates until the model has enough proven data to answer FINAL("...").

The system can work entirely locally using something like Ollama with Qwen-Coder, or with hosted models like DeepSeek, which are much smarter by default. It also works as an MCP that you can plug in and let your agent use to solve problems.

Finally, I used Universal Tool Calling Protocol (UTCP) patterns from code-mode to generate strict TypeScript interfaces. This provides the LLM with a strict contract such as:

// The LLM sees exactly this signature in its system prompt
declare function fuzzy_search(query: string, limit?: number): Array<{
  line: string;
  lineNum: number;
  score: number; // 0 to 1 confidence
}>;

One problem is that LLMs tend to be messy coders; they forget semicolons, use hallucinated imports, etc. The way around that is to add a self-healing layer. If the sandbox throws a syntax error, a lightweight intermediate step attempts to fix imports and syntax before re-running. This keeps the reasoning chain alive and minimizes round trips to the model.

As a demo to test out the concept, I made a document containing a bunch of scattered data, with 5 distinct sales figures hidden inside 4,700 characters of Lorem Ipsum filler and unrelated business jargon.

Predictably, feeding the text into a standard context window and asking for the total promptly resulted in a hallucinated total of $480,490. It just grabbed numbers that looked like currency from unrelated sections, mashed them together, and called it a day.

Running the same query through RLM was a completely different story. The model took 4 iterations to converge on the actual solution. Instead of trying to guess, it started writing code to explore the document. It first checked the file size:

const stats = text_stats();
console.log(`Document length: ${stats.length}, Lines: ${stats.lineCount}`);

Next, it used fuzzy search to locate relevant lines, ignoring the noise:

const matches = fuzzy_search("SALES_DATA");
console.log(matches);
// Output: [
//   { line: "SALES_DATA_NORTH: $2,340,000", ... },
//   { line: "SALES_DATA_SOUTH: $3,120,000", ... }
// ]

And finally, it wrote a regex to parse the strings into integers and summed them programmatically to get the correct result:

// ...regex parsing logic...
console.log("Calculated Total:", total); // Output: 13000000

Only after the code output confirmed the math did the model verify the answer.

The key difference is that the traditional approach asks the model "what does this document say," while the recursive coding approach asks it to "write a program to find out what this document says." The logic is now expressed using actual code, and the role of the LLM is to write the code and read the results as opposed to working with the document directly.

As with all things, there is a trade-off here: the RLM approach is slower since it takes multiple turns and can generate more tokens as a result. However, if the document you're working on is itself large, then you will actually save context tokens by not loading it directly.

MCP Integration

The project also includes an MCP (Model Context Protocol) server, making it available as a tool for coding agents like Crush. Once configured, you can ask the agent to analyze documents that would otherwise exceed its context window or require precise data extraction.

The server exposes an analyze_document tool that takes a query and file path. The tool can then use the RLM approach to explore documents by writing code, executing it in the sandbox, and iterating until it finds the answer.

This creates an interesting dynamic where you agent writes the high-level query, the RLM's backing model (which can be a local Ollama instance) does the iterative code exploration, and the verified results come back to your agent. The grounding problem is solved at the tool level, so the agent can trust the results it receives.

The implementation is available at https://github.com/yogthos/Matryoshka.

Edge-Aware Pixelation for Better Pixel Art

Thu, 11 Dec 2025 00:00:00 +0000

Pixelation is everywhere these days, from retro game aesthetics to modern design trends. But if you've ever tried to create pixel art from a photograph, you've probably noticed that the results often look a bit off. Edges get jagged, important features get distorted, and the whole image tends to lose its character.

Most pixelation algorithms rely on the same approach of downscaling the image, then upscaling it back. While this is fast and cheap to implement, it treats every part of the image the same way forcing a rigid grid that cuts indiscriminately across edges, faces, and fine details.

In this post, we'll take a look at a strategy of using edge-aware pixelation. The idea here is to use an algorithm that adapts to the structure of the image rather than forcing a uniform grid onto it. We'll look at how it works, why it tends to produce better results, and the trade-offs involved.

The Traditional Approach

Let's start by looking at what most pixelation libraries do under the hood. The most common approach uses standard image scaling with smoothing.

// Pseudo-code for traditional pixelation
const downscaled = scaleDown(image, pixelSize); // Uses bilinear/bicubic
const pixelated = scaleUp(downscaled, originalSize); // Nearest-neighbor

While this works, there are a few obvious problems. Smoothing filters tend to blur important edges before downsampling, and the pixel grid doesn't care about the image content. This often leads to artifacts where edges get "chopped" across pixel boundaries, creating jagged lines. You also end up losing fine features like eyes or text.

Some approaches try to improve on this using median color sampling instead of averaging.

// Sample colors in each block, pick median
const blockColor = median(colorsInBlock);

This avoids some of the blurring, but it still suffers from the same fixed grid issue. It ignores the image structure and can create harsh transitions between blocks.

Basically, all these methods force a uniform grid onto the image without considering its content. The result is usually pixel blocks that cut across important features. For example, if we use the following image from wikimedia as the input.

The result ends up looking something like the following when using a naive pixelation algorithm:

Edge-Aware Adaptive Grid

Instead of forcing a rigid grid, we can let the grid adapt to the image. This is the core idea behind edge-aware pixelation. We can treat pixelation as an optimization problem with four main stages.

1. Edge Detection

First, we need to understand where the important features are in the image.

// Sobel operators detect gradient magnitude and direction
const gradient = calculateGradient(image);
const edges = applyNonMaximumSuppression(gradient);
const edgeMap = thresholdEdges(edges, sharpness);

We use Sobel operators to compute the gradient magnitude, and then apply non-maximum suppression to thin the edges to a single-pixel width. Finally, we use percentile-based thresholding to adapt to the edge distribution of the specific image. Since this can be computationally expensive, using WebGL can provide a significant speedup here.

This gives us an edge map where bright pixels represent edges and dark pixels represent smooth regions.

2. Grid Initialization

Next, we start with a regular grid matching the target pixel size.

const grid = createUniformGrid(width, height, pixelSize);

Unlike traditional methods, the grid is just a starting point. Each cell is defined by four corner points which we can move around.

3. Grid Optimization

This is where the actual work happens. We iteratively move the grid corners to align the cell boundaries with the detected edges.

for (let iteration = 0; iteration < numIterations; iteration++) {
  for (each corner in grid) {
    // Search nearby positions
    const candidates = searchNeighborhood(corner, stepSize);

    // Evaluate how well each position aligns edges
    for (const candidate of candidates) {
      const score = evaluateEdgeAlignment(candidate, edgeMap);
      if (score > bestScore) {
        bestPosition = candidate;
      }
    }

    // Move corner toward best position (with damping)
    corner.moveToward(bestPosition, damping);
  }
}

For each corner, we test multiple positions in a local neighborhood and evaluate the alignment by sampling edge strength along the grid edges. We want to find positions where the grid boundaries follow the edges continuously. We also use damping to prevent over-optimization and maintain stability.

The result is a grid that bends and adapts to align with the natural structure of the image.

4. Color Assignment

Finally, we need to assign colors to our optimized cells.

for (each cell in grid) {
  const pixels = samplePixelsInCell(cell);

  // Blend between average (soft) and median (crisp) based on edge presence
  if (cell.hasEdges) {
    color = blend(average(pixels), median(pixels), sharpness);
  } else {
    color = average(pixels); // Smooth regions use average
  }

  renderCell(cell, color);
}

Here we can use a blending strategy. For smooth regions, we use the average color for natural blending. For edge regions, we can blend between average and median based on the desired sharpness. This lets us tune the aesthetic from soft, blended edges to crisp, high-contrast ones. Looking at the two methods side by side, we can see how much smoother the resulting image is:

Edge detection	Naive

Discussion

There are several advantages to this approach. The most obvious is edge preservation. Traditional methods create jagged artifacts because pixel boundaries cut across edges. By aligning the grid boundaries with the edges, we can preserve continuity and create smoother transitions.

This also means we don't get those choppy edges on outlines, and we can preserve fine details like facial features or text much better. The algorithm effectively has some semantic awareness of what's important in the image.

However, there are trade-offs to consider. The adaptive grid and color blending can produce softer edges compared to traditional methods. If you're looking for extremely crisp, high-contrast pixel art with hard edges, like what you'd see in retro games, traditional downsampling might actually be a better fit.

You also get less contrast in some cases. The color blending can reduce the overall "punchiness" compared to median sampling.

Performance is another factor. Edge-aware pixelation is computationally more intensive. You have to handle edge detection, iterative grid optimization, and spatial hashing for rendering. While WebGL optimizations make it practical taking 100-500ms on most images, simple downsampling will always be faster.

I've found that edge-aware pixelation works best for photographs, portraits, and complex scenes where preserving structure is important. It's less ideal for abstract art or images where a uniform grid is desired for stylistic reasons.

Conclusion

By detecting edges and optimizing alignment, edge-aware pixelation produces pixel art that does a good job of preserving the essence of the original while still achieving low-resolution aesthetic.

If you're interested in trying this out live here, and all the code is available as an open source library called Pixel Mosaic that implements both traditional and edge-aware pixelation.

I hope this gives you some ideas for your own image processing projects. Sometimes the simple method is enough, but for complex images, the extra effort of edge-aware processing can be well worth it.

Role of Debate in Problem Solving

Sat, 10 Jun 2023 00:00:00 +0000

Throughout my career, I have come to appreciate the vital role of open debate in the successful execution of projects. Curiously, arguments and disagreements often carry negative connotations, yet they serve as indispensable tools for fostering a shared comprehension of the problems at hand.

Individuals frequently shy away from engaging in arguments due to personal insecurities that hinder the expression of contrary opinions. Moreover, the fear of causing offense through disagreement serves as a deterrent, preventing many from sharing crucial insights with their teams. When these valuable insights remain private, teams are susceptible to investing significant time and effort in constructing solutions that are fundamentally flawed.

Open debate, encompassing the nature of the problem being addressed and the proposed solutions, stands as a pivotal factor in establishing a well-functioning team. It is crucial to recognize that disagreements generally arise from individuals approaching the problem from different perspectives. One developer may ponder a specific set of trade-offs, while another contemplates a different set.

Each developer possesses only a partial view of the overall picture, rendering the solution proposed by the other developer seemingly incorrect from their respective vantage points. The path to comprehending the complete picture involves engaging in dialogue to grasp the trade-offs considered by each person. Once this pivotal step is taken, it becomes possible to arrive at a solution that addresses the concerns of all involved.

The ultimate objective is not necessarily to discover the optimal solution, but rather to consciously select a set of trade-offs that everyone finds agreeable. By doing so, we ensure that all team members possess a shared understanding of the problem at hand and the rationale behind adopting a specific approach to tackle it.

It is vital to recognize that the ability to engage in frank discussions is a gradual process. Engaging in activities such as pair programming, where developers collaborate closely and solve problems together, can facilitate the development of comfort and familiarity. As individuals grow more acquainted with one another, the prospect of openly disagreeing becomes more natural and effortless.

Finally, I highly recommend watching a recent talk from Rich Hickey discussing this in more detail.

Making a Resume with Node.js Babashka (nbb)

Fri, 12 May 2023 00:00:00 +0000

I recently had to update my resume and decided that I might as well have some fun with it while I do it. One thing that I've always found frustrating using an editor like OpenOffice is that it conflates the tasks of formatting and editing content. I don't want to have to worry about look and feel when I'm thinking about the content of the resume, and vice versa.

The obvious solution is to create a template for how the resume should look, and then populate it with the relevant data. Of course, there are already off the shelf tools such as JSON Resume that do this, but what fun is there in using an existing tool when you can build one that does exactly what you want. Let's take a look at what's involved in making a similar tool with Clojure and nbb.

I decided to create a Hiccup template that would represent the layout of the HTML document and then to put the contents of the resume in a separate EDN file. Then all I'd need to do would be to walk over the template, inject the data, and render HTML which could then be converted to a PDF document.

Specification

Generally, I find it's helpful to start by defining what the API will look like first, and then figure out what the best way to implement it is. This way there is less of a chance that implementation details will bleed into the API. In this scenario the API will be the format of the EDN file and the Hiccup template.

The EDN is just a data structure that's used to organize the data in the resume. I based mine on the schema that JSON Resume uses. The schema contains sections such as :basics, :work, :education, and so on. For example, the :basics section might look as follows:

{:basics {:name "John Doe"
          :label "Programmer"
          :image "profile.jpg"
          :email "john@gmail.com"          
          :summary "A summary of John Doe…"}}

This data then needs to be fed into the Hiccup template that might look like this:

[:html
 [:body
  [:header#header
   [:div.container
    [:div.row 
     [:div.col-sm-9.col-sm-push-3
      [:h1 :data/basics.name]
      [:h2 :data/basics.label]]]]]
  [:page/image {:src   :data/basics.image
                :width "60px"}]
  [:div [:strong "Email"] [:td [:span.email :data/basics.email]]]
  [:section#about.row
   [:aside.col-sm-3
    [:h3 "About"]]
   [:div.col-sm-9
    [:p :data/basics.summary]]]]]

I decided to use namespaced keys to specify the dynamic elements in the template. This provides a clean way to differentiate them from the static Hiccup tags and hint at the type of key. For example, keys namespaced with data indicate that they're paths that should be looked up within the EDN document. The :data/basics.name keyword translates into (get-in data [:basics :name]) when the template is parsed.

Another example is using page namespace to indicate a tag that needs to be processed in a special way. The :page/image tag above will create an :img tag with the image at the path specified using :data/basics.image injected as a base 64 string. This trick provides a flexible way to specify dynamic behaviors in the template.

Finally, I wanted to handle evaluation of forms in order to handle things like iteration within the template. In the snippet below, for macro is called on the data found under the projects key in order to create a list of projects:

[:section#projects.row
    [:aside.col-sm-3
     [:h3 "Projects"]]
    [:div.col-sm-9
     [:div.row
      (for [{:keys [name summary url]} :data/projects]
        [:div.col-sm-12
         [:h4.strike-through
          [:span name]]
         [:div summary]
         [:div
          [:a {:href url} url]]])]]]

Examples above cover all the functionality I expect to need for making a nice looking resume. Let's take a look at what's involved in putting it all together.

Implementation

Conveniently, nbb provides support for starting up nREPL by running nbb nrepl-server :port 1337. This facilitates interactive development that Clojure developers know and love. First thing I decided to do after getting the REPL fired up was to make a little Hiccup parser borrowing the relevant bits from the original implementation:

(defn normalize-body [body]
  (if (coll? body) (apply str (doall body)) (str body)))

(defn as-str  
  [& xs]
  (apply str (map normalize-body xs)))

(defn escape-html  
  [text]
  (-> (as-str text)
      (string/replace #"&" "&amp;")
      (string/replace #"<" "&lt;")
      (string/replace #">" "&gt;")
      (string/replace #"'" "&apos;")))

(defn xml-attribute [id value]
  (str " " (as-str (name id)) "=\"" (escape-html value) "\""))

(defn render-attribute [[name value]]
  (cond
    (true? value) (xml-attribute name name)
    (not value) ""
    :else (xml-attribute name value)))

(defn render-attr-map [attrs]
  (apply str (sort (map render-attribute attrs))))

(defn merge-attributes [{:keys [id class]} map-attrs]
  (->> map-attrs
       (merge (when id {:id id}))
       (merge-with #(if %1 (str %1 " " %2) %2) (when class {:class class}))))

(defn normalize-element [[tag & content]]
  (let [re-tag    #"([^\s\.#]+)(?:#([^\s\.#]+))?(?:\.([^\s#]+))?"
        [_ tag id class] (re-matches re-tag (as-str (name tag)))
        tag-attrs {:id    id
                   :class (when class (string/replace class #"\." " "))}
        map-attrs (first content)]
    (if (map? map-attrs)
      [tag (merge-attributes tag-attrs map-attrs) (next content)]
      [tag tag-attrs content])))

(defn render-element [[tag attrs & content]]
  (str "<" (name tag) (render-attr-map attrs) ">" (as-str (flatten content)) "</" (name tag) ">"))

(defn render-hiccup [hiccup]
  (postwalk
   (fn [node]
     (if (and (not (map-entry? node)) (vector? node))
       (-> node normalize-element render-element)
       node))
   hiccup))

Next, I wrote a template parser that would walk the Hiccup template and inject relevant data into it:

(def path-sep (.-sep path))

(defn image? [node]
  (and (vector? node) (= :page/image (first node))))

(defn css? [node]
  (and (vector? node) (= :page/css (first node))))

(defn data-node? [node]
  (and (keyword? node) (= "data" (namespace node))))

(defn eval-forms [template]
  (prewalk
   (fn [node]
     (if (list? node)
       (eval node)
       node))
   template))

(defn slurp [filename & {:keys [encoding]}]
  (.toString
   (if encoding
     (fs/readFileSync filename encoding)
     (fs/readFileSync filename))))

(defn spit [filename data & {:keys [encoding mode flag]
                             :or   {encoding "utf8"
                                    mode     "0o666"
                                    flag     "w"}}]
  (let [data (if (string? data) data (str data))]
    (fs/writeFileSync filename data encoding mode flag)))

(defn inject-css [theme ref]
  [:style
   {:type "text/css"}
   (slurp (str theme path-sep ref))])

(defn image->b64 [file-path {:keys [theme]}]
  (when file-path
    (let [format    (last (string/split file-path #"\."))]
      (str
       "data:image/" format ";base64, "
       (-> (path/resolve (str theme path-sep file-path))
           (fs/readFileSync)
           (.toString "base64"))))))

(defn inject-image [[_ path] opts]
  [:img {:src (image->b64 path opts)}])

(defn parse-path [path]
  (mapv keyword (string/split path #"\.")))

(defn parse-template [{:keys [theme template data] :as opts}]
  (eval-forms
   (postwalk
    (fn [node]
      (cond
        (css? node)
        (map (partial inject-css theme) (rest node))
        (image? node)
        (inject-image node opts)
        (data-node? node)
        (get-in data (parse-path (name node)))
        :else node))
    template)))

As you can see, postwalk is used to navigate the template. Each node is inspected and then handled using the appropriate function based on its type. After all the data is injected in the template, the result is passed to eval-forms to evaluate any code such as the for macro we saw above.

With these pieces above in place, I can now generate a nice looking HTML page with the resume content. The last interesting bit is to convert the resulting HTML into a PDF document. The easiest way I found was to use puppeteer in headless mode to render the page and write it out as a PDF:

(defn write-pdf [{:keys [browser pending target pdf-opts]} html]
  (-> browser
      (.then #(.newPage %))
      (.then
       (fn [page _] 
         (-> (.setContent page html)
             (.then #(.emulateMediaType page "screen"))
             (.then (fn [_ _]
                      (-> (.pdf page (clj->js (merge {:path target} pdf-opts)))
                          (.then
                           (fn [_] (reset! pending false)))
                          (.catch #(js/console.error (.-message %))))))
             (.catch #(js/console.error (.-message %))))))))

The entire code for this ended up weighing in at around 200 lines, and I'm pretty happy with the result. Being able to solve these kinds of tasks in a few lines of code is what makes Clojure such a productive language for me. Incidentally, here's a link to my resume, and I am currently open to collaboration or employment opportunities.

The entire project is available here for reference.

Structuring Clojure Applications

Sun, 18 Dec 2022 00:00:00 +0000

This post will take a look at a strategy for structuring Clojure applications that I've found useful in my projects.

While the idea of writing applications in a pure functional style is appealing, it's not always clear how to separate side effects from pure compuation in practice. Variations of Clean Architecture approach are often suggested as a way to accomplish this goal. This style dictates that IO should be handled in the outer layer that wraps pure computation core of the application.

While this notion is appealing, it only works in cases where the totality of the data that will be operated on is known up front. Unfortunately, it's impossible to know ahead of time what data will be needed in most real world applications. In many cases additional data needs to load conditionally based on the type of input and the current state of processing.

What we can do, however, is break up our application into small components that can be reasoned about in isolation. Such components can then be composed together to accomplish tasks of increased complexity. I like to think of this as a Lego model of software development. Each component can be viewed as a Lego block, and we can compose these Lego block in many different ways as we solve different problems.

The problem being solved can be expressed in terms of a workflow represented by a graph where the nodes compute the state, and the edges represent transitions between the states. Each time we enter a node in this graph, we look at the input, decide what additional data we may need, run the computation, and transition to the next state. Each node in the graph is a Lego block that accomplishes a particular task. These nodes are then connected by a layer of code governs the data flow.

One approach to implement the above architecture is to use a map to describe overall state, then pass it through multimethods that each operate on a particular type of state and produce a new one. Each multimethod takes the state map as a parameter, does some operations on it, and then returns a new map that gets passed to the next multimethod. If you're thinking that this sounds a like a state machine then you're very much correct.

Implemention

Let's take a look at a concrete example of what this looks like in practice. Say we have a workflow where one user would like to send an email money transfer to another user using our system. There are a few cases we might want to handle here.

There's the happy path scenario where both users are in the system. In this case we simply withdraw the amount from the payor account and deposit it into the payee account.

Another scenario could be that the payor does not have the sufficient funds to do the transaction. In this case we may want to suspend the transaction until the user adds more funds.

Finally, the user receiving the funds may not be in the system, and they need to be invited before they can accept the transfer.

We can start by defining a few helper functions that represent interactions with external resources.

(def store (atom {:workflows {"33a19b1f-c7d1-45d8-9864-0ea17e01a26d"
                              {:id "33a19b1f-c7d1-45d8-9864-0ea17e01a26d"
                               :from   {:email "bob@foo.bar"}
                               :to     {:email "alice@bar.baz"}
                               :amount 200
                               :action :transfer}}
                  :users {"bob@foo.bar" {:funds 100}
                          "alice@bar.baz" {:funds 10}}}))

(defn persist [store {:keys [id] :as state}]
  (swap! store assoc-in [:workflows id] state))

(defn query [store email]
  (get-in @store [:users email]))

(defn load-state [store workflow-id]
  (get-in @store [:workflows workflow-id]))

(defn send-invite [email]
  (println "sending invite to" email))

(defn notify-user [email message]
  (println "notifying" email message))

(defn send-transfer [store from to amount]
  (println "transfering from" from "to" to amount)
  (swap! store
         #(-> %
             (update-in [:users from :funds] - amount)
             (update-in [:users to :funds] + amount))))

Next, we'll create a map to represent the initial state of the workfow.

{:id "33a19b1f-c7d1-45d8-9864-0ea17e01a26d"
 :from   {:email "bob@foo.bar"}
 :to     {:email "alice@bar.baz"}
 :amount 200
 :action :transfer}

The map will contain a unique id, some initial data that represents user input, and an :action key indicating what action should be applied to the current state of the workflow.

Let's define a multimethod that will dispatch the approprate action handler based on the value of the :action key. The multimethod will accept a map of resources as the first argument. The resources represent any code that deals with IO side effects such as database connections. The map representing the state of the workflow will be passed in as the second argument.

(defmulti handle-action (fn [_resources {:keys [action]}] action))

We can now define a handler for the :transfer operation. This multimethod will hydrate some additional data about the users from the datastore, take the appropriate action, and return a new state with the updated :action key to indicate the next step in the workflow.

(defmethod handle-action :transfer [{:keys [store]} {:keys [from to amount] :as state}]
   (let [from-info (query store (:email from))
         to-info   (query store (:email to))
         available-funds (:funds from-info)
         state     (-> state
                       (update :from merge from-info)
                       (update :to merge to-info))] 
     (cond
       (nil? to-info)
       (assoc state :action :invite) 
       (>= available-funds amount)
       (do
         (send-transfer store (:email from) (:email to) amount)
         (assoc state :action :done))
       (< available-funds amount)
       (assoc state :action :notify-missing-funds))))

Let's add the handlers for :invite and :notify-missing-funds actions.

(defmethod handle-action :notify-missing-funds [{:keys [store]} {:keys [from] :as state}] 
  (notify-user (:email from) "missing funds")
  (persist store (assoc state :action :transfer))
  {:action :await})

(defmethod handle-action :invite [{:keys [store]} {:keys [to] :as state}]
  (send-invite to)
  (persist store (assoc state :action :transfer))
  {:action :await})

Note that :invite and :notify-missing-funds actions persist the state and return the :await action when they complete. We'll use this behavior to indicate that the workflow is blocked on an external action and needs to be suspended.

Finally, we'll add a function that executes the state machine. This function will accept a map containing the resources along with a workflow id. It will load the current state and execute it by dispatching the multimethod defined above.

(defn run-workflow
  [{:keys [store] :as resources} workflow-id]
  (loop [state (load-state store workflow-id)] 
    (condp = (-> state :action)
      :done state
      :await :workflow-suspended
      (let [state (handle-action resources state)]
        (recur state)))))

For simplicity's sake let's use an atom as our mock data store.

(def store (atom {:workflows {"33a19b1f-c7d1-45d8-9864-0ea17e01a26d"
                              {:id "33a19b1f-c7d1-45d8-9864-0ea17e01a26d"
                               :from   {:email "bob@foo.bar"}
                               :to     {:email "alice@bar.baz"}
                               :amount 200
                               :action :transfer}}
                  :users {"bob@foo.bar" {:funds 100}
                          "alice@bar.baz" {:funds 10}}}))

We can now try running this workflow in the REPL. If we run it with the initial state, then we should see that the workflow was suspended because there were insufficient funds to transfer.

=> (run-workflow {:store store} "33a19b1f-c7d1-45d8-9864-0ea17e01a26d")

notifying bob@foo.bar missing funds
:workflow-suspended

The workflow tries to notify the user of the missing funds and returns. Let's add more funds to the account trying to send the transfer.

=> (swap! store assoc-in [:users "bob@foo.bar" :funds] 300)

The workflow restarts where it left off and completes the transfer successfully.

=> (run-workflow {:store store} "33a19b1f-c7d1-45d8-9864-0ea17e01a26d")

transfering from bob@foo.bar to alice@bar.baz 200
{:id "33a19b1f-c7d1-45d8-9864-0ea17e01a26d",
 :from {:email "bob@foo.bar", :funds 300},
 :to {:email "alice@bar.baz", :funds 10},
 :amount 200,
 :action :done}

Formalizing Side Effects

We can make one futher improvement over the implementation above by formalizing resource providers using protocols. Doing so will make it clear what the external dependecies are and facilitate mocking. Let's create Notify and DataStore protocols that look as follows.

(defprotocol Notify
  (send-invite [email])
  (notify-user [email message]))

(defprotocol DataStore
  (persist [_ state])
  (query [_ email])
  (add-funds [_ email amount])
  (load-state [_ workflow-id])
  (send-transfer [_ from to amount]))

Next, let's add a couple of records that implement these protocols.

(defrecord MockNotify []
  Notify
  (send-invite [_ email]
    (println "sending invite to" email))
  (notify-user [_ email message]
    (println "notifying" email message)))

(defrecord AtomDataStore [store]
  DataStore
  (persist [_ {:keys [id] :as state}]
    (swap! store assoc-in [:workflows id] state))
  (query [_  email]
    (get-in @store [:users email]))
  (add-funds [_ email amount]
    (swap! store assoc-in [:users "bob@foo.bar" :funds] 300))
  (load-state [_ workflow-id]
    (println "hi")
    (get-in @store [:workflows workflow-id]))
  (send-transfer [_ from to amount]
    (println "transfering from" from "to" to amount)
    (swap! store
           #(-> %
                (update-in [:users from :funds] - amount)
                (update-in [:users to :funds] + amount)))))

We'll also need to modify our multimethods to use Notify protocol instead of simply calling the functions we defined earlier.

(defmethod handle-action :notify-missing-funds [{:keys [store notify]} {:keys [from] :as state}]
  (notify-user notify (:email from) "missing funds")
  (persist store (assoc state :action :transfer))
  {:action :await})

(defmethod handle-action :invite [{:keys [store notify]} {:keys [to] :as state}]
  (send-invite notify to)
  (persist store (assoc state :action :transfer))
  {:action :await})

Finally, we'll instantiate the records and passing them to our run-workflow function.

(def store (->AtomDataStore (atom {:workflows {"33a19b1f-c7d1-45d8-9864-0ea17e01a26d"
                                                {:id "33a19b1f-c7d1-45d8-9864-0ea17e01a26d"
                                                :from   {:email "bob@foo.bar"}
                                                :to     {:email "alice@bar.baz"}
                                                :amount 200
                                                :action :transfer}}
                                    :users {"bob@foo.bar" {:funds 100}
                                            "alice@bar.baz" {:funds 10}}})))
(def notify (->MockNotify))

(run-workflow {:store store
               :notify notify} 
              "33a19b1f-c7d1-45d8-9864-0ea17e01a26d")

(add-funds store "bob@foo.bar" 100)

(run-workflow {:store store
               :notify notify} 
              "33a19b1f-c7d1-45d8-9864-0ea17e01a26d")

Discussion

There are several aspects of the above approach that I've found to be particularly useful when building applications.

Each multimethod can be treated as a small program that can be reasoned about and tested independently. These multimethods can easily be structured using Clean Architecture style keepng IO at the edges.

Passing resources in as an explicit parameter allows decoupling IO from computation. This design lends itself well to testing since resources, such as the data store, are passed in explicitly. We can pass in a map of mock resources when running tests without any changes to the rest of the code. In fact, we can start developing against mock resources and ensure that the workflow logic works as intended before having to worry about connecting to databases or other systems.

Using a map to track the state of the execution makes it easy to inspect it. We can log this map to see what operation we're doing, what the data looks like, and so on. The state can also be easily serialized, allowing us to suspend and resume computation as needed. This is particularly useful in cases when the workflow needs to be suspended pending some external action as we saw earlier.

This design also plays well with Integrant which can be used to manage the system map containing stateful resources.

Most importantly, this type of architecture creates reusable components without implicit coupling. Each multimethod can be used indepenently of the others, and composed into different workflows. This gives us composable Lego blocks that we can use to build larger structures.

Using nREPL as System Interface

Sat, 26 Nov 2022 00:00:00 +0000

Clojure REPL is a powerful tool for developing programs interactively. Connecting the editor to the REPL allows us to get instant feedback on the code we're writing and have confidence that it works as intended as we're developing the application. However, the REPL isn't inherently limited to application development. It provides us with an interface to the language, and the language in turn be used as an interface to the host system. Let's take a look at how we can use Babashka along with Osquery to inspect the state of the host.

Osquery is a handy tool that allows using SQL commands in order to leverage a relational data-model to describe a device. Different aspects of the system are mapped to relational tables using the following schema. The tables give us access to files, ports, mounts, and many other aspects of the system. One aspect of Osqeury that's particularly useful to us is that it's able to return results in JSON format that we can parse into EDN and work with as structured data in the REPL.

To see how this works we'll start the nREPL server by running bb --nrepl-server. The REPL will start on port 1667 by default, we can also set a custom port by providing port number as the second argument to bb. Once the REPL is running we can connect any nREPL compatible editor such as Calva or Emacs.

Let's create a file called osquery.clj and open it in the editor and add some code to drive Osquery. First thing we'll need to do is to require the namespaces for interacting with the shell and parsing JSON:

(require '[clojure.java.shell :refer [sh]]
         '[cheshire.core :as json])

Next, we'll define the osquery function that will take a SQL query as text, execute osqueryi command and return its result as EDN:

(defn osquery [query]
  (let [{:keys [exit out err]} (sh "osqueryi" "--json" query)]
    (if (zero? exit)
      (json/decode out true)
      (throw (Exception. err)))))

And we're now ready to try to query some information about the system. Let's run a query to see all the routes where destination is ::1

(osquery "select * from routes where destination = '::1'")

We should get back a list of routes that looks something like the following:

({:hopcount "0",
  :interface "lo0",
  :mtu "16384",
  :type "local",
  :source "",
  :gateway "::1",
  :netmask "128",
  :flags "2098181",
  :destination "::1",
  :metric "0"})

The result is just a plain Clojure data structure we can trivially manipulate using full power of Clojure.

We can even go a step further using HoneySQL library that will allow us to make structured queries. We'll need to require deps and pull in the library as follows:

(require '[babashka.deps :as deps])

(deps/add-deps '{:deps {com.github.seancorfield/honeysql {:mvn/version "2.2.861"}}})

(require '[honey.sql :as hsql])

Finally, we'll update our osquery function as follows:

(defn osquery [query]
  (let [{:keys [exit out err]} (apply sh "osqueryi" "--json" (hsql/format query {:inline true}))]
    (if (zero? exit)
      (json/decode out true) 
      (throw (Exception. err)))))

With the changes above we can now write our queries in EDN:

(osquery {:select [:*] :from [:routes] :where [:= :destination "::1"]})

I hope this little example illustrates how the REPL can be used as a powerful OS interaction tool as well as a programming tool and inspires you to use the REPL in new and exciting ways. Babashka in particular is a great tool for such REPL driven interaction due to fast startup and wide range of useful libraries that let us access databases, HTTP servers, and other resources. This makes Babashka an excellent tool for doing devops.

Introducing Kit Framework

Sat, 08 Jan 2022 00:00:00 +0000

Kit is a Clojure web framework building on experience from Luminus while embracing latest tools and best practices that emerged over the years. Kit shares the same goals as Luminus while aiming to address its deficiencies. Before we dive into Kit, let's take a moment to establish some background. Kit was created as a collaboration between Nik, Dan, and myself. The project leverages our collective experience developing web applications using Clojure. Nik provides his own rationale for the project here.

Background

My original motivation behind Luminus was to provide a frictionless beginner experience for Clojure web development. Clojure community favors structuring applications by leveraging composable libraries. This approach affords gives the developer full control over the structure of their project making it easy for experienced Clojure users to build lean applications that contain only the necessary components.

Unfortunately, having to know what libraries and tools to use and how to put them together effectively creates a significant barrier for Clojure beginners. This is one of the major problems addressed by frameworks where such decisions are made by the maintainers of the framework.

Frameworks allow users to focus on the business logic of their application while aiming to handle all the cross-cutting concerns. Downside of this approach is that the framework has to accommodate for many different types of projects. Users of the framework end up inheriting the complexity of the entire framework regardless of the actual needs of their project.

Luminus provides a middle ground between these approaches using project templates. The template makes all the decisions regarding the libraries that are used and the structure of the project. Using such a template allows users to create a skeleton project that works out of the box. However, the user is free to modify the code in the project any way they see fit without being locked into the decisions made by the maintainers of the framework.

Just like a traditional framework, Luminus provides a curated stack of libraries that are known to be maintained and to work well together. This stack is coupled with a documentation site that illustrates how to accomplish common tasks such as HTML templating, routing, and database access.

One major deficiency of Luminus is that it's based around Leiningen templates. Any template features that the user wants to use in their project have to be known at project creation time. For example, if you created a barebones project and then later decided that you wanted to add ClojureScript support, then all the wiring would have to be done manually.

Another problem with baking all the features directly into the template is the resulting maintenance overhead. Any features supported by Luminus have to be maintained in the official repository. As the number of supported features grows so does the maintenance burden.

With all that in mind, let's take a look at what Kit does differently and what improvements it introduces over Luminus.

Kit Stack

Kit stack is similar to Luminus with a few notable changes. Let's see what they are and explore the rationale behind them.

Kit uses Integrant to manage stateful components in the project. While Mount is a great library for managing stateful components, it doesn't lend itself well towards creating modules since the component is described as code within the project. On the other hand, Integrant uses an EDN configuration file for managing state making it easy to package component configuration in the modules. Integrant also follows data oriented approach favored by Clojure community where the entire system of components and their relationships is described as a map. This makes it easy to see what all the resources and their relationships in the project are at a glance.

Another major change is that tools.deps along with tools.build are used for managing project lifecycle in favor of Leiningen. While Leiningen is a fine tool, it's clear that the community is moving towards using official tooling and Kit embraces this decision. Using official tools also means that there is one less tool to install, making for a smoother beginner experience.

Aside from these changes, the stack and project structure will be familiar to existing Luminus users. Ring is used as the HTTP server abstraction, Reitit being used for routing, Selmer for HTML templating, Migratus for migrations, and HugSQL for managing SQL queries.

With that out of the way, let's take look at the most exciting aspect of the framework which is its module system.

Kit Modules

Kit modules are templates that can be applied to an existing project using kit-generator library. Modules are managed using git repositories, and official modules can be found here. Let's take a brief look at what a module repository looks like.

A module repository must contain a modules.edn file describing the modules that are provided. For example, here are the official modules provided by Kit:

{:name "kit-modules"
 :modules
 {:kit/html
  {:path "html"
   :doc "adds support for HTML templating using Selmer"}
  :kit/sqlite
  {:path "sqlite"
   :doc "adds support for SQLite embedded database"}
  :kit/cljs
  {:path "cljs"
   :doc "adds support for cljs using shadow-cljs"}}}

As we can see above, the official repository contains three modules. Let's take a look at the :kit/html module to see how it works. This module contains a config.edn file and a folder called assets. Let's take a look at the configuration for the module:

{:default
 {:require-restart? true
  :actions
  {:assets           [["assets/home.html"    "resources/html/home.html"]
                      ["assets/error.html"    "resources/html/error.html"]
                      ["assets/css/screen.css"    "resources/public/css/screen.css"]
                      ["assets/img/kit.png" "resources/public/img/kit.png"]
                      ["assets/src/pages.clj"    "src/clj/<<sanitized>>/web/routes/pages.clj"]
                      ["assets/src/layout.clj"   "src/clj/<<sanitized>>/web/pages/layout.clj"]]
   :injections       [{:type   :edn
                       :path   "resources/system.edn"
                       :target []
                       :action :merge
                       :value  {:reitit.routes/pages
                          {:base-path ""
                             :env       #ig/ref :system/env}}}
                      {:type   :edn
                       :path   "deps.edn"
                       :target [:deps]
                       :action :merge
                       :value  {selmer/selmer {:mvn/version "1.12.49"}
                                luminus/ring-ttl-session {:mvn/version "0.3.3"}}}
                      {:type   :clj
                       :path   "src/clj/<<sanitized>>/core.clj"
                       :action :append-requires
                       :value  ["[<<ns-name>>.web.routes.pages]"]}]}}}

We can see that the module has a :default profile. Kit module profiles allow providing variations of a module with different configurations. For example, a database module could have different profiles for different types of databases. In case of HTML, we only need a single profile.

The:require-restart? key specifies that the runtime needs to be restarted for changes to take effect. This is necessary for modules that add Maven dependencies necessitating JVM restarts to be loaded.

Next, the module specifies the actions that will be performed. The first action called :assets specifies new assets that will be added to the project. These are template files that will be read from the assets folder and injected in the project. Assets are akin to traditional Leiningen templates.

The other configuration action is called :injections and specifies code that will be injected into existing files within the project. In order to provide support for rendering HTML templates, the module must update Integrant system configuration by adding a reference for new routes to system.edn, add new dependencies to deps.edn, and finally require the namespace that contains the routes for the pages in the core namespace of the project.

Trying Things Out

Let's create a new Kit project and see how this all works in practice. Kit uses clj-new templates for instantiating the project, make sure you have it installed locally to follow along. Let's create a project called guestbook by running the following command:

clojure -X:new :template io.github.kit-clj :name kit/guestbook

Once the project is created, you can start the REPL by running clj -M:dev -M:repl, alternatively if you have make installed just run make repl instead. Once the REPL starts, you can run (go) to start the HTTP server.

Default project provides a minimal configuration with a health status API located at http://localhost:3000/api/health. Let's see how we can add support for rendering HTML pages using Selmer by installing the official HTML module.

Adding Modules

Kit projects use a configuration file called kit.edn that specifies some metadata about the project and allows the user to reference module repositories. Default configuration will look something like the following:

{:full-name "kit/guestbook"
 :ns-name   "kit.guestbook"
 :sanitized "kit/guestbook"
 :name      "guestbook"
 :modules   {:root         "modules"
             :repositories [{:url  "git@github.com:kit-clj/modules.git"
                             :tag  "master"
                             :name "kit-modules"}]}}

REPL driven workflow

Kit embraces the REPL and the generator library is aliased in the user namespace as kit. Let's see how we can us it to install HTML module in the project. First, we'd need to sync our module repositories. This is done by running the following command in the REPL:

user=> (kit/sync-modules)
2021-11-30 11:42:41,010 [main] DEBUG org.eclipse.jgit.util.FS - readpipe [git, --version],/usr/local/bin
2021-11-30 11:42:41,030 [main] DEBUG org.eclipse.jgit.util.FS - readpipe may return 'git version 2.33.1'
2021-11-30 11:42:41,030 [main] DEBUG org.eclipse.jgit.util.FS - remaining output:
...
2021-11-30 11:42:41,769 [main] DEBUG o.e.jgit.transport.PacketLineOut - git> 0000
2021-11-30 11:42:41,769 [main] DEBUG o.e.jgit.transport.PacketLineOut - git> done

2021-11-30 11:42:41,835 [main] DEBUG o.e.jgit.transport.PacketLineIn - git< NAK
nil
user=>

Once the modules are synchronized, we can list the available modules by running kit/list-modules:

user=> (kit/list-modules)
:kit/html - adds support for HTML templating using Selmer
:kit/sqlite - adds support for SQLite embedded database
:kit/cljs - adds support for cljs using shadow-cljs
nil
user=>

We can see that the three modules specified in the official modules repository are now available for use. Let's install the HTML module by running kit/install-module function and passing it the keyword specifying the module name:

user=> (kit/install-module :kit/html)
updating file: resources/system.edn
updating file: deps.edn
updating file: src/clj/kit/guestbook/core.clj
applying
 action: :append-requires
 value: ["[kit.guestbook.web.routes.pages]"]
:kit/html installed successfully!
restart required!
nil
user=>

Let's restart the REPL and run (go) command again to start the application. We should now be able to navigate to http://localhost:3000 and see the default HTML page provided by the module.

Generator aims to be idempotent, and will err on the side of safety in case of conflicts. For example, if we attempt to install :kit/html module a second time then we'll see he following output:

user=> (kit/install-module :kit/html)
:kit/html requires following modules: nil
module :kit/html is already installed!
nil
user=>

Generator lets us know that the module already exists and there is nothing to be done.

Conclusion

I hope this post convinced you that Kit approach is an improvement over Luminus for both users and developers. With Kit, you no longer have to know what features you're going to be using up front. New functionality can now be added gradually as you discover the need for it. You're no longer restricted to using official modules either. Anyone can make a repository with their own modules that template common functionality that they need and use these along side or even in place of the official modules. For example, if you wanted to use Hiccup instead of Selmer, then you could trivially add support for that yourself based on the example above.

Kit was created by Dmitri Sotnikov, Nikola Peric, and Dan Boykis. We hope that this project will make it easier for Clojure developers to make web applications going forward. Get in touch with us on Clojurians slack at #kit-clj, we're excited to hear community feedback, ideas and suggestions for the project.

Advantages of Data Oriented Programming

Wed, 08 Apr 2020 00:00:00 +0000

One of the biggest advantages I've found working with Clojure is its data oriented nature. Ultimately, all that code is doing is transform data. A program starts with one piece of data as the input and produces another as its output. Mainstream languages attempt to abstract over that using object oriented semantics. While practical value of such abstraction is not entirely clear, there are some tangible problems associated with this approach. Let's take a look at some drawbacks to structuring programs using OO style.

Traditionally, an object can be thought of as a type of a state machine that contains some data fields representing its internal state and provides some methods for manipulating it. An object represents the smallest compositional unit in OO, and a program is structured as a graphs of such objects that interact with one another by manipulating each others state.

The first problem we have is that each object is an ad hoc DSL. When a developer designs an object they define its API in form of methods and come up with the behaviors the object will have. This makes each object unique, and knowing how one object behaves tells you nothing regarding how the next object might behave. Rich Hickey illustrates this point in detail in his Clojure, Made Simple talk. The more objects you define the more behaviors you have to keep in your head. Thus, cognitive overhead grows proportionally with the size of the program.

Any mutable objects present in the program require the developer to know the state of the objects in order to know how the program will behave. A program that is structured as a graph of interdependent state machines quickly becomes impossible to reason about. The problem stems from objects being implicitly connected via references to each other resulting in shared mutable state. This leads to lack or referential transparency and makes it impossible to do local reasoning about the code. In order to tell what a piece of code is doing you also have to track down all the code that shares references with the code you're reading.

This is one reason why sophisticated debugging tools are needed to work with code effectively in object oriented languages. The only way to tell what's happening in a large program is to run it in a debugger, try to put it in a particular state and then inspect it. Unfortunately, this approach is just a heuristic since there may be many different paths that get you to a particular state, and it's impossible to guarantee that you've covered them all.

Another notable problem with objects is that there is no standard way for serializing them creating additional pain at program boundaries. For example, we can't just take an object graph on from a web server and send it to the client. We must write custom serializers for every object adding complexity and boilerplate to our programs. A related problem occurs when composing libraries that define their own classes leading to prevalence of wrapper and adapter patterns.

All these problems disappear in a data oriented language like Clojure. Modern FP style embraces the fact that programs can be viewed as data transformation pipelines where input data is passed through a series of pure functions to transform it into desired output. Such functions can be reasoned about in isolation without having to consider the rest of the program. Plain data doesn't have any hidden behaviors or state that you have to worry about. Immutable data is both transparent and inert while objects are opaque and stateful.

As a concrete example, Pedestal HTTP server has around 18,000 lines of code, and 96% of it is pure functions. All the IO and side effects are encapsulated in the remaining 4% of the code. This has been a common scenario for the vast majority of Clojure programs I've worked on.

Cognitive overhead associated with reasoning about code is localized as opposed to being directly influenced by the size of the application as often happens with OO. Each function can be thought of as an small individual program, and we simply pipe these programs together to solve bigger problems. Incidentally, this is the exact same approach as advocated by Ken Thompson in Unix philosophy.

Data can also be passed across program boundaries since it's directly serializable. A Clojure web server can send its output directly to the client, and client code can operate on this data without any additional ceremony. I've discussed some of the practical benefits that stem from having standard serialization semantics in this presentation.

Another advantage of separating data from logic is code reuse. A pure function that transforms one piece of data into another can be used in any context. A common set of functions from the standard library can be used to manipulate data regardless where it comes from. Once you learn a few common patterns for transforming data, you can apply these patterns everywhere.

I strongly suspect that data driven APIs are a major reason why Clojure libraries tend to be so stable. When a library is simply transforming data then it's possible to get to a state where it's truly done. Once the API consisting of all the supported transformations has been defined and tested, then the API is complete. The only times the library has to be revisited is when use cases missed by tests are discovered or new features are added. This tends to happen early on during library lifecycle, and hence mature libraries need little attention from their maintainers.

Of course, this is not to say that large software cannot be written effectively using OO languages. Clearly plenty of great software has been produced using these techniques. However, the fact that complex applications can be written in a particular fashion is hardly interesting of itself. Given enough dedication and ingenuity it's possible to write complex software in any language. It's more useful to consider how different approaches impact development style in different languages.

Running Luminus on Dokku

Sat, 19 Jan 2019 00:00:00 +0000

Luminus provides a great way to get up and running with a Clojure web application. However, building your app is only half the work. Once you've got your app working, the next step is to host it somewhere so that the users can access it.

Cloud platforms, such as AWS, are a popular choice for deploying large scale solutions. On the other hand, VPS services like Digital Ocean and Linode provide a more economical alternative for small scale applications. The downside of running your own VPS is that managing it can be labor intensive. This is where Dokku comes in. It's a private PaaS modelled on Heroku that you can use to provision a VPS.

Let's take a look at what's involved in provisioning a Digital Ocean droplet with Dokku and deploying a Luminus web app to it.

Set up the server

Let's create a droplet with Ubuntu LTS (18.0.4 at the time of writing) and SSH into it. We'll need to add new APT repositories before we install Dokku.

add the universe repository sudo add-apt-repository universe
add the key wget -nv -O - https://packagecloud.io/dokku/dokku/gpgkey | apt-key add -
add the Dokku repo echo "deb https://packagecloud.io/dokku/dokku/ubuntu/ bionic main" > /etc/apt/sources.list.d/dokku.list

Once the repositories are added, we'll need to update the dependencies and install Dokku.

update dependencies sudo apt-get update && sudo apt-get upgrade
install dokku apt-get install dokku

Once Dokku is installed, we'll create an application and a Postgres database instance.

create the app dokku apps:create myapp
install dokku-postgres plugin sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git
create the db dokku postgres:create mydb
link the db to the app dokku postgres:link mydb myapp

We're now ready to deploy the app.

Create a new Luminus application

Let's create a Luminus application on your local machine.

lein new luminus myapp +postgres
cd myapp

Let's update the app to run migrations on startup by updating the myapp.core/start-app function to run the migrations.

(defn start-app [args]
  (doseq [component (-> args
                        (parse-opts cli-options)
                        mount/start-with-args
                        :started)]
    (log/info component "started"))

  ;;run migrations  
  (migrations/migrate ["migrate"] (select-keys env [:database-url]))

  (.addShutdownHook (Runtime/getRuntime) (Thread. stop-app)))

Next, we need to update env/prod/resources/logback.xml to use STDOUT for the logs:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <statusListener class="ch.qos.logback.core.status.NopStatusListener" />
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <!-- encoders are assigned the type
             ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
        <encoder>
            <charset>UTF-8</charset>
            <pattern>%date{ISO8601} [%thread] %-5level %logger{36} - %msg %n</pattern>
        </encoder>
    </appender>
    <logger name="org.apache.http" level="warn" />
    <logger name="org.xnio.nio" level="warn" />
    <logger name="com.zaxxer.hikari" level="warn" />
    <root level="INFO">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

Deploy the application to Dokku

We're now ready to deploy the app. First, we'll need to create a Git repo and add the app contents to it.

git init
git add .gitignore Procfile project.clj README.md src/* env/* test/* resources/*
git commit -a -m "initial commit"

Note that you do not want to check in Dockerfile that's generated by the template. Dokku will use it as the preferred strategy for creating the container.

Next, we'll add the remote for the Dokku repository on the server and push the project to the remote. Dokku will automatically build the project once it's pushed, and deploy the application when the build is successful.

git remote add dokku dokku@<server name>:myapp
git push dokku master

The app will be pushed to the server where it will be compiled and run. If everything went well you should see output that looks something like the following:

...
-----> Building with Leiningen
       Running: lein uberjar
       Compiling sample.app
       2019-01-18 01:10:30.857:INFO::main: Logging initialized @6674ms to org.eclipse.jetty.util.log.StdErrLog
       Created /tmp/build/target/myapp-1.0.1.jar
       Created /tmp/build/target/myapp.jar
...
=====> web=1
...
-----> Waiting for 10 seconds ...
-----> Default container check successful!
-----> Running post-deploy
-----> Configuring myapp.<server name>...(using built-in template)
-----> Creating http nginx.conf
-----> Running nginx-pre-reload
       Reloading nginx
-----> Setting config vars
       DOKKU_APP_RESTORE:  1
=====> 8dc31ac11011111117f71e4311111ca5962cf316411d5f0125e87bbac26
=====> Application deployed:
       http://myapp.<server name>

To http://<server name>:myapp
   6dcab39..1c0c8b7  master -> master

We can check the status of the application in the logs by running dokku logs myapp command on the server. The output should looks something like the following.

Picked up JAVA_TOOL_OPTIONS: -Xmx300m -Xss512k -XX:CICompilerCount=2 -Dfile.encoding=UTF-8
2019-01-19 19:09:48,258 [main] INFO  myapp.env -
-=[myapp started successfully]=-
2019-01-19 19:09:50,490 [main] INFO  luminus.http-server - starting HTTP server on port 5000
2019-01-19 19:09:50,628 [main] INFO  org.xnio - XNIO version 3.3.6.Final
2019-01-19 19:09:51,236 [main] INFO  org.projectodd.wunderboss.web.Web - Registered web context /
2019-01-19 19:09:51,242 [main] INFO  myapp.core - #'myapp.config/env started
2019-01-19 19:09:51,243 [main] INFO  myapp.core - #'myapp.db.core/*db* started
2019-01-19 19:09:51,243 [main] INFO  myapp.core - #'myapp.handler/init-app started
2019-01-19 19:09:51,244 [main] INFO  myapp.core - #'myapp.handler/app started
2019-01-19 19:09:51,249 [main] INFO  myapp.core - #'myapp.core/http-server started
2019-01-19 19:09:51,249 [main] INFO  myapp.core - #'myapp.core/repl-server started
2019-01-19 19:09:51,250 [main] INFO  myapp.core - running migrations
2019-01-19 19:09:51,257 [main] INFO  migratus.core - Starting migrations
2019-01-19 19:09:51,418 [main] INFO  migratus.database - creating migration table 'schema_migrations'
2019-01-19 19:09:51,992 [main] INFO  migratus.core - Running up for [20190118214013]
2019-01-19 19:09:51,997 [main] INFO  migratus.core - Up 20190118214013-add-users-table
2019-01-19 19:09:52,099 [main] INFO  migratus.core - Ending migrations

You should now be able to check your application in the browser by navigating to http://<server name>.

Troubleshooting the database

The startup logs for the application indicate that it was able to connect to the database and run the migrations successfully. Let's confirm this is the case by connecting a psql shell to the database container on the server.

dokku postgres:connect mydb
mydb=# \d
               List of relations
 Schema |       Name        | Type  |  Owner
--------+-------------------+-------+----------
 public | schema_migrations | table | postgres
 public | users             | table | postgres
(2 rows)

We can see that the database contains the schema_migrations table and the users table that were created when the app migrations ran.

Sometimes it might be useful to connect a more advanced client such as DBeaver. This can done by exposing the database on the server using the following command.

sudo dokku postgres:expose mydb 5000

Next, we'll enter the container for the application to get the database connection details.

dokku enter myapp web
echo $DATABASE_URL

The DATABASE_URL environment variable in the container will contain the connection string that looks as follows.

postgres://postgres:<password>@dokku-postgres-mydb:5432/mydb

We can now map the port to the local machine using SSH, and connect to the database as if it was running on the local machine using the connection settings above.

ssh -L 5432:localhost:5000 <server name>

Set up HTTPS using Let's Encrypt

As the last step we'll set up HTTPS for the application using dokku-letsencrypt plugin. We'll set the app to run on the root domain on the server.

add the root domain to the app dokku domains:add myapp <server name>
remove the subdomain from the app dokku domains:remove myapp myapp.<server name>
install the plugin sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git
set the email for renewal warnings dokku config:set --no-restart myapp DOKKU_LETSENCRYPT_EMAIL=<your email>
add HTTPS to the app sudo dokku letsencrypt myapp
set up auto-renew for the certificate dokku letsencrypt:auto-renew

That's all there is to it. The application is now deployed to the droplet, it's hooked up to the database, and it's using Let's Encrypt SSL/TLS Certificates.

Any further updates to the application simply involve committing the changes to the local Git repo and pushing them to the server as we did with our initial deploy.

I recommend taking look at the official documentation on the Dokku site for more information about Dokku. I think it provides an excellent solution for running your VPS. If you're evaluating different options for deploying your Clojure apps give Dokku a look.

Capturing ClojureScript Errors on the Server

Fri, 12 Jan 2018 00:00:00 +0000

Logging errors is an important aspect of writing real-world applications. When something goes wrong at runtime it's very helpful to have a log detailing what went wrong in order to fix the problem. This is a straightforward process when we're working on the backend code. We can catch the exception and log it along with the stack trace. However, we need to get a bit more creative in order to handle client-side errors.

In this post we'll take a look at propagating errors from a Reagent based app back to the server. A naive implementation might look something like the following. We'll write a function that accepts an event containing the error, then send the error message along with the stack trace to the server:

(defn report-error! [event]
  (let [error (.-error event)
        message (.-message error)
        stacktrace (.-stack error)]    
    (ajax/POST "/error"
               {:headers
                {"x-csrf-token"
                 (.-value (js/document.getElementById "__anti-forgery-token"))}
                :params
                {:message     message
                 :stacktrace stacktrace}})))

Next, we'll set the report-error! function as the global error event listener:

(defn init! []
  (.addEventListener js/window "error" report-error!)
  (reagent/render [home-page] (.getElementById js/document "app")))

The home-page function will render a button that will throw an error when it's clicked:

(defn home-page []
  [:div>h2 "Error Test"
   [:div>button
    {:on-click #(throw (js/Error. "I'm an error"))}
    "throw an error"]])

If we pop up the console in the browser we should see something like the following there:

Uncaught Error: I'm an error
    at app.core.home_page (core.cljs:25)
    at Object.ReactErrorUtils.invokeGuardedCallback (react-dom.inc.js:9073)
    at executeDispatch (react-dom.inc.js:3031)
    at Object.executeDispatchesInOrder (react-dom.inc.js:3054)
    at executeDispatchesAndRelease (react-dom.inc.js:2456)
    at executeDispatchesAndReleaseTopLevel (react-dom.inc.js:2467)
    at Array.forEach (<anonymous>)
    at forEachAccumulated (react-dom.inc.js:15515)
    at Object.processEventQueue (react-dom.inc.js:2670)
    at runEventQueueInBatch (react-dom.inc.js:9097)

This gives us the namespace and the line number in the ClojureScript source that caused the error. However, if we print the message that we received on the server it will look as follows:

 Error: I'm an error
    at app.core.home_page (http://localhost:3000/js/out/app/core.js:51:8)
    at Object.ReactErrorUtils.invokeGuardedCallback (http://localhost:3000/js/out/cljsjs/react-dom/development/react-dom.inc.js:9073:16)
    at executeDispatch (http://localhost:3000/js/out/cljsjs/react-dom/development/react-dom.inc.js:3031:21)
    at Object.executeDispatchesInOrder (http://localhost:3000/js/out/cljsjs/react-dom/development/react-dom.inc.js:3054:5)
    at executeDispatchesAndRelease (http://localhost:3000/js/out/cljsjs/react-dom/development/react-dom.inc.js:2456:22)
    at executeDispatchesAndReleaseTopLevel (http://localhost:3000/js/out/cljsjs/react-dom/development/react-dom.inc.js:2467:10)
    at Array.forEach (<anonymous>)
    at forEachAccumulated (http://localhost:3000/js/out/cljsjs/react-dom/development/react-dom.inc.js:15515:9)
    at Object.processEventQueue (http://localhost:3000/js/out/cljsjs/react-dom/development/react-dom.inc.js:2670:7)
    at runEventQueueInBatch (http://localhost:3000/js/out/cljsjs/react-dom/development/react-dom.inc.js:9097:18)

The stack trace is there, but it's no longer source mapped. So we'll know what namespace caused the error, but not the line in question. In order to get a source mapped stack trace we'll have to use a library such as stacktrace.js. Unfortunately, we won't be able to use the new :npm-deps option in the ClojureScript compiler. This works as expected when :optimizations are set to :none, but fails to provide us with the source mapped stack trace in the :advanced mode.

Instead, we'll use the WebJars dependency along with the ring-webjars middleware:

:dependencies
[...
 [ring-webjars "0.2.0"]
 [org.webjars.bower/stacktrace-js "2.0.0"]]

The middleware uses the /assets/<webjar>/<asset path> pattern to load the resources packaged in WebJars dependencies. Here's how this would look for loading the stacktrace-js resource.

We'll require the middleware:

(ns app.handler
 (:require
  ...
  [ring.middleware.webjars :refer [wrap-webjars]]))

Wrap the Ring handler with it:

 (defn -main []
  (run-jetty
   (-> handler
       (wrap-webjars)
       (wrap-defaults site-defaults))
   {:port 3000 :join? false}))

The stacktrace.min.js file packaged in the org.webjars.bower/stacktrace-js dependency will be available as a resource at the following path /assets/stacktrace-js/dist/stacktrace.min.js:

(defroutes handler
  (GET "/" []
    (html5
      [:head
       [:meta {:charset "utf-8"}]
       (anti-forgery-field)]
      [:body
       [:div#app]
        (include-js "/assets/stacktrace-js/dist/stacktrace.min.js"
                    "/js/app.js")]))
  
  (POST "/error" {:keys [body]}
    (let [{:keys [message stacktrace]}
          (-> body
              (transit/reader :json)
              (transit/read))]
      (println "Client error:" message "\n" stacktrace))
    "ok")
  
  (resources "/")
  (not-found "Not Found"))

Finally, the ClojureScript compiler configuration will look as follows:

{:output-dir "target/cljsbuild/public/js"
 :output-to  "target/cljsbuild/public/js/app.js"
 :source-map "target/cljsbuild/public/js/app.js.map"
 :optimizations :advanced
 :infer-externs true
 :closure-warnings {:externs-validation :off
                    :non-standard-jsdoc :off}}

We need to specify the name of the source map file when using the advanced optimization, tell the compiler to infer the externs, and optionally suppress the warnings.

The new version of the report-error! function will look similar to the original, except that we'll now be passing the error to the StackTrace.fromError function. This function returns a promise containing the source mapped stack trace that we'll be sending to the server:

(defn report-error! [event]
  (let [error (.-error event)]
    (-> (js/StackTrace.fromError error)
        (.then
         (fn [stacktrace]
           (ajax/POST "/error"
                      {:headers
                       {"x-csrf-token"
                        (.-value (js/document.getElementById "__anti-forgery-token"))}
                       :params
                       {:message    (.-message error)
                        :stacktrace (->> stacktrace
                                          (mapv #(.toString %))
                                          (string/join "\n "))}}))))))

This time around we should see the source mapped error on the server with all the information that we need:

I'm an error
 Error()@http://localhost:3000/js/app/core.cljs:27:23
 mountComponent()@http://localhost:3000/js/app.js:40:5631
 focusDOMComponent()@http://localhost:3000/js/app.js:38:22373
 focusDOMComponent()@http://localhost:3000/js/app.js:38:22588
 focusDOMComponent()@http://localhost:3000/js/app.js:38:18970
 focusDOMComponent()@http://localhost:3000/js/app.js:38:19096
 didPutListener()@http://localhost:3000/js/app.js:41:12120
 focusDOMComponent()@http://localhost:3000/js/app.js:38:20154
 mountComponent()@http://localhost:3000/js/app.js:40:5880

We can see that the error occurred on line 27 of the app.core namespace which is indeed where the code that throws the exception resides. The full listing for the example is available on GitHub.

While the example in this post illustrates bare bones exception handling, we can do more interesting things in a real world application. For example, re-frame based application could send the entire state of the re-frame database at the time of the error to the server. This allows us to put the application in the exact state that caused the error when debugging the problem.

Moving Forward by Letting Go

Tue, 03 Oct 2017 00:00:00 +0000

Lisps are famous for having powerful metaprogramming facilities derived from their homoiconic nature. Core language consists of a small set of built in primitives, and the rest is implemented using macros and functions in the standard library.

Since these same tools are available to the users, anybody can easily extend the language to add semantics for their problem domain. This is the aspect of macros that's discussed most often.

While this is certainly the case, Clojure community tends to discourage using macros if they can be avoided. The rationale behind this being that macros introduce additional complexity, and Clojure values simplicity. I generally agree with this sentiment, and I find that I tend to use macros sparingly myself. However, saying that macros shouldn’t be overused is not the same as saying they shouldn’t be used at all.

One place where macros work well is library APIs. Libraries that express a particular domain can create a DSL in their API that cleanly maps to that domain. Compojure, Specter, and Clara Rules are great examples of effective macro use. Such libraries are a powerful demonstration of users extending semantics of the language.

Since most ideas can be expressed via libraries, it becomes possible to experiment with different approaches to solving problems without modifying the core language. Extending the language through libraries has the advantage of keeping these extensions contained. It also lets them fade away when you no longer use them.

Clojure has been around for a decade now, and the core language hasn't changed much in that time. Some new features have been added, most notably reducers and transducers, but overall the language has stayed small and focused. In fact, I've even seen concerns that Clojure is stagnating because features aren't being added at the rate of other languages.

The idea of using libraries to add features is used by the Clojure core team as well. Consider the example of the core.async library that brings Go channel semantics and the CSP programming model to Clojure.

Perhaps, in time a better idea will come along, and core.async library will become deprecated. At that point developers would stop using the library and move on to use whatever happens to replace it.

Meanwhile, existing projects will not be affected as they can continue using the library. The community will move on, and most people won't have to learn about core.async semantics. This cycle may happen many times with many different ideas, without any baggage being accumulate by the language itself.

Unfortunately, mainstream languages are designed in a way where it's not practical to add new features without updating the language specification to accommodate them. Popular languages such as Java, Python, and JavaScript have accumulated a lot of complexity over the years.

As usage patterns change, new features are being added, while existing features become deprecated. Removing features is difficult since many projects in the wild end up relying on them, so they're typically left in place.

Having lots of features in a language can seem like a positive at first glance, but in practice, features often turn into a mental burden for the developer. Eventually languages become too large to understand fully, and developers settle on a subset of the features considered to be the current best practice.

In my opinion, this is the real power of homoiconicity. A language that can be extended in user space can evolve without accumulating baggage. New ideas can be implemented as libraries, and later discarded when better ideas come along. The end result is a small and focused language that doesn't sacrifice flexibility.

Comparing Reagent to React.js and Vue.js for dynamic tabular data

Sun, 26 Mar 2017 00:00:00 +0000

I recently ran across a comparison of React.js to Vue.js for rendering dynamic tabular data, and I got curious to see how Reagent would stack up against them.

The benchmark simulates a view of football games represented by a table. Each row in the table represents the state of a particular game. The game states are updated once a second triggering UI repaints.

I structured the application similarly to the way that React.js version was structured in the original benchmark. The application has a football.data namespace to handle the business logic, and a football.core namespace to render the view.

Implementing the Business Logic

Let's start by implementing the business logic in the football.data namespace. First, we'll need to provide a container to hold the state of the games. To do that we'll create a Reagent atom called games:

(ns football.data
  (:require [reagent.core :as reagent]))

(defonce games (reagent/atom nil))

Next, we'll add a function to generate the fake players:

(defn generate-fake-player []
  {:name               (-> js/faker .-name (.findName))
   :effort-level       (rand-int 10)
   :invited-next-week? (> (rand) 0.5)})

You can see that we're using JavaScript interop to leverage the Faker.js library for generating the player names. One nice aspect of working with ClojureScript is that JavaScript interop tends to be seamless as seen in the code above.

Now that we have a way to generate the players, let's add a function to generate fake games:

(defn generate-fake-game []
  {:id                 (-> js/faker .-random (.uuid))
   :clock              0
   :score              {:home 0 :away 0}
   :teams              {:home (-> js/faker .-address (.city))
                        :away (-> js/faker .-address (.city))}
   :outrageous-tackles 0
   :cards              {:yellow 0 :read 0}
   :players            (mapv generate-fake-player (range 4))})

With the functions to generate the players and the games in place, we'll now add a function to generate a set of initial game states:

(defn generate-games [game-count]
  (reset! games (mapv generate-fake-game (range game-count))))

The next step is to write the functions to update the games and players to simulate the progression of the games. This code translates pretty much directly from the JavaScript version:

(defn maybe-update [game prob path f]
  (if (< (rand-int 100) prob)
    (update-in game path f)
    game))

(defn update-rand-player [game idx]
  (-> game
      (assoc-in [:players idx :effort-level] (rand-int 10))
      (assoc-in [:players idx :invited-next-week?] (> (rand) 0.5))))

(defn update-game [game]
  (-> game
      (update :clock inc)
      (maybe-update 5 [:score :home] inc)
      (maybe-update 5 [:score :away] inc)
      (maybe-update 8 [:cards :yellow] inc)
      (maybe-update 2 [:cards :red] inc)
      (maybe-update 10 [:outrageous-tackles] inc)
      (update-rand-player (rand-int 4))))

The last thing we need to do is to add the functions to update the game states at a specified interval. The original code uses Rx.js to accomplish this, but it's just as easy to do using the setTimeout function with Reagent:

(defn update-game-at-interval [interval idx]
  (swap! games update idx update-game)
  (js/setTimeout update-game-at-interval interval interval idx))

(def event-interval 1000)

(defn update-games [game-count]
  (dotimes [i game-count]
    (swap! games update i update-game)
    (js/setTimeout #(update-game-at-interval event-interval i)
                   (* i event-interval))))

The update-games function updates the state of each game, then sets up a timeout for the recurring updates using the update-game-at-interval function.

Implementing the View

We're now ready to write the view portion of the application. We'll start by referencing the football.data namespace in the football.core namespace:

(ns football.core
  (:require
    [football.data :as data]
    [reagent.core :as reagent]))

Next, we'll write the components to display the players and the games:

(defn player-component [{:keys [name invited-next-week? effort-level]}]
  [:td
   [:div.player
    [:p.player__name
     [:span name]
     [:span.u-small (if invited-next-week? "Doing well" "Not coming again")]]
    [:div {:class-name (str "player__effort "
                            (if (< effort-level 5)
                              "player__effort--low"
                              "player__effort--high"))}]]])

(defn game-component [game]
  [:tr
   [:td.u-center (:clock game)]
   [:td.u-center (-> game :score :home) "-" (-> game :score :away)]
   [:td.cell--teams (-> game :teams :home) "-" (-> game :teams :away)]
   [:td.u-center (:outrageous-tackles game)]
   [:td
    [:div.cards
     [:div.cards__card.cards__card--yellow (-> game :cards :yellow)]
     [:div.cards__card.cards__card--red (-> game :cards :red)]]]
   (for [player (:players game)]
     ^{:key player}
     [player-component player])])

(defn games-component []
  [:tbody
   (for [game @data/games]
     ^{:key game}
     [game-component game])])

(defn games-table-component []
  [:table
   [:thead
    [:tr
     [:th {:width "50px"} "Clock"]
     [:th {:width "50px"} "Score"]
     [:th {:width "200px"} "Teams"]
     [:th "Outrageous Tackles"]
     [:th {:width "100px"} "Cards"]
     [:th {:width "100px"} "Players"]
     [:th {:width "100px"} ""]
     [:th {:width "100px"} ""]
     [:th {:width "100px"} ""]
     [:th {:width "100px"} ""]]]
   [games-component]])

You can see that HTML elements in Reagent components are represented using Clojure vectors and maps. Since s-expressions cleanly map to HTML, there's no need to use an additional DSL for that. You'll also notice that components can be nested within one another same way as plain HTML elements.

Noe thing to note is that the games-component dereferences the data/games atom using the @ notation. Dereferencing simply means that we'd like to view the current state of a mutable variable.

Reagent atoms are reactive, and listeners are created when the atoms are dereferenced. Whenever the state of the atom changes, any components that are observing the atom will be notified of the change.

In our case, changes in the state of the games atom will trigger the games-component function to be evaluated. The function will pass the current state of the games down to its child components, and this will trigger any necessary repaints in the UI.

Finally, we have a bit of code to create the root component represented by the home-page function, and initialize the application:

(defn home-page []
  [games-table-component])

(defn mount-root []
  (reagent/render [home-page] (.getElementById js/document "app")))

(def game-count 50)

(defn init! []
  (data/generate-games game-count)
  (data/update-games game-count)
  (mount-root))

We now have a naive implementation of the benchmark using Reagent. The entire project is available on GitHub. Next, let's take a look at how it performs.

Profiling with Chrome

When we profile the app in Chrome, we'll see the following results:

Here are the results for React.js and Vue.js running in the same environment for comparison:

As you can see, the naive Reagent version spends about double the time scripting compared to React.js, and about four times as long rendering.

The reason is that we're dereferencing the games atom at top level. This forces the top level component to be reevaluated whenever the sate of any game changes.

Reagent provides a mechanism for dealing with this problem in the form of cursors. A cursor allows subscribing to changes at a specified path within the atom. A component that dereferences a cursor will only be updated when the data the cursor points to changes. This allows us to granularly control what components will be repainted when a particular piece of data changes in the games atom. Let's update the view logic as follows:

(defn player-component [player]
  [:td
   [:div.player
    [:p.player__name
     [:span (:name @player)]
     [:span.u-small
      (if (:invited-next-week? @player)
        "Doing well" "Not coming again")]]
    [:div {:class-name (str "player__effort "
                            (if (< (:effort-level @player) 5)
                              "player__effort--low"
                              "player__effort--high"))}]]])

(defn game-component [game]
  [:tr
   [:td.u-center (:clock @game)]
   [:td.u-center (-> @game :score :home) "-" (-> @game :score :away)]
   [:td.cell--teams (-> @game :teams :home) "-" (-> @game :teams :away)]
   [:td.u-center (:outrageous-tackles @game)]
   [:td
    [:div.cards
     [:div.cards__card.cards__card--yellow (-> @game :cards :yellow)]
     [:div.cards__card.cards__card--red (-> @game :cards :red)]]]
   (for [idx (range (count (:players @game)))]
     ^{:key idx}
     [player-component (reagent/cursor game [:players idx])])])

(def game-count 50)

(defn games-component []
  [:tbody
   (for [idx (range game-count)]
     ^{:key idx}
     [game-component (reagent/cursor data/games [idx])])])

(defn games-table-component []
  [:table
   [:thead
    [:tr
     [:th {:width "50px"} "Clock"]
     [:th {:width "50px"} "Score"]
     [:th {:width "200px"} "Teams"]
     [:th "Outrageous Tackles"]
     [:th {:width "100px"} "Cards"]
     [:th {:width "100px"} "Players"]
     [:th {:width "100px"} ""]
     [:th {:width "100px"} ""]
     [:th {:width "100px"} ""]
     [:th {:width "100px"} ""]]]
   [games-component]])

(defn home-page []
  [games-table-component])

The above version creates a cursor for each game in the games-components. The game-component in turn creates a cursor for each player. This way only the components that actually need updating end up being rendered as the state of the games is updated. Let's profile the application again to see how much impact this has on its performance:

The performance of the Reagent code using cursors now looks similar to that of the Vue.js implementation. You can see the entire source for the updated version here.

Conclusion

In this post we saw that ClojureScript with Reagent provides a compelling alternative to JavaScript offerings such as React.js and Vue.js.

Reagent allows writing succinct solutions that perform as well as those implemented using native JavaScript libraries. It also provides us with tools to intuitively reason about what parts of the view are going to be updated.

Likewise, we get many benefits by simply switching from using JavaScript to ClojureScript.

For example, We already saw that we didn't need any additional syntax, such as JSX, to represent HTML elements. Since HTML templates are represented using regular data structures, they follows the same rules as any other code. This allows us to transform them just like we would any other data in our project.

In general, I find ClojureScript to be much more consistent and less noisy than equivalent JavaScript code. Consider the implementation of the updateGame function in the original JavaScript version:

function updateGame(game) {
    game = game.update("clock", (sec) => sec + 1);

    game = maybeUpdate(5, game, () => game.updateIn(["score", "home"], (s) => s + 1));
    game = maybeUpdate(5, game, () => game.updateIn(["score", "away"], (s) => s + 1));
    
    game = maybeUpdate(8, game, () => game.updateIn(["cards", "yellow"], (s) => s + 1));
    game = maybeUpdate(2, game, () => game.updateIn(["cards", "red"], (s) => s + 1));

    game = maybeUpdate(10, game, () => game.update("outrageousTackles", (t) => t + 1));

    const randomPlayerIndex = randomNum(0, 4);
    const effortLevel = randomNum();
    const invitedNextWeek = faker.random.boolean();

    game = game.updateIn(["players", randomPlayerIndex], (player) => {
        return player.set("effortLevel", effortLevel).set("invitedNextWeek", invitedNextWeek);
    });

    return game;
}

Compare it with the equivalent ClojureScript code:

(defn update-rand-player [game idx]
  (-> game
      (assoc-in [:players idx :effort-level] (rand-int 10))
      (assoc-in [:players idx :invited-next-week?] (> (rand) 0.5))))

(defn update-game [game]
  (-> game
      (update :clock inc)
      (maybe-update 5 [:score :home] inc)
      (maybe-update 5 [:score :away] inc)
      (maybe-update 8 [:cards :yellow] inc)
      (maybe-update 2 [:cards :red] inc)
      (maybe-update 10 [:outrageous-tackles] inc)
      (update-rand-player (rand-int 4))))

ClojureScript version has a lot less syntactic noise, and I find this has direct impact on my ability to reason about the code. The more quirks there are, the more likely I am to misread the intent. Noisy syntax results in situations where code looks like it's doing one thing, while it's actually doing something subtly different.

Another advantage is that ClojureScript is backed by immutable data structures by default. My experience is that immutability is crucial for writing large maintainable projects, as it allows safely reasoning about parts of the code in isolation.

Since immutability is pervasive as opposed to opt-in, it allows for tooling to be designed with it in mind. For example, Figwheel plugin relies on this property to provide live hot reloading in the browser.

Finally, ClojureScript compiler can do many optimizations, such as dead code elimination, that are difficult to do with JavaScript. I highly recommend the Now What? talk by David Nolen that goes into more details regarding this.

Overall, I'm pleased to see that ClojureScript and Reagent perform so well when stacked up against native JavaScript libraries. It's hard to overstate the fact that a ClojureScript library built on top of React.js can outperform React.js itself.

Using Chrome DevTools with Macchiato

Mon, 26 Dec 2016 00:00:00 +0000

Chrome DevTools provide a lot of useful features for debugging and profiling applications in the browser. As it happens, you can to connect DevTools to a Node.js process as well. Let's take a look at debugging the guestbook project from the examples repository.

You'll first have to start Figwheel to compile the project by running the following command:

lein build

Once the project is compiled, you have to start Node with the --inspect flag:

$ node --inspect target/out/guestbook.js

Debugger listening on port 9229.
Warning: This is an experimental feature and could change at any time.
To start debugging, open the following URL in Chrome:
    chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=127.0.0.1:9229/0dbaef2a-996f-4229-8a52-6c4e50d0bf18
INFO [guestbook.core:19] - guestbook started on 127.0.0.1 : 3000
Figwheel: trying to open cljs reload socket
Figwheel: socket connection established

You'll see that there's a URL printed in the console when the app starts. Copy this URL and open it in Chrome to connect DevTools to your Node process. At this point you can use all the tools the same way you would with an application running in the browser. You can debug ClojureScript files, profile the process, and so on.

Gotchas

Unfortunately, there's a small bug in the ClojureScript compiler that prevents timestamped source maps from working with Node. The problem is that that the compiler assumes that ClojureScript is running in browser and appends ?timestamp at the end of the file name as if it was a URL. Since Node is looking for actual files on disk, it fails to find the source map.

Currently, the workaround for this is to set :source-map-timestamp false in the compiler options. However, since Node caches the source maps, you have to restart the process any time you make a change in the code to get accurate line numbers.

The good news is that restarts happen instantaneously, and you can automate this process using Node supervisor as follows:

npm install supervisor -g
supervisor --inspect target/out/guestbook.js

That's all there is to it.

Macchiato Modules

Tue, 20 Dec 2016 00:00:00 +0000

As I discussed in the last post, Ring middleware stack closely resembles modules in a framework. However, one notable difference is that middleware functions aren't directly aware of one another. When the handler is passed to a middleware function, that function has no way of knowing what other middleware might have been wrapped around the handler by the time it got to it.

Conversely, these functions can't know what middleware will be wrapped after that they may depend on. Since middleware that was wrapped last will be invoked first, inner middleware ends up being dependent on the outer middleware.

This presents a number of problems. We can end up with multiple copies of the same middleware wrapped around the handler, middleware could be wrapped in the wrong order, or required middleware might be missing altogether. All of the above cases can lead to unpredictable behaviors, and can be difficult to debug.

One way to mitigate the problem is by creating a default middleware stack, such as seen in the ring-defaults library. This takes care of ensuring that all the core middleware is wrapped correctly, but doesn't help with middleware libraries added by the user. Another approach is to wrap the Ring stack in a higher level abstraction as seen with Integrant.

The solution I came up with for Macchiato is to use metadata attached to the handler to track the middleware that's been applied to it. This metadata can be used to inform how the middleware is loaded, and address the problems outlined above.

Let's take a look at an example of how this works in practice. Let's say we have the default handler such as:

(defn handler [req res raise]
  (res {:body (str (-> req :params :name))}))

Then, let's say we have two pieces of middleware we wish to wrap the handler with. The first will parse the request params, and the second will keywordize the params. The second middleware function depends on the first in order to work.

(defn parse-params [req]
  ;;parses request parameters into a map
  )

(defn wrap-params [handler]
  (fn [req res raise]
    (handler (parse-params req) res raise)))

(defn keywordize-params [params]
  ;;keywordizes the params
  )
          
(defn wrap-keyword-params [handler]
  (fn [req res raise]
    (handler (update req :params keywordize-params) res raise)))

We have to make sure that the middleware is chained as follows to get keywordized params:

(def wrapped-handler (-> handler
                         wrap-keyword-params
                         wrap-params))

However, it's not possible to deduce that this actually happened given the resulting handler function. Let's see how we can use metadata to address this problem. We'll update the wrap-params and the wrap-keyword-params functions as follows:

(defn
  ^{:macchiato/middleware
    {:id :wrap-params}}    
  wrap-params [handler]
  (fn [req res raise]
    (handler (parse-params req) res raise)))

(defn
 ^{:macchiato/middleware
   {:id :wrap-keyword-params
    :required [:wrap-params]}}
  wrap-keyword-params [handler]
  (fn [req res raise]
    (handler (update req :params keywordize-params) res raise)))

The :id key in the metadata is meant to specify the specific type of middleware as opposed to a concrete implementation. If two pieces of middleware happen to implement the same functionality they should use the same :id.

The :required key specifies the keys for the :ids that the particular middleware function depends on. In this case, wrap-keyword-params requires wrap-params to be present.

Next, we can write the code that will update the handler metadata each time it's wrapped with a middleware function.

(defn update-middleware-meta [handler handler-middleware middleware-meta]
  (with-meta
    handler
    {:macchiato/middleware
     (conj handler-middleware middleware-meta)}))

(defn loaded? [middleware {:keys [id]}]
  (some #{id} (map :id middleware)))

(defn- middleware-from-handler [handler]
  (->> handler meta :macchiato/middleware (remove nil?) vec))

(defn wrap
  ([handler middleware-fn]
   (wrap handler middleware-fn nil))
  ([handler middleware-fn opts]
   (let [handler-middleware (middleware-from-handler handler)
         middleware-meta    (-> middleware-fn meta :macchiato/middleware)]
     (if (loaded? handler-middleware middleware-meta)
       handler
       (update-middleware-meta
         (if opts
           (middleware-fn handler opts)
           (middleware-fn handler))
         handler-middleware
         middleware-meta)))))

The wrap function uses the :macchiato/middleware metadata key to get the currently applied middleware. When a middleware function with the same :id is already present, then the original handler is returned. Otherwise, the handler is wrapped with the middleware and its metadata is updated.

Let's update the original code that wrapped the handler to use the wrap function:

(def wrapped-handler (-> handler
                         (wrap #'wrap-keyword-params)
                         (wrap #'wrap-params)))

We can now use the meta function to access the metadata that was generated for the handler:

(meta wrapped-handler)

{:macchiato/middleware
 [{:id :wrap-params}
  {:id :wrap-keyword-params
   :required [:wrap-params]}]}

This tells us exactly what middleware has been applied to the handler and in what order, allowing us to validate that the middleware chain. This is accomplished as follows:

(defn validate [handler-middleware
   {:keys [id required] :as middleware-meta}]
  (when (not-empty (difference (set required)
                               (set (map :id handler-middleware))))
    (throw (js/Error. (str id " is missing required middleware: " required))))
  middleware-meta)
  
(defn validate-handler [handler]
  (let [middleware (middleware-from-handler handler)]
    (loop [[middleware-meta & handler-middleware] middleware]
      (when middleware-meta
        (validate handler-middleware middleware-meta)
        (recur handler-middleware)))
    handler))

With the above code in place we're now able to ensure that middleware functions are not loaded more than once, and that the order of middleware is correct.

Finally, Macchiato provides the macchiato.middleware/wrap-middleware convenience function that allows wrapping multiple middleware functions around the handler:

(m/wrap-middleware
  handler
  #'wrap-anti-forgery
  [#'wrap-session {:store (mem/memory-store)}]
  #'wrap-nested-params
  #'wrap-keyword-params
  #'wrap-params)

I think that the approach of using metadata provides an elegant view into the state of the middleware chain, while allowing Macchiato to stay compliant with Ring middleware semantics.

Another advantage of using metadata is that it makes the mechanism user extensible. If you're using a piece of middleware that doesn't have the metadata you need, you can always set it yourself.

The latest release of Macchiato has all the core middleware tagged with the appropriate metadata, and macchiato-defaults generates a handler that has the :macchiato/middleware key pointing to the vector of the middleware that was applied.

Frameworks, Foundations, and Macchiato

Sat, 17 Dec 2016 00:00:00 +0000

I've been making steady progress on Macchiato in the past weeks. This post will discuss some of my thought process and design decisions I settled on during this time.

One of the core questions is what problem the project aims to solve, and how it aims to do that.

The goal for Macchiato is to provide a way to build Node web applications using CojureScript. Ultimately, I'd like to produce something that's immediately usable and works well out of the box. The best way to accomplish that is to leverage the existing work in this domain.

The Ring stack is the most popular platform for developing Clojure applications on the JVM, and rightfully so in my opinion. It does an excellent job of abstracting the HTTP protocol, and provides a simple and intuitive API to the user.

Ring added async handler support in version 1.6, making it possible to implement compatible HTTP handlers on top of Node. This in turn allowed to port the core middleware stack to Macchiato.

As I was porting ring-core on Node, I've come to realize that Ring middleware libraries have a lot in common with framework modules.

These libraries are meant to be used together in a standard way, they're designed to compose, and they're often built on top of each other.

However, the Ring stack acts as a foundation rather than a framework. To understand this idea, let's first look at the traditional framework approach.

Frameworks

The core problem the frameworks attempt to solve is to provide a standard way to build software where the user can focus on writing the code that's relevant to their application. Meanwhile, the framework attempts to take care of all the incidental details around it.

The way traditional frameworks, such as Spring, accomplish this is through inversion of control. However, since the connections are no longer expressed directly in code, it makes it difficult to navigate them clouding the logic of the application.

Another problem with this approach is that the framework necessarily has to make a lot of decisions up front. Yet, a general purpose framework also has to be flexible enough to accommodate many types of application.

A framework typically turns into an exercise in designing a solution without knowing the problem. My experience is that it's not an effective way to write software in practice.

However, I think that the problem the frameworks attempt to solve is real. Having to artisanally handcraft each application from ground up is tedious and error prone.

Foundations

A better way to approach this problem is by addressing the known common needs. The key insight of Ring is that majority of reusable work is centred around processing the incoming HTTP requests and outgoing responses.

Ring provides a simple core that different middleware can be attached to in order to extend its functionality. We can add middleware that facilitates authentication, sessions, and so on. Buddy, compojure-api, and Sente are all great examples of this approach in practice.

One of the downsides of the library approach is that libraries aren't aware of one another, and the user has to glue them together. However, Ring middleware stack is not just a set of random libraries. Since Ring defines what the request and response must look like, it informs the design of libraries built on top of it.

The Ring stack is a mature and battle tested foundation for building the rest of the application on top of. At the same time, it doesn't attempt to guess the problems that are specific to your application. You're free to solve them in a way that makes sense to you.

Macchiato

Macchiato implements Ring 1.6 async handlers on top of the ClientRequest and the ServerResponse classes exposed by the Node HTTP module. Using the same API provides a consistent experience developing web applications on both platforms, and facilitates code reuse between them.

One immediate benefit of making Macchiato compatible with Ring was the ability to leverage its test harness. As I port the middleware to Node, I'm able to verify that it still behaves the same as the original. Going forward, it will be possible to write cljc middleware that targets both Ring and Macchiato.

Alongside the creation of the core libraries, I've been working on the template that packages everything together for the user. This template is informed by my experience working on Luminus and uses many of the same patterns and structure. If you're already familiar with Luminus, then you'll feel right at home with Macchiato.

As I noted in the last post, Macchiato development experience is very similar to working with Clojure on the JVM, and Chrome devtools along with Dirac look promising for debugging and profiling apps.

Meanwhile, the project has already garnered interest from the community. Ricardo J. Méndez has been working on creating a HugSQL style database access library, and Andrey Antukh, has been working on the dost crypto library.

It's great to see such prominent members of the community take interest in the project in the early stages. My hope is that as Macchiato matures we'll see many more supporting libraries built around it.

There's now a #macchiato channel on Clojurians slack. Feel free to drop by and discuss problems and ideas.

If you're looking to contribute to an open source project, Macchiato is a great opportunity. The project is still in the early days and there are many low hanging fruit. The project needs more tests, libraries, and documentation. This is a great time to make an impact on its future direction.

Macchiato: ClojureScript Arrives on the Server

Wed, 30 Nov 2016 00:00:00 +0000

I recently started the Macchiato project to provide a platform for building ClojureScript based apps on top Node.js.

First, let's look at some of the reasons for running ClojureScript on the server. The JVM is an excellent platform, it's mature, performant, and has a large ecosystem around it. This makes it a solid choice for a wide range of applications.

However, there are situations where the JVM might not be a good fit. It's a complex piece of technology that requires experience to use effectively. It has a fairly large footprint even from small applications. The startup times can be problematic, especially when it comes to loading Clojure runtime.

Meanwhile, Node.js also happens to be a popular platform with a large ecosystem around it. It requires far less resources for certain types of applications, has very fast startup times, and its ecosystem is familiar to many JavaScript developers.

Another appeal for Node based servers comes from building full stack ClojureScript single-page applications, since using Node on the server facilitates server-side rendering for any React based libraries.

While there are a few existing experiments using ClojureScript on Node, such as Dog Fort, none of these appear to be actively maintained. Since ClojureScript and its ecosystem have evolved in the meantime, I wanted to create a fresh stack using the latest tools and best practices.

Overview

My goal for Macchiato is to provide a stack modeled on Ring based around the existing Node ecosystem, and a development environment similar to what's available for Clojure on the JVM.

The Stack

I think it makes sense to embrace the Node ecosystem and leverage the existing modules whenever possible. For example, Ring style cookies map directly to the cookies NPM module. Conversely, there are a number of excellent ClojureScript libraries available as well, such as Timbre, Bidi, and Mount.

I used a Ring inspired model where I created wrappers around Node HTTP request and response objects. This allowed adapting parts of Ring, such as its session store implementation, with minimal changes.

The ClientRequest object is translated to a Clojure map, and the response map is written to the ServerResponse object. The request handler is implemented as follows:

(defprotocol IHTTPResponseWriter
  (-write-response [data res] "Write data to a http.ServerResponse"))

(defn response [req res opts]
  (fn [{:keys [cookies headers body status]}]
    (cookies/set-cookies cookies req res (:cookies opts))
    (.writeHead res status (clj->js headers))
    (when (-write-response body res)
      (.end res))))

(defn handler [handler-fn & [opts]]
  (let [opts (or opts {})]
    (fn [req res]
      (handler-fn (req->map req res opts) (response req res opts)))))

The handler accepts a handler-fn function that's passed the request map produced by the req->map helper. The handler-fn is expected to return a request handler function that will be used to generate the response. This function should accept the request map and the response call back function that writes the response map to the ServerResponse object. The IHTTPResponseWriter protocol is used to serialize different kinds of responses.

Concurrent Request Handling

JVM servers commonly use a listener thread for accepting client requests, the connections are then passed on to a thread pool of request handlers. This allows the listener to continue accepting connections while the requests are being processed.

Since Node is single threaded, long running request handlers block the server until they finish. While async operations can be used to handle IO in the background, any business logic will end up preventing the server from accepting new connections while it's running.

One way around this is to use the cluster module that spins up a single listening process that forks child processes and dispatches the requests to them. Setting this up is pretty straight forward:

(defstate env :start (config/env))

(defstate http :start (js/require "http"))

(defn app []
  (mount/start)
  (let [host (or (:host env) "127.0.0.1")
        port (or (some-> env :port js/parseInt) 3000)]
    (-> @http
        (.createServer
          (handler
            router
            {:cookies {:signed? true}
             :session {:store (mem/memory-store)}}))
        (.listen port host #(info "{{name}} started on" host ":" port)))))

(defn start-workers [os cluster]
  (dotimes [_ (-> os .cpus .-length)]
    (.fork cluster))
  (.on cluster "exit"
       (fn [worker code signal]
         (info "worker terminated" (-> worker .-process .-pid)))))

(defn main [& args]
  (let [os      (js/require "os")
        cluster (js/require "cluster")]
    (if (.-isMaster cluster)
      (start-workers os cluster)
      (app))))

However, it's worth noting that unlike threads, processes don't share memory. So, each child that gets spun up will require its own copy of the memory space.

The Template

I setup a template that creates a minimal app with some reasonable defaults. This template is published to Clojars, and you can try it out yourself by running:

lein new macchiato myapp

The template is setup similarly to Luminus. The source code for the project is found in the src folder, and the env folder contains code that's specific for dev and prod environments.

The project.clj contains dev and release profiles for working with the app in development mode and packaging it for production use. The app can be started in development mode by running:

lein build

This will clean the project, download NPM modules, and start the Figwheel compiler. Once Figwheel compiles the sources, you can run the app with Node in another terminal as follows:

node target/out/myapp.js

The app should now be available at http://localhost:3000.

Figwheel also starts the nREPL at localhost:7000. You can connect to it from the editor and run (cljs) to load the ClojureScript REPL.

Packaging the app for production is accomplished by running:

lein package

This will print out package.json for the app and generate the release artifact called target/release/myapp.js.

Looking Forward

Overall, I think that ClojureScript on top of Node is ready for prime time. It opens up server-side Clojure development to a large community of JavaScript developers, and extends the reach of Clojure to any platform that supports Node.

While the initial results are very promising, there is still much work to be done in order to provide a solid stack such as Luminus. If you think this project is interesting, feel free to ping me via email or on the Clojurians slack. I would love to collaborate on making Macchiato into a solid choice for developing Node based applications.

PostgreSQL Async Notifications in Luminus

Sat, 05 Nov 2016 00:00:00 +0000

There are many situations where the application needs to react to changes in the data. The simplest way to handle this requirement is to keep state in the server session. Unfortunately, this makes it difficult to scale applications horizontally, and can incur additional memory requirements.

A common solution to this problem is to use an external queue service that each instance of the application subscribes to. However, this adds a new component to the architecture that needs to be maintained.

A less known option is to use Postgres NOTIFY command to send push notifications from the database. This allows multiple instances of the application can subscribe directly to the database to listen for events.

This post will walk you through configuring a Luminus app to listen for Postgres notification, and broadcast them to the connected clients over a WebSocket.

prerequisites:

Let's start by creating a new project for our app:

lein new luminus pg-feed-demo +postgres +re-frame

The database

The first step is to create a schema for the app, and set the connection URL in the profiles.clj, e.g:

{:profiles/dev
 {:env
  {:database-url
   "jdbc:pgsql://localhost:5432/feeds_dev?user=feeds&password=feeds"}}

Migrations

Once the schema is ready, we can write a migrations script that creates a table called events, and sets up a notification trigger on it. Let's run the following command in the project root folder to create the migration files:

lein migratus create events-table

Next, add the following script as the up migration:

CREATE TABLE events
(id SERIAL PRIMARY KEY,
 event TEXT);
--;;
CREATE FUNCTION notify_trigger() RETURNS trigger AS $$
DECLARE
BEGIN
 -- TG_TABLE_NAME - name of the table that was triggered
 -- TG_OP - name of the trigger operation
 -- NEW - the new value in the row
 IF TG_OP = 'INSERT' or TG_OP = 'UPDATE' THEN
   execute 'NOTIFY '
   || TG_TABLE_NAME
   || ', '''
   || TG_OP
   || ' '
   || NEW
   || '''';
 ELSE
   execute 'NOTIFY '
   || TG_TABLE_NAME
   || ', '''
   || TG_OP
   || '''';
 END IF;
 return new;
END;
$$ LANGUAGE plpgsql;
--;;
CREATE TRIGGER event_trigger
AFTER INSERT or UPDATE or DELETE ON events
FOR EACH ROW EXECUTE PROCEDURE notify_trigger();

Thenotify_trigger function will broadcast a notification with the table name, the operation, and the parameters when available. The event_trigger will run it whenever insert, update, or delete operations are performed on the messages table.

We'll also add the down migration for posterity:

DROP FUNCTION notify_trigger() CASCADE;
DROP TABLE events;

We can now run migrations as follows:

lein run migrate

Queries

Let's open the resources/sql/queries.sql file and replace the default queries with the following:

-- :name event! :! :n
-- :doc insert a new event
INSERT INTO events (event) VALUES (:event)

The server

Unfortunately, the official Postgres JDBC driver cannot receive asynchronous notifications, and uses polling to check if any notifications were issued. Instead, we'll use the pgjdbc-ng driver that provides support for many Postgres specific features, including async notifications. Let's update our app to use this driver instead by swapping the dependency in project.clj:

;[org.postgresql/postgresql "9.4.1211"]
[com.impossibl.pgjdbc-ng/pgjdbc-ng "0.6"]

Notification listener

Let's open up the pg-feed-demo.db.core namespace and update it to fit our purposes. Since we're no longer using the official Postgres driver, we'll need to update the namespace declaration to remove any references to it. We'll also add the import for the PGNotificationListener class that will be used to add listeners to the connection. To keep things simple, we'll also remove any protocol extensions declared there. The resulting namespace should look as follows:

(ns pg-feed-demo.db.core
  (:require
    [cheshire.core :refer [generate-string parse-string]]
    [clojure.java.jdbc :as jdbc]
    [conman.core :as conman]
    [pg-feed-demo.config :refer [env]]
    [mount.core :refer [defstate]])
  (:import
    com.impossibl.postgres.api.jdbc.PGNotificationListener))

(defstate ^:dynamic *db*
  :start (conman/connect! {:jdbc-url (env :database-url)})
  :stop (conman/disconnect! *db*))

(conman/bind-connection *db* "sql/queries.sql")

In order to add a notification listener, we first have to create a connection. Let's create a Mount defstate called notifications-connection to hold it:

(defstate notifications-connection
  :start (jdbc/get-connection {:connection-uri (env :database-url)})
  :stop (.close notifications-connection))

Next, we'll add functions that will allow us to add and remove listeners for a given connection:

(defn add-listener [conn id listener-fn]
  (let [listener (proxy [PGNotificationListener] []
                   (notification [chan-id channel message]
                     (listener-fn chan-id channel message)))]
    (.addNotificationListener conn listener)
    (jdbc/db-do-commands
      {:connection notifications-connection}
      (str "LISTEN " (name id)))
    listener))

(defn remove-listener [conn listener]
  (.removeNotificationListener conn listener))

Let's start the application by running lein run in the terminal. Once it starts, the nREPL will become available at localhost:7000. When the REPL is connected, run the following code in it to start the database connection and register a listener:

(require :reload 'pg-feed-demo.db.core)
(in-ns 'pg-feed-demo.db.core)

(mount.core/start
  #'*db*
  #'notifications-connection)
                  
(add-listener
  notifications-connection
  "events"
  (fn [& args]
    (apply println "got message:" args)))

We can now test that adding a new message produces the notification:

(event! {:event "hello world"})

One the function runs, we should see something like the following printed in the terminal as the message is added to the database:

got message: 32427 messages INSERT (0,"hello world")

WebSocket connection

We're now ready to setup the WebSocket connection that will be used to push notifications to the clients. We'll update the pg-feed-demo.routes.homenamespace to look as follows:

(ns pg-feed-demo.routes.home
  (:require [pg-feed-demo.layout :as layout]
            [compojure.core :refer [defroutes GET]]
            [pg-feed-demo.db.core :as db]
            [mount.core :refer [defstate]]
            [immutant.web.async :as async]
            [clojure.tools.logging :as log]))

(defstate channels
  :start (atom #{}))

(defstate ^{:on-reload :noop} event-listener
  :start (db/add-listener
           db/notifications-connection
           :events
           (fn [_ _ message]
             (doseq [channel @channels]
               (async/send! channel message))))
  :stop (db/remove-listener
          db/notifications-connection
          event-listener))

(defn persist-event! [_ event]
  (db/event! {:event event}))

(defn connect! [channel]
  (log/info "channel open")
  (swap! channels conj channel))

(defn disconnect! [channel {:keys [code reason]}]
  (log/info "close code:" code "reason:" reason)
  (swap! channels #(remove #{channel} %)))

(defn home-page []
  (layout/render "home.html"))

(defroutes home-routes
  (GET "/" []
    (home-page))
  (GET "/events" request
    (async/as-channel
      request
      {:on-open    connect!
       :on-close   disconnect!
       :on-message persist-event})))

The channels state will contain a set of all the channels for the currently connected clients.

The event-listener will create a new listener that's triggered when events are stored in the database. The handler function will broadcast each event to all the connected clients. Note that we need ^{:on-reload :noop} metadata on the listener to prevent it being registered multiple times in case the namespace is reloaded during development.

Whenever the server receives a message from a client, the message will be persisted to the database by the persist-event! function.

Finally, we'll create the /events route that will be used to manage WebSocket communication with the clients.

The client

The client will need to track the currently available messages, allow the user to send new messages to the server, and update the available messages based on server WebSocket notifications.

Let's run Figwheel to start the ClojureScript compiler before we start working on the client-side code by running the following command:

lein figwheel

Once Figwheel compiler starts, navigate to http://localhost:3000 in the browser to load the client-side of the application.

Re-frame events

We'll start by adding a handler for adding messages in the pg-feed-demo.handlers namespace:

(reg-event-db
  :event
  (fn [db [_ event]]
    (update db :events (fnil conj []) event)))

Next, we'll add a corresponding subscription to see the current messages in the pg-feed-demo.subscriptions namespace:

(reg-sub
  :events
  (fn [db _]
    (:events db)))

WebSocket connection

We can now add a pg-feed-demo.ws namespace to manage the client-side of the WebSocket connection:

(ns pg-feed-demo.ws)

(defonce ws-chan (atom nil))

(defn send
  [message]
  (if @ws-chan
    (.send @ws-chan message)
    (throw (js/Error. "Websocket is not available!"))))

(defn connect-ws [url handler]
  (if-let [chan (js/WebSocket. url)]
    (do
      (set! (.-onmessage chan) #(-> % .-data handler))
      (reset! ws-chan chan))
    (throw (js/Error. "Websocket connection failed!"))))

User interface

Finally, we'll update the pg-feed-demo.core namespace to list incoming events and allow the user to generate an event. To do that, We'll update the namespace to look as follows:

(ns pg-feed-demo.core
  (:require [reagent.core :as r]
            [re-frame.core :as rf]
            [pg-feed-demo.handlers]
            [pg-feed-demo.subscriptions]
            [pg-feed-demo.ws :as ws]))

(defn home-page []
  [:div.container
   [:div.navbar]
   [:div.row>div.col-sm-12>div.card
    [:div.card-header>h4 "Events"]
    [:div.card-block>ul
     (for [event @(rf/subscribe [:events])]
       ^{:key event}
       [:li event])]]
   [:hr]
   [:div.row>div.col-sm-12>span.btn-primary.input-group-addon
    {:on-click #(ws/send (str "user event " (js/Date.)))}
    "generate event"]])

(defn mount-components []
  (r/render [#'home-page] (.getElementById js/document "app")))

(defn init! []
  (rf/dispatch-sync [:initialize-db])
  (ws/connect-ws
    (str "ws://" (.-host js/location) "/events")
    #(rf/dispatch [:event %]))
  (mount-components))

That's all there is to it. We should now be able to send events to the server and see the notifications in the browser. We should also be able to generate events by running queries directly in the database, or in another instance of the application.

The complete source for the project is available here.

Configuring Atom for Luminus

Sat, 15 Oct 2016 00:00:00 +0000

There are many editors and IDEs available for Clojure today. The most popular ones are Emacs with CIDER and IntelliJ with Cursive. While both of these options provide excellent development environments, they also require a bit of learning to become productive in.

Good news is that you don't have to learn a complex environment to get started. This post will walk you through the steps of configuring Atom editor editor to work with a Luminus project. We'll see how to configure Atom for editing Clojure code and how to connect it to the remote REPL started by the Luminus app for interactive development.

Prerequisites

You'll need the following installed to follow along with this post:

Configuring Atom

Let's take a look at the bare minimum Atom configuration for working with Clojure. Once you're up and running, you may with to look here for a more advanced configuration. We'll start by installing the following packages:

parinfer or lisp-paredit package for structural editing
proto-repl to connect to a Clojure REPL

Structural Editing

A structural editor understands the structure of Clojure code and provides shortcuts for manipulating s-expressions instead of lines of text. It also eliminates the need to manually balance the parens. This takes a bit of getting used to, but it will make working with Clojure a lot more pleasant in the long run.

Parinfer

The parinfer mode will attempt to automatically infer the necessary parens based on the indentation. This mode has a gentle learning curve and attempts to get our of your way as much as possible. You can read more about how it works here.

Paredit

The paredit mode takes a bit more getting used to, but provides you with precise control over code structure. Whenever you add a peren, a matching closing paren will be inserted automatically. Paredit will also prevent you you from deleting parens unless you have an empty pair.

The package also provides a handy ctrl-w shortcut that will extend the selection by s-expression. This is the recommended way to select code as you don't have to manually match the start and end of an expression when selecting.

The REPL

The REPL is an essential tool for working with Clojure. When integrated with the editor, it allows running any code that you write directly in the application.

Connecting the REPL

We'll create a new Luminus project with SQLite database support by running the following command:

lein new luminus myapp +sqlite

Once the project is created, we can go to the project folder and run the migrations:

cd myapp 
lein run migrate

We're now ready to start the app in development mode:

lein run

The app will start the nREPL server on localhost:7000 once it loads. Let's open the project in Atom and connect to the nREPL instance.

The default keybinding for connecting to the nREPL is ctrl-alt-, y on Windows/Linux and cmd-alt-, y on OS X. This should pop up a dialog asking for the host and the port.

Enter 7000 as the port and hit enter. If everything went well the REPL should now be connected to your project.

Once the REPL is connected we can try to evaluate some code in it. For example, let's check what namespace we're currently in by typing *ns* in the REPL and then hitting shift-enter. The result should look something like the following:

Let's navigate to the myapp.routes.home namespace and try to run some of the database query functions from there. We'll first need to require the database namespace:

(ns myapp.routes.home
  (:require [myapp.layout :as layout]
            [compojure.core :refer [defroutes GET]]
            [ring.util.http-response :as response]
            [clojure.java.io :as io]
            ;; add a reference to the db namespace
            [myapp.db.core :as db]))

Once we've done that, we'll need to reload myapp.routes.home namespace. To do that we'll need to send the code from the editor to the REPL for evaluation.

There are a few commands for doing this. I recommend starting by using the ctrl-alt-, B shortcut that sends the top-level block of code to the REPL for execution. Place the cursor inside the ns declaration and hit ctrl-alt-, B to send it to the REPL. We can see that the REPL displays the code that was sent to it along with the result:

Now that we have the db namespace required, we can start the database connection state by typing the following command in the REPL:

(mount.core/start #'db/*db*)

The result should look as follows:

With the database is started, let's add a user to it by running the following code in the REPL:

(db/create-user!
 {:id "foo"
  :first_name "Bob"
  :last_name "Bobberton"
  :email "bob@foo.bar"
  :pass "secret"})

We can also test that the user was added successfully by running:

(db/get-user {:id "foo"})

We can see that the user record exists in the database:

{:id "foo"
 :admin nil
 :is_active nil
 :last_login nil
 :first_name "Bob"
 :last_name "Bobberton"
 :email "bob@foo.bar"
 :pass "secret"}

As you can see, the code that we run in the REPL executes in the context of the application and has access to all the resources and the application state. Let's take a closer look at how this helps us during development.

You might have noticed that the records we get back from the database use the _ character as word separator. Meanwhile, idiomatic Clojure code uses the - character. Let's write a couple of functions to transform the key names in the results.

A Clojure map represents its entities as vectors containing key-value pairs. We'll start by writing a function to rename underscores to dashes in map entries:

(defn clojurize [[k v]]
  [(-> k name (.replaceAll "_" "-") keyword) v])

We'll load the function in the namespace by placing the cursor anywhere inside it and hitting ctrl-alt-, B to load it. Let's run this function in the REPL to see that it works:

(clojurize [:first_name "Bob"])
=>[:first-name "Bob"]

We can see that the result is what we expect. Next, let's write a function to rename the keys in a map:

(defn clojurize-keys [m]
  (->> m (map clojurize) (into {})))

We'll load the new function and test that this works as expected in the REPL:

(clojurize-keys (db/get-user {:id "foo"}))

We see that the result is the translated map that we want:

{:id "foo"
 :admin nil
 :is-active nil
 :last-login nil
 :first-name "Bob"
 :last-name "Bobberton"
 :email "bob@foo.bar"
 :pass "secret"}

Now that we have a nicely formatted result, let's add a route to query it in the browser:

(defroutes home-routes
  (GET "/" [] (home-page))
  (GET "/user/:id" [id]
       (-> (db/get-user {:id id})
           (clojurize-keys)
           (response/ok)))
  (GET "/about" [] (about-page)))

We can now navigate to http://localhost:3000/user/foo and see the user data.

Conclusion

That's all there is to it. While this setup is fairly minimal, it will let you play with a lot of Clojure features without having to spend practically any time learning and configuring an editor.

Reusable Components

Sun, 25 Sep 2016 00:00:00 +0000

One of the projects my team works is a clinical documentation platform. The goal of the project is to facilitate the implementation of different kinds of workflows for the clinics at our hospital.

Requirements

One major requirement for the platform is support for multiple concurrent users working on the same document. For example, both a physician and a pharmacist may have to enter the prescribed medications for a patient. Both users have to be able to complete their work concurrently and to be aware of the changes made by the other.

Another requirement is to visualize the data differently depending on the discipline. Patient lab results may need to be shown as a table in one place, but as a trending chart in another. A physician may call a piece of data by one name, while the pharmacist calls it by another.

In other words, the data model needs to have different views associated with it. Furthermore, some information may not be shown in a particular view at all, but it would still need to be updated when a field in the view changes.

Consider an example where you're collecting patient height and weight, then the BMI is calculated based on that. The user may only be interested in seeing height and weight in their workflow, but once that data changes the BMI still needs to be recalculated even if it's not displayed in that view.

Finally, we have a large data model based on the Hl7 FHIR standard. This standard specifies resources for describing different kinds clinical data, such as patient demographics, medications, allergies and so on. An example of a resource definition can be seen in the Resources section.

Architecture

The concurrent user requirement means that the changes made by different users have to be kept in sync. Meanwhile, business rules have to be applied transactionally for each change.

The easiest way to address the above requirements is to keep the master document on the server. Any time a client makes a change, a request is sent to the server over a WebSocket. The server updates the field in the document and runs the business rules. It will then notify the clients viewing a particular document of all the fields that were updated in the transaction.

The clients simply reflect the state of the document managed by the server and never make local updates to the model. This ensures that all the changes are handled centrally, and that the business rules are applied regardless of what is displayed on the client.

The second problem is the creation of views for the data. Since we have many distinct fields, but only a small number of types of fields, it made sense for us to create widgets to represent specific data types. The widgets are bound to the fields in the data model using the path as a unique identifier.

Let's take a look at a sample project that illustrates the above architecture to see how this works in practice.

Server-Side State Management

We'll start by examining the server-side implementation of the architecture starting with the components-example.document namespace. The server in our example keeps its state in a ref, and updates it transactionally whenever it receives an update from the client.

(defonce document (ref {}))

(defn bmi [weight height]
  (when (and weight height (pos? height))
    (/ weight (* height height))))

(defn bmi-rule [doc]
  (let [weight (get-in doc [:vitals :weight])
        height (get-in doc [:vitals :height])]
    [{:path  [:vitals :bmi]
      :value (bmi weight height)}]))

(def rules
  {[:vitals :weight] bmi-rule
   [:vitals :height] bmi-rule})

(defn run-rules [doc {:keys [path]}]
  (when-let [rule (rules path)]
    (rule doc)))

(defn update-document! [{:keys [path value] :as path-value}]
  (dosync
    (let [current-document (alter document assoc-in path value)
          updated-paths    (run-rules current-document path-value)]
      (doseq [{:keys [path value]} updated-paths]
        (alter document assoc-in path value))
      (into [path-value] updated-paths))))

Note the use of the dosync block in the update-document! function to update the document and run the business rules as a transaction.

Each rule can in turn create additional changes in the document. A vector of updated path-value pairs is returned as the result of the update. Our setup has a single rule that calculates the BMI. This rule is triggered whenever the weight or height fields are changed.

While the example keeps the document in memory, there's nothing stopping us from keeping it in the database and running the updates using a transaction against it. This is especially easy to do with PostgreSQL as it supports working with individual JSON fields directly.

Client-Server Communication

When the client loads, it establishes a WebSocket connection with the server. This connection is used to notify the server of the user actions and to push the changes back to the clients.

Server side of the connection can be found in the components-example.routes.ws namespace. The part that's of most interest to us is the handle-message multimethod that's keyed on the :document/update event:

(defmethod handle-message :document/update [{:keys [?data]}]
  (let [updated-paths (update-document! ?data)]
    (doseq [uid (-> @socket :connected-uids deref :any)]
      ((:send-fn @socket) uid [:document/update updated-paths]))))

The multimethod calls the update-document! function we just saw and then notifies the connected clients with its result.

Conversely, the client portion of the WebSocket connection is found in the components-example.ws namespace. Here we have the update-value function that sends the update event to the server, and the handle-message multimethod that handles incoming update messages:

(defn update-value [path-value]
  ((:send-fn @socket) [:document/update path-value]))

(defmethod handle-message :document/update [[_ updated-paths]]
  (doseq [{:keys [path value]} updated-paths]
    (dispatch [:set-doc-value path value])))

The multimethod dispatches a re-frame event for each path/value pair in the message. Let's take a look at the re-frame handlers and subscriptions next.

Client-Side State Management

Re-frame handlers are found in the components-example.handlers namespace, where the document state is updated using the following handlers:

(reg-event-db
  :set-doc-value
  (fn [db [_ path value]]
    (assoc-in db (into [:document] path) value)))

(reg-event-db
  :save
  (fn [db [_ path value]]
    (ws/update-value {:path path :value value})
    db))

The :save event creates a WebSocket call to notify the server of the change. Meanwhile, the :set-doc-value event is used to update the client state with the set of changes. This event will be triggered by a WebSocket message from the server, whenever the master document is updated.

We also need to have a corresponding subscription to view the state of the document. This subscription is found in the components-example.subscriptions namespace:

(reg-sub
  :document
  (fn [db [_ path]]
    (let [doc (:document db)]
      (if path (get-in doc path) doc))))

Next, let's take a look at how the UI components are defined and associated with the data model.

Application Components

The UI for the application consists of widgets representing individual data types. When a widget is instantiated it's associated with a particular path in the document. The widgets are found in the components-example.widgets namespace.

The set of all valid paths is contained in the components-example.model namespace. This namespace is written using CLJC, and provides a single schema for both the client and the server portions of the application.

The widgets are associated with the model using the components-example.model-view namespace. Each of the paths found in the model can have multiple views associated with it. In our example, we have the form for entering the data and a preview for displaying it.

Finally, we have the components-example.view namespace that provides the layout for the page. This namespace instantiates the widgets defined in the model-view namespace and lays them out as needed for a particular page in the application.

Let's explore each of these namespaces in detail below.

Model

The data model in our application consists of a map that's keyed on the element path where each key points to the type of data found in that element. Let's take a look at a simple demographics model below:

(def Name
  {:first s/Str
   :last  s/Str})

(def demographics
  {[:demographics :mrn]
   s/Str
   
   [:demographics :name]
   Name

   [:demographics :name :dob]
   #?(:clj java.util.Date
      :cljs js/Date)

   [:demographics :address :province]
   (s/enum "AB" "BC" "MB" "NB" "NL" "NS" "NT" "NU" "ON" "PE" "QC" "SK" "YT")})

We can see that the demographics model contains the name, the date of birth, and the province for the patient.

The paths can point to any type of data structure. For example, the [:demographics :name] path points to a map containing the first and the last name.

Meanwhile, the [:demographics :name :dob] path leverages CLJC to provide different validators for Clojure and ClojureScript.

Widgets

Now, let's take a look at the approach we took to map the FHIR data model to the UI in the application.

At the lowest level we have widgets that represent a particular type of element. These would include text fields, datepickers, dropdowns, tables, and so on. The way we chose to represent the widgets was to use multimethods. The widgets are initialized using a map containing the :type key:

(defmulti widget :type)

Given the multimethod definition above, a text input widget might look as follows:

(defmethod widget :text-input [{:keys [label path]}]
  (r/with-let [value    (r/atom nil)
               focused? (r/atom false)]
    [:div.form-group
     [:label label]
     [:input.form-control
      {:type      :text
       :on-focus  #(do
                    (reset! value @(rf/subscribe [:document path]))
                    (reset! focused? true))
       :on-blur   #(do
                    (rf/dispatch
                      [:save path @value])
                    (reset! focused? false))
       :value     (if @focused? @value @(subscribe-doc path))
       :on-change #(reset! value (-> % .-target .-value))}]]))

The text input widget subscribes to the given path in the document as its value. Since we don't want to generate unnecessary WebSocket events while the user is typing, the input keeps a local state while it's focused.

When the user focuses the input, its local state is set to the current document state, and when the focus is lost, the update event is generated with the new value.

Each widget is a reusable component that is associated with a path in the document to create a concrete instance:

[widget {:type :text-input
         :lable "first name"
         :path [:patient :name :first]}]

Since the widgets are mapped to the data elements via the path when instantiated, they can easily be composed into larger components. For example, we'll create a patient name component using two :text-input widgets:

(defmethod widget :name [{:keys [first-name last-name path]}]
  [:div
   [widget {:label first-name
            :type :text-input
            :path (conj path :first)}]
   [widget {:label last-name
            :type :text-input
            :path (conj path :last)}]])

Composite widgets provide us with the ability to describe complex data elements that are common among different resources.

Model-View

The widgets are associated with the concrete paths using a model-view map. This map is keyed on the same paths as the model map, but points to widget declarations instead of the types. We can represent the MRN and name fields as follows:

(def demographics-form
  {[:demographics :mrn]
   {:label "medical record number"
    :type  :text-input}
    
   [:demographics :name]
   {:first-name "first name"
    :last-name  "last name"
    :type       :name}})

The model/view map contains a set of UI elements for representing the data model. Note that this approach allows us to create multiple view definitions for any particular data element.

This is useful as we may wish to present the data differently depending on the use case. For example, some users may manipulate the data, while others will simply want to view it.

View

This brings us to the view portion of the architecture. The view aggregates the widgets defined in the model-view map into a particular layout. The demographics view could look as follows:

(defn create-widget [view path]
  (let [opts (view path)]
    [widget (assoc opts :path path)]))

(defn form-row [view path]
  [:div.row>div.col-md-12
   (create-widget view path)])
   
(defn demographics [view]
  [:div
   (form-row demographics-form [:demographics :mrn])
   (form-row demographics-form [:demographics :name])])

Here we use a create-widget helper function that looks up the options for a widget in the view and instantiate it with the given path.

The widgets are then wrapped in the layout tags in the form-row and inserted in the the div that represents the demographics view.

Once the widgets are defined, it becomes trivial to create different kinds of interfaces using them. This is perfect for our use case where we have a large common data model with many different views into it.

Conclusion

I hope this provides a bit of an insight into building large UIs with reusable components using Reagent and re-frame. My team has found that this approach scales very nicely and allows us to quickly build different kinds of UIs against a common data model.

Static Typing vs WebSockets

Tue, 06 Sep 2016 00:00:00 +0000

A recent post compared WebSocket server performance in Clojure, C++, Elixir, Go, NodeJS, and Ruby. Chris Allen wrote a nice follow-up post where he implemented the benchmark using Haskell.

The initial results looked extremely favorable for Haskell. However, it turned out that the Haskell implementation failed to deliver messages reliably, dropping 98% of the messages it received. What's interesting is that this is exactly the kind of behavior we would expect Haskell type system to prevent from happening. So, how did the fact that messages were being dropped slip by completely undetected?

update

As a couple of people helpfully pointed out, the problem was not in fact caused by using unsafe functions. It's simply a type of error that would not be caught by the Haskell type system in general.

While the problems I outline with the unsafe operations are still present, it's clearly possible for serious problems to slip by even when you're not using them.

If anything, I think this bolsters the argument for the importance of a mature ecosystem and specification testing.

Type system escape hatches

Haskell provides escape hatches from its type system, and these are often used in practice to achieve reasonable performance. When we look at code in the unagi-chan library used in the Haskell implementation, we can see that it uses unsafeInterleaveIO to get the channel contents.

This is an example of an escape hatch that bypasses the type checker entirely. While Haskell is conceptually a pure language, the internal GHC implementation is imperative in nature. GHC runtime evaluates impure functions that produce side effects making the order of evaluation important. Functions like unsafeInterleaveIO expose the impure runtime to the user, and open the gate for all the types of errors we're familiar with from imperative languages.

The way GHC implements Haskell inherently precludes safety guarantees by its type system. The purity is effectively an honor system, and cannot be proved by the compiler. In other words, once we use a library that happens to use unsafe operations any guarantees that we get from the type system go out of the window.

Types are not a specification

While Haskell type system can help ensure that our code is self-consistent, it clearly can't provide any guarantees regarding the behavior of third party code. Since most real world applications tend to rely on many third party libraries, it means that unless we know what each library is doing we can't ever be certain that our code will work as expected.

The developer can't possibly be expected to audit every library they use in their project to ensure that it behaves safely. Since most applications rely on large amounts of third party code, availability of mature and reliable libraries is a major factor when it comes to building robust applications.

While the benchmark in this example is trivial, it's a good example of real world problems many projects have to deal with. Most applications have to interact with the external resources such as queues, databases, and other services. Therefore, we need mature and tested libraries in order to accomplish these tasks effectively.

I think this is one of the major reasons why hosted languages have been gaining popularity in recent years. When the language relies on a mature ecosystem, such as the JVM, it inherits a lot of battle tested code along with it.

However, this problem exists in every language. Ultimately, we need to know what the code is doing, and clearly types don't provide us with enough information to really be sure the code is doing what was intended.

Achieving correctness

The only way to know that the code is doing what was intended is to have a specification, and test the code against it. This is true pretty much for any language in use today. Tests allow us to validate complex properties that are difficult or even impossible to encode using most type systems.

Consider the trivial case of validating a user generated password. We need to check its length, combinations of characters it contains, and whether it matches the retyped password. All most type systems can tell us is that we have to pass the function a couple of strings and it will return a boolean.

To check any of the properties that prove that the function does what was intended, we need to come up with a specification and test the code against it. While the tests do not provide an exhaustive proof of correctness, they provide proof that the code satisfies the intended use cases.

An argument can be made that types save time in finding bugs when the tests fail. However, my experience is that it's often trivial to track down the actual problem once you're aware of it.

I think this is where the trade-off between static and dynamic languages lies. The former forces us to describe the types up front, and makes it easier to track down potential errors. Meanwhile, the latter approach allows us to skip this step at the cost of potentially having to do more work to find bugs later.

To the best of my knowledge nobody knows whether one approach is strictly more efficient than the other. The overall amount of work appears to be comparable with both approaches. However, the nature of work is different, therefore each approach appeals to a different mindset.

One interesting approach is to generate types from tests as seen in recent version of Typed Clojure. Using tests to drive type generation has the potential to offer the best of both worlds. We can work with a dynamic language, and offload the work of figuring out the type relationships to a library. As long as we're diligent about writing tests, we get the types for free.

Another powerful tool for writing robust code is the REPL. When it's integrated with the editor, testing code as you write it becomes very natural. It's quite common for me to test functions as I develop them, then extract the REPL session into a test suite for the feature I'm working on.

Takeaways

Even a strong type system, such as one found in Haskell, provides a very weak specification in practice. Just because the code compiles doesn't mean it's actually doing what was intended.

The type system does not help debugging many real world problems. The code in this benchmark worked as expected under small load, and started exhibiting errors when it was stress tested.

The ecosystem around the language is an important factor when it comes to productivity. When we use mature and battle tested libraries, we're much less likely to be surprised by their behavior.

Tests are ultimately the only practical way to provide a specification for the application. Behaviors that are easily tested can be difficult or impossible to encode using a type system.

Web Development with Clojure, Second Edition

Sat, 23 Jul 2016 00:00:00 +0000

I'm glad to announce that the second edition of Web Development with Clojure is finally finished. The book took longer to put together than I anticipated, and I ended up missing the original release target by a few months.

However, all the delays resulted in a much better book in the end. Having a long beta period allowed me to collect a lot of feedback from the readers and address any concerns that came up. This process helped ensure that the material is clear and easy to follow, while keeping a good pace. I discussed the specifics of what the book covers an earlier post here.

It's my sincere hope that the book will provide the readers with a smooth path into the wonderful world of Clojure web development.

Working around the Java SSL trust store

Fri, 15 Jul 2016 00:00:00 +0000

The Java standard library provides a rich networking API. For example, the java.net.URL class provides a simple way to access resources using a URL location pattern. We can do fun stuff like this using it:

(-> "https://gist.githubusercontent.com/yogthos/f432e5ba0bb9d70dc479/raw/768050c7fae45767b277a2ce834f4d4f00158887/names.clj"
    (java.net.URL.)
    (slurp)
    (load-string))

(gen-name 11 6)

Unfortunately, the SSL certificates bundled with the default Java runtime aren't comprehensive. For example, the https://http.cat/ site has a valid certificate that's not part of the default Java trust store.

Let's write a function to read an image from the site using java.net.URL, then save it to a file to see what happens.

(defn read-image [url]
  (let [conn (.openConnection (java.net.URL. url))]    
    (.getInputStream conn)))
    
(clojure.java.io/copy
  (read-image "https://http.cat/200")
  (java.io.FileOutputStream. "200.jpg"))

When we try to access the resource, we end up with a security exception because the default trust store does not contain the right certificate.

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException:
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target
...

One way we could work around this problem would be to add the certificate to the local store. This is the proper solution that should be used in the vast majority of cases.

However, there are situations where this approach isn't possible. I've run into many situations working in the enterprise where SSL was misconfigured, and the application would need to connect to an intranet service over such a connection. At the same time I had no control over the deployment environment and wasn't able to manage the keystore there.

An alternative approach is to replace the default certificate check for a specific connection with a custom one. Let's take a look at how this can be accomplished.

We'll first have to create a proxy TrustManager, then use it to create a socket factory for our connection as seen in the following code:

(defn set-socket-factory [conn]
  (let [cert-manager (make-array X509TrustManager 1)
        sc           (SSLContext/getInstance "SSL")]
    (aset cert-manager 0
          (proxy [X509TrustManager][]
            (getAcceptedIssuers [])
            (checkClientTrusted [_ _])
            (checkServerTrusted [_ _])))
    (.init sc nil cert-manager (java.security.SecureRandom.))
    (.setSSLSocketFactory conn (.getSocketFactory sc))))

The custom socket factory will use the X509TrustManager proxy that we provide and rely on it for validation. We can simply return nil from each of the validation methods to skip the certificate validation.

Note that while we're skipping validation entirely in the above example, we'd likely want to supply a custom validator that validates against an actual certificate in practice.

Next, let's update the read-image function to set the custom socket factory for the connection before trying to read from it:

(defn read-image [url]
  (let [conn (.openConnection (java.net.URL. url))]
    (set-socket-factory conn)
    (.getInputStream conn)))

(clojure.java.io/copy
  (read-image "https://http.cat/200")
  (java.io.FileOutputStream. "200.jpg"))

We should now have a 200.jpg file on our file system with the following content:

That's all there is to it. We can now enjoy consuming cat HTTP status pictures using the java.net.URL and even make some silly Ring middleware using it. :)

Consider Hoplon

Mon, 06 Jun 2016 00:00:00 +0000

A recent discussion of Hoplon vs React has been making rounds. While I don't necessarily agree that using React is as difficult as Micha makes it sound, I do think that Hoplon provides an interesting alternative to React that has a number of benefits.

The main selling point for Hoplon is that it's simple. Hoplon doesn't use a virtual DOM, and thus it doesn't have a component lifecycle. One major benefit of this approach is in making it natural to use with existing Js libraries that expect to work with the browser DOM.

An example of this would be something like using a jQuery date picker widget. With Reagent, we'd have to use the lifecycle hooks, and make sure that the component is mounted in the browser DOM before the library is called. Conversely, we may need to consider the case of the component updating separately. While, it's not difficult to reason about in most cases, it does introduce some mental overhead. Using the same date picker in Hoplon can be seen here.

However, while I found the idea of Hoplon interesting, I've never gave it a serious look due to the fact that it looked to be a monolithic stack. When you read Hoplon documentation, it's easy to get the impression that it has to be used with Boot, you have to use special .hl files to define ClojureScript, and you're expected to work with its server implementation.

This all can be appealing if you're looking for a full-stack solution where decisions have been made for you, but it's a bit of a deterrent for somebody who already has a preferred workflow and uses other tools such as Figwheel and Leiningen.

After having a discussion with Micha on Reddit, I realized that this wasn't the case and decided to give Hoplon another shot.

The Setup

I used the reagent-template that I maintain as the base for he project by running the following command in the terminal:

lein new reagent hoplon-app

Next, I updated the dependencies in project.clj to remove the references to Reagent, and add the Hoplon dependency instead:

  :dependencies [[org.clojure/clojure "1.8.0"]
                 [ring-server "0.4.0"]
                 [hoplon "6.0.0-alpha15"]
                 [ring "1.4.0"]
                 [ring/ring-defaults "0.2.0"]
                 [compojure "1.5.0"]
                 [hiccup "1.0.5"]
                 [yogthos/config "0.8"]
                 [org.clojure/clojurescript "1.9.36"
                  :scope "provided"]
                 [secretary "1.2.3"]]

That's all the changes I had to do in order to switch to using Hoplon in the project.

The next step was to open up the ClojureScript source in the src/cljs/hoplon_app/core.cljs file and replace the references to Reagent with Hoplon:

(ns hoplon-app.core
  (:require
    [hoplon.core
     :as h
     :include-macros true]
    [javelin.core
     :refer [cell]
     :refer-macros [cell= dosync]]))

(h/defelem home []
  (h/div
    :id "app"
    (h/h3 "Welcome to Hoplon")))

(defn mount-root []
  (js/jQuery #(.replaceWith (js/jQuery "#app") (home))))

(defn init! []
  (mount-root))

At this point I could start Figwheel and see the page load in the browser by running:

lein figwheel

As you can see the main difference so far is that we mount the Hoplon DOM using plain jQuery call, and the elements are defined using Hoplon helper macros.

Let's see how we can add a bit of state to our Hoplon app. Hoplon state management is handled by the Javelin library. It uses a similar concept to the Reagent atom where we can define cells, and then whenever the state of the cells changes any elements that are looking at its value will be notified.

We'll create a simple to-do list to illustrate how this works. First, we need to create a cell to hold the data. We'll add the following code at the top of the namespace to do that:

(def todo-items (cell ["foo"]))

The above code will define a Javelin cell that contains a vector with the string "foo" in it. We can now render the value of the cell as follows the the home element:

(h/defelem home []
  (h/div
    :id "app"
    (h/h3 "Welcome to Hoplon")
    (h/p (cell= todo-items))))

The cell= call is reactive and whenever the state of the cell changes the paragraph will be repainted to with its current value. We can now add some code to add new items to the to-do list:

(h/defelem add-todo []
  (let [new-item (cell "")]
    (h/div
      (h/input :type "text"
               :value new-item
               :change #(reset! new-item @%))
      (h/button :click #(dosync
                          (swap! todo-items conj @new-item)
                          (reset! new-item ""))
                (h/text "Add #~{(inc (count todo-items))}")))))

The above code should be fairly familiar to anybody who's used Reagent. We define a local state in a let binding and create a div that contains an input and a button. The input displays the value of the new-item cell and updates it in its :change event. Meanwhile, the button will swap the todo-items cell and add the value of the new item, then reset it to an empty string.

Notice that the button text displays the current item count. This is accomplished by Hoplon #~ helper that allows us to easily display cell values within strings.

We should now be able to update our home element as follows to have the add-todo component show up on the page:

(h/defelem home []
  (h/div
    :id "app"
    (h/h3 "Welcome to Hoplon")
    (h/p (cell= todo-items))
    (add-todo)))

When we add to-do items, they should be showing up in the list. So far everything looks nearly identical to working with Reagent.

Now, let's update the items to be rendered in the list a bit nicer. We'll write the following element to render the list:

(h/defelem todo-list [{:keys [title]}]
  (h/div
      (h/h4 (or title "TODO"))
      (h/ul
        (h/for-tpl [todo todo-items]
          (h/li todo)))))

The element uses the Hoplon for-tpl macro to run through the elements in the list. The macro is used to map dynamically sized collections to DOM nodes. With the element in place, we can update our home element to display a nice HTML list:

(h/defelem home []
  (h/div
    :id "app"
    (h/h3 "Welcome to Hoplon")
    (todo-list {:title "TODO List"})
    (add-todo)))

We should now see a list of items displayed that will get updated as we add items using the add-todo element. That's all there's to it. While it's a trivial app, I hope it gives you a taste of what working with Hoplon is like. The full source for the project can be seen here.

Conclusion

I was very pleasantly surprised by how easy it was to use Hoplon in a project setup with Leiningen and Figwheel. The semantics that Hoplon provides are very similar to Reagent, and are arguably simpler since there's no need to worry about the component lifecycle.

The one aspect of Reagent that I prefer is that the UI is defined declaratively using the Hiccup syntax. This makes it possible to manipulate UI elements as plain data. However, I don't think that using functions to define the UI is a deal breaker.

Overall, I think that Hoplon is often overlooked when ClojureScript UI libraries are considered, and this is very unfortunate. It's a solid library that provides clean and simple semantics to the user.

If, like me, you've been avoiding Hoplon because you were under the impression that you have to use it in a specific way, then I strongly urge you to give it another look.

Update

Luminus now provides a Hoplon profile using the +hoplon flag.

Writing an expandable widget with Reagent

Fri, 20 May 2016 00:00:00 +0000

I recently needed to create an expandable widget and I wanted to be able to close it by clicking elsewhere on the screen. An example would be an input field and an associated component to select the input value such as a date picker.

We'll define an example component to look as follows:

(defn expandable-component []
  [:div
   [:input
    {:type :text}]
   [:table>tbody      
    (for [row (range 5)]
      [:tr
       (for [n (range 5)]
         [:td>button.btn
          {:on-click
           #(do
              (reset! value n)
              (reset! expanded? false))} n])])]])

Next, we'll use the with-let statement to define some state for the component.

(defn expandable-component1 []
  (r/with-let
    [expanded? (r/atom false)
     value     (r/atom nil)]
    [:div
     [:input
      {:type :text
       :value @value
       :on-click #(swap! expanded? not)}]
     [:table>tbody
      {:style (if @expanded?
                {:position :absolute}
                {:display "none"})}
      (for [row (range 5)]
        [:tr
         (for [n (range 5)]
           [:td>button.btn.btn-secondary
            {:on-click
             #(do
                (reset! value n)
                (reset! expanded? false))} n])])]]))

The table is now hidden by default, and it's displayed when the user clicks the input. The table contains cells with numbers. When the user clicks a number, then the table is hidden and the input is reset to the value selected.

This works fine. However, the only way we can hide the table is by either picking a number or clicking on the input itself. It's not terrible, but it would be nicer if we could simply click off the table to have it go away.

The problem is that there is no local event the widget can use to detect that the user clicked elsewhere. So, what can we do here?

The solution I ended up using was to use a combination of events to detect the state of the widget. Let's see how this works below.

First, I added the :on-blur event to the input. When the input loses focus, the table is hidden. Now if I click elsewhere on the screen the table will disappear as intended.

Unfortunately, this breaks the interaction with the table itself. Since now the focus goes away and I'm no longer able to select the number I want.

In order to get around that problem we can use the :on-mouse-enter and :on-mouse-leave events on the table. This way we can check if the mouse is in the table before changing the visibility.

(defn expandable-component []
  (r/with-let
    [expanded? (r/atom false)
     value     (r/atom nil)
     mounse-on-table? (r/atom false)]
    [:div
     [:input
      {:type :text
       :value @value
       :on-click #(swap! expanded? not)
       :on-blur #(when-not @mounse-on-table? (reset! expanded? false))}]
     [:table>tbody
      {:style (if @expanded? {:position :absolute} {:display "none"})
       :on-mouse-enter #(reset! mounse-on-table? true)
       :on-mouse-leave #(reset! mounse-on-table? false)}
      (for [row (range 5)]
        [:tr
         (for [n (range 5)]
           [:td>button.btn.btn-secondary
            {:on-click
             #(do
                (reset! value n)
                (reset! expanded? false))} n])])]]))

The new approach works as intended. The table will now close whenever the user clicks outside it. You can see this in action here.

Hopefully this trick will save you some time creating these types of components in Reagent.

Luminus Workflow

Thu, 28 Apr 2016 00:00:00 +0000

I recently presented at the Philly ETE conference, and it was a really great experience. The conference was well organized, there were lots of great talks, and I got to meet a bunch of interesting people.

My talk focused on the workflow using Luminus and Reagent. During the talk I built a simple app from scratch. The app illustrates how to work with a relational database, create a documented service API, and build a UI for it using Reagent. The live demo portion stats around the 11 minute mark.

If you're interested in my workflow using Luminus and Cursive, then I definitely recommend watching the talk.

The State of Clojure Web Development

Sun, 17 Apr 2016 00:00:00 +0000

The Kickstarter for the Arachne framework was just announced. I think this is very exciting, and I sincerely hope that it will be successful. There is plenty of room for frameworks in the Clojure web application domain. However, I also think that the pitch in the video severely misrepresents the current state of Clojure web development.

Is it hard to make an app?

Luke says that the motivation for the project is that there is no simple way to put a Clojure web app together. You want to make a website quickly, but unfortunately Clojure is not well suited for this task because the lead time is just too long.

Luke goes as far as to say that starting a new Clojure web application with all the parts together, that's actually deployable is a one to three months process.

Simplifying this process is precisely the motivation behind Luminus. In fact, Luminus, and other frameworks such as Hoplon, have been filling this exact niche for years now. While I’m not as familiar with Hoplon, I will focus on contrasting the stated goals behind Arachne and the goals for Luminus.

First thing I’d like to address is the claim that it takes a long time to create a web application following best practices. Creating a new Luminus app is as easy as running lein new luminus myapp in the terminal. Perhaps what Luke meant was that creating an application using his preferred stack and defaults takes a long time.

Luminus is based on over a decade of experience working in the enterprise environment and building real world applications. It's built on top of mature libraries that are known to work well together. These are wrapped up in a template that follows the best practices, and makes it easy to create a working application that's ready for deployment out of the box.

Some of the things Luke lists are:

Overall structure
Resource lifecycle management
Routing
Content negotiation
HTML rendering
Authentication & authorization
Validation
Logging
Testing

All of these are supported out of the box in Luminus.

What about beginners?

Another problem Luke identifies is that there needs to be a tool for beginners, or people new to the Clojure language. Once again, this is precisely the target demographic for Luminus.

I've literally spent years working with existing libraries, creating my own when necessary, writing documentation, and putting things together for that express purpose. I've even written a couple of books on this topic now.

Arachne aims to experiment with creating an easy to start with solution that will scale. Luminus is designed to scale, and it’s currently being used in production in the enterprise. It's not experimental in any way, it's an actual proven solution that exists today.

Luminus allows you to start fast and deploy out of the box, but it is also designed to be built on as you go. Like Arachne aims to do, Luminus already embraces modular design. It's built on top of battle tested libraries such as Ring and Compojure, but it doesn't lock you into doing things a particular way.

Luminus makes it trivial to swap things like the underlying HTTP server, templating engine, the database you're using, and so on. The core template provides a minimal skeleton app. This template can then be extended using hints to provide additional functionality.

But is it modular?

Arachne has an ambitious goal to provide a way to specify the application using a data driven approach. The idea being that this makes it easier to swap different components in the existing project.

I’ve considered similar approaches for Luminus, but ultimately decided against that. First, I think that Ring middleware already provides an extremely powerful mechanism for injecting functionality in the request handling pipeline. This is where most of the changes will happen in your project. You might decide to swap out or change things like session handling middleware as your project evolves.

However, my experience is that in most cases it’s not possible to simply swap a component such as the database for a different one without having to change some code in the application.

For example, if I switch the templating engine, then I have to update my HTML templates. When I switch a database from SQL to Datomic, I have to rewrite the queries and the business logic. That’s where the most of effort will end up being spent.

That said, the code that deals with any particular component in Luminus is minimal by design. So, the vast majority of the code you’d have to change would be the code that you’ve written yourself.

The one place I found it to be possible to provide swappable components is the underlying HTTP server. Luminus provides wrapper libraries for all the popular servers, and it’s possible to swap them out by simply changing the dependency in the project.

I think it would be possible to build things like authentication and authorization modules that are swappable. However, a generic solution is necessarily more complex than a focused one. A component that can be used in many different situations will always be more complex than one that solves a specific problem.

For this reason, I firmly believe that such design decisions should be left up to the user. The libraries should provide focused functionality, while the user decides how to put them together in a way that makes sense for their project.

Conclusion

At the end of the day, Luminus isn’t based just on my experience, but also that of the contributors and the users over the years. Arachne will be informed by Luke’s experience and that necessarily means that it will provide a new and interesting way to put together Clojure web applications.

Overall, I think it will be great to see a new web framework for Clojure. There is plenty of room for alternatives to Luminus, and Arachne could explore many areas that aren't the focus for Luminus at the moment. Therefore, I wholeheartedly urge you to support the Kickstarter so that we can have more choices for Clojure web development.

Luminus embraces HugSQL

Mon, 22 Feb 2016 00:00:00 +0000

There are lots of Clojure libraries available for working with SQL. The core libraries that provide the low level functionality are clojure.java.jdbc and the more recent clojure.jdbc. Some of the more popular libraries built on top of them are Korma, Honey SQL, and Yesql.

I've been a huge fan of the approach that Yesql takes since it was released. Every time I've worked with a DSL for SQL, I've found that I'd always run into cases where I knew exactly what to do if I was writing plain SQL, but I couldn't find a clean way to express using the abstraction on top of it. Since Yesql approach lets you keeps your SQL as SQL the problem largely goes away.

Luminus has been using Yesql since it came out and I think it made the framework much more approachable. Unfortunately, Yesql doesn't appear to be actively developed, and I found myself falling back to using clojure.java.jdbc directly for things like batch inserts.

Another problem from Luminus perspective is that Yesql API defines query functions directly. Luminus uses conman for connection management, and it creates its own connection-aware functions. This required an ugly hack of using a shadow namespace for interning the functions generated by Yesql.

I recently learned about the HugSQL library that is inspired by Yesql, and addresses all the issues I've run into using it. The official site does a good job enumerating the major differences from Yesql. Some of the highlights for HugSQL are:

snippets that facilitate composable queries
support for multi-row inserts
supports multiple backends such as clojure.java.jdbc and clojure.jdbc
great documentation

The latest version of HugSQL provides an API that returns a map of query functions keyed on their names as well as the ability to define the functions directly.

I think this is a very useful feature even if you're not using conman or Luminus. Having a map of the query functions allows the user to decide what they want to do with them explicitly. For example, you're able to do things like the following:

(def queries (hugsql.core/map-of-db-fns "queries.sql")

(defn get-user [db opts]
  ((-> queries :get-user :fn) db opts))

Yesql vs HugSQL

Let's take a look at the basic usage of HugSQL and differences from Yesql.

The core syntax in HugSQL is quite similar to Yesql. Both Yesql and HugSQL use comments with a special format to provide the metadata for generating the functions to work with queries.

Yesql

-- name: create-user!
-- creates a new user record
INSERT INTO users
(id, pass)
VALUES (:id, :pass)

-- name: get-users
-- retrieve all users
SELECT * FROM users

-- name: get-user
-- retrieve a user given the id.
SELECT * FROM users
WHERE id = :id

Yesql uses the -- name: fn-name syntax to specify the function name, the comment below the function name is implicitly used as the doc for the function. The ! at the end of the function name is used as a convention to indicate that it mutates the data. The query parameter placeholders are identified using the : prefix.

HugSQL

-- :name create-user! :! :n
-- :doc creates a new user record
INSERT INTO users
(id, pass)
VALUES (:id, :pass)

-- :name get-users :? :*
-- :doc retrieve all users
SELECT * FROM users

-- :name get-user :? :1
-- :doc retrieve a user given the id
SELECT * FROM users
WHERE id = :id

The HugSQL version uses the -- :name syntax instead that mirrors the Clojure keyword syntax. The function name is followed by two additional flags. The first flag indicates the SQL command type and the second indicates the result.

This provides more flexibility for handling the results. For example, the get-users query indicates that it selects multiple records, while the get-user indicates that it selects exactly one record. This helps document the intent of the query and cuts down on boilerplate, as you'd have to write a wrapper otherwise that gets the first result from the query.

command flags

:query or :? - query with a result-set (default)
:execute or :! - any statement
:returning-execute or :<! - support for INSERT ... RETURNING
:insert or :i! - support for insert and jdbc .getGeneratedKeys

result flags

:one or :1 - one row as a hash-map
:many or :* - many rows as a vector of hash-maps
:affected or :n - number of rows affected (inserted/updated/deleted)
:raw - passthrough an untouched result (default)

In HugSQL, all the comments that represent metadata start with a key describing the type of metadata. In the examples above, the doc string is explicitly specified using the -- :doc prefix.

HugSQL also supports additional syntax within its queries. For example, if we wanted to insert multiple records using a single query, then we could use a vector of records as follows:

-- :name add-users! :! :n
-- :doc add multiple users
INSERT INTO users
(id, pass)
VALUES :t*:users

(add-users! db {:users
                [["bob" "Bob"]
                 ["alice" "Alice"]]})

The syntax for for in-list queries is also a bit different from Yesql. The SQL query uses the :v* flag to indicate the value list parameter.

-- :name find-users :? :*
-- :doc find users with a matching ID
SELECT *
FROM users
WHERE id IN (:v*:ids)

The function parameters will now consist of a map with the key :ids that points to a vector of ids that we would like to match on.

(find-users db {:ids ["foo" "bar" "baz"]})

As you can see, the syntactic differences for basic queries are very minor. I've migrated a number of projects to HugSQL already, and found the process to be completely painless.

I haven't covered the advanced features of HugSQL, but I highly recommend looking over the official documentation to see what's available.

Contrasting Component and Mount

Tue, 19 Jan 2016 00:00:00 +0000

There was a recent wave of discussions on pros and cons of using Component and Mount approaches to state management. Both libraries aim to provide a clean way to manage stateful resources in the application. However, each one takes a very different approach.

Component is the currently accepted way to manage state, and it works well when you structure your application around it. However, it does require certain trade-offs in order to make the most of it. Let's take a look at some of the reasons you may wish to choose Mount over Component for your project.

Managing the State with Component

Component uses the dependency injection approach to managing stateful resources in the application. A system map is used to track all the components and their relationships. This map is then passed around the application explicitly, and used to provide access to the resources.

This approach encourages coupling between the code managing the resources and the business logic. A common pattern is to pass the component system around the application. The system is injected from the top level, and then functions pick parts of the system to pass down until they're eventually used by a function that relies on a particular resource.

One side-effect of this design is that it becomes impossible to test any part of the application without having the resources available. Therefore, if we wish to run tests in the REPL, then we need to instantiate a separate system map using the test resources. This problem makes it important to be able to create multiple instances of the components at runtime.

Component and the REPL

Since Component is based on protocols it doesn't play well with the REPL workflow, as changes to defrecord do not affect the instances that have already been created. The whole app needs to be restarted in order to make sure that the REPL is still in a good state.

This problem is discussed in detail by Stuart Sierra in his post on the reloaded workflow. I find that the reloaded workflow used with Component is much closer to TDD than the traditional Lisp style REPL driven workflow.

However, one of the advantages of working with a language like Clojure is that we shouldn't need to run tests all that often. Since development is primarily done using the REPL, we should have a good idea of what the code is doing while we're working on it.

RDD provides a very tight feedback loop. I can develop the features interactively, then create the tests based on the REPL session once the code is doing what I need.

There are only a handful of situations where I find it necessary to run the full test suite. I run tests before I commit code, I run tests on the CI server, but I don't find it necessary to rerun tests any time I write a bit of code during development. I certainly shouldn't have to reload the whole app for changes to take effect.

I like the guard rail metaphor Rich Hickey used in his Simple Made Easy talk:

"I can make changes 'cause I have tests! Who does that?! Who drives their car around banging against the guard rails saying, "Whoa! I'm glad I've got these guard rails!"

This is a good reminder that the tests are primarily a sanity check. We should have confidence in our code because we understand it and are able to reason about it.

To facilitate understanding, most code in the application should be pure, and we shouldn't have to rely on any external resources to test such code. I think that it helps to treat core business logic as you would a library. It should be completely agnostic about where the data is coming from and where it's going.

However, Component encourages a different kind of design where business logic ends up being reliant on the resources. In this situation, it's no longer possible to test the business logic in isolation.

Finally, any functions that takes the system map as a parameter are invariably tied to it. This is at odds with treating functions as core reusable building blocks.

The Mount Approach

Mount takes the approach of encapsulating stateful resources using namespaces. This leads to a natural separation between the code that deals with state from the pure core of the application logic.

When the core is kept pure, then it can be safely tested in isolation without having to provide any mock resources to it. This also makes the code reusable out of the box.

Mount makes it trivial to integrate into existing applications. The app doesn't need to be designed up front to take advantage of it, as it does with Component.

Since Mount doesn't rely on protocols, it doesn't impact the REPL driven workflow. There's no need for an elaborate setup to facilitate reloading the whole application any time a change is made.

The primary disadvantages of Mount are that it's not possible to create multiple instances of a resource, and that we have to be mindful not to couple namespaces representing resources to the business logic.

Conceptually, most resources in the application are singletons by their very nature. When we have a database connection or a queue, there is precisely one external resource that we're working with.

As I illustrated earlier, the primary reason for having multiple instances of a resource is testing. Mount provides a simple solution for running tests with alternate implementations as described in its documentation. This facilitates practically the same workflow as Component, where an instance of the resource can be swapped out with another for testing. However, the bigger advantage is that we no longer need to have resources available to test majority of the code in the first place.

Another argument is that you may have different instances of the same type of resource. For example, an application might use multiple queues that all have the same API. In this case, we can use defrecord to define the class representing the API for the queue. We'll then manage the lifecycle of each instance using defstate.

While we do have to be mindful of our design when using Mount, the same is true for Component as well. For example, nothing stops us from putting the Component system in a var that's referenced directly. The same reasoning we use to avoid doing that should be used when working with Mount as well.

In general, I see very few downsides to using Mount and I see a lot of practical benefits, some of which I've outlined above.

Conclusion

I think that both Component and Mount have their own sets of trade-offs. Component is a framework that requires the application to be structured around it. Conversely, it necessitates a TDD style workflow to work with it effectively.

On the other hand, Mount is not prescriptive about the workflow or application structure. I think this makes it a more flexible solution that's a great fit for many kinds of applications. The flip side is that we have to be more careful about how we structure the application as Mount is agnostic regarding the architecture.

Web Development with Clojure 2nd Edition

Fri, 01 Jan 2016 00:00:00 +0000

Beta Release Update

Unfortunately, it looks like there's been a bit of editorial delay and we missed the 13th release date. The new date is set for the second of February. Apologies to everybody who's waiting for the release.

I'm glad to report that the second edition of Web Development with Clojure is just around the corner. The beta release is planned for January the 13th, and then I'm hoping to have the final release shortly after. The second edition took a bit longer than I anticipated, but I think the wait will be worth it.

What to Expect

The primary goal of the book is to provide a solid introduction to the world of Clojure web development. Clojure community is growing rapidly and most new users come from languages such as Java, Ruby, and Python.

My aim is to illustrate how to build typical web applications using Clojure in a style that's friendly to those who come from using a mainstream language. In order to keep the material accessible, I deliberately chose to focus on the core concepts and avoid some of the more advanced topics. Conversely, if you're already familiar with Clojure and would like to get into web development then the book will serve as a great introduction.

The first edition of my book came out at the time when Clojure web stack was in a great deal of flux. Noir micro-framework just got deprecated, Cognitect announced Pedestal, and ClojureScript was in its infancy. It was hard to predict where things would go from there.

Much has changed in the past year in Clojure web ecosystem, and the second edition of the book reflects that. While the first edition was fairly unopinionated and gave a broad overview of different libraries, the new version primarily focuses on the Luminus stack.

Majority of the tools and libraries I cover are the ones I've used professionally to build real world applications. This naturally means that there is an inherent bias towards my workflow and the way I like to build applications. On the other hand, if you're new to building Clojure web applications it's helpful to learn using a particular workflow that you can adjust to your needs as you get comfortable.

Just like the first edition, the book is based around projects that illustrate how to accomplish different tasks in a typical web app. Each chapter introduces a particular concept and gets the reader to work through it by building a project from scratch. By the end of the book the reader should be comfortable writing many typical web applications using Clojure and have the background to explore further on their own.

Topics Covered

The book will consist of the following chapter:

Getting Your Feet Wet
- takes the reader through setting up the environment and building a simple application using Luminus
Clojure Web Stack
- takes a step back and looks at Ring and Compojure in detail
Luminus Architecture
- provides an overview of the Luminus template and ways to organize your projects
Add ClojureScript
- illustrates how to convert the applications from the first chapter to a SPA style app using Reagent
Real-time Messaging With Websockets
- illustrates how to use Websockets in a Clojure/Script application using Sente
Writing RESTful Web Services
- covers the basics of using the compojure-api library to provide Swagger style service API
Database Access
- introduces clojure.java.jdbc and Yesql, and how to use these libraries to work with a relational database
Picture Gallery
- ties all the above concepts together by working through a picture gallery application
Finishing Touches
- covers testing and packaging the application for deployment

The book will also contain a number of appendices that deal with topics such as NoSQL databases.

What's Not Covered

As Clojure web ecosystem continues to evolve, many new tools and libraries, such as the JUXT stack, have appeared just this year. While I would love to cover them all, that simply wouldn't fit the goals I set for this project.

One notable omission from the book is Om Next. First, I'd like to say that it's a fantastic library and I think very highly of it as well as the ideas behind it. However, I simply haven't used it in anger as I have Reagent. I also think that Reagent is the simpler of the two libraries and therefore more suitable for beginners. I hope that the book will provide a solid foundation for the reader to explore libraries like Om on their own.

Trouble with AOT

Sat, 26 Dec 2015 00:00:00 +0000

I recently ran into an interesting issue when I added the slf4j-timbre dependency to a project. As soon as the dependency was added the project would fail to build and I'd see the following error:

Caused by: java.io.FileNotFoundException: Could not locate clojure/tools/reader/impl/ExceptionInfo__init.class or clojure/tools/reader/impl/ExceptionInfo.clj on classpath.

The slf4j-timbre library does not depend on clojure.tools.reader, and at first glance there's nothing in it that should've caused problems. I did notice that the library depends on com.taoensso/timbre 4.1.4 that in turn depends on com.taoensso/encore 2.18.0, and it uses on an older version of clojure.tools.reader.

At this point I thought the solution would be simple. I'd just include the latest version of encore and everything should work fine. However that didn't turn out to be the case.

I decided to take another look at slf4j-timbre to see what else might be happening. At this point I noticed that it uses :aot :all in the project configuration. This causes the library to be compiled to Java classes as opposed to being distributed at source. This is necessary since the library has to implement the SLF4J interface and has to provide a Java class in its implementation.

When the namespace that references Timbre is compiled then any namespaces it depends on are also compiled and packaged in the jar. These compiled classes will take precedence over the source dependencies when the library is included in the project.

So, even though I was explicitly including the version of encore that uses the latest clojure.tools.reader, the compiled version packaged in slf4j-timbre would end up being used causing the exception above. As far as I can tell there's no way to overwrite these in the project configuration.

Implications for Luminus

Unfortunately, Luminus dependencies require both a SLF4J compliant logger and the latest clojure.tools.reader. While I think Timbre is an excellent library, it's just not the right fit at the moment.

Luckily, clojure.tools.logging provides a SLF4J compliant API for Clojure logging. The latest release of Luminus uses clojure.tools.logging along with the log4j library as the default logging implementation. It's a mature library that has excellent performance and provides a plethora of logging appenders.

Since log4j can be configured using a properties file, it fits the Luminus approach of using 12 factor style configuration. The library will look for a file called log4j.properties on the classpath to get its default configuration. Luminus packages this file in the resources folder with the following configuration:

### stdout appender
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d][%p][%c] %m%n

### rolling file appender
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=./log/app-name.log

log4j.appender.R.MaxFileSize=100KB
log4j.appender.R.MaxBackupIndex=20

log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=[%d][%p][%c] %m%n

### root logger sets the minimum logging level
### and aggregates the appenders
log4j.rootLogger=DEBUG, stdout, R

As you can see the configuration is very straight forward, it's also well documented. The default configuration can be overridden at runtime by setting the :log-config environment variable. You can now create a custom logging configuration on the target system and then set an environment variable to point to it as seen below:

export LOG_CONFIG=prod-log.properties

I think that the new approach provides a solid solution for most situations with minimal changes from the existing behavior.

Final Thoughts

The moral of the story is that you want to be very careful when using AOT in libraries. Whenever possible it is best to avoid it, and if you do have to use it then try to find the minimal subset of the namespaces that absolutely have to be compiled.

Managing State in Luminus

Sat, 05 Dec 2015 00:00:00 +0000

The problem of State

Most real-world applications will interact with external resources such as databases. Typically, in order to interact with a database we have to initialize a connection when our application starts, use this connection to access the database, and then tear it down when the application shuts down.

In some cases these resources may even depend on one another. We may be getting configuration from one resource and then using it to initialize another. A large application may have a number of different resources that are interdependent and have to be coordinated.

Using Component

One popular approach to addressing this problem is to use the component library. Component creates a graph that describes all the resources and then we pass it around to any functions that rely on them. This library was originally developed to support the reloaded workflow advocated by Stuart Sierra.

The advantage of this approach is that it allows us to keep the application code completely stateless and lets us inject the stateful resources at runtime. The two main benefits are that the core application logic remains pure and we can easily provide mock resources to it for testing. If you're interested in learning more about building applications based on component I recommend checking out the Duct framework by James Reeves that I covered in an earlier post.

I think that's a fine approach for building applications, but I also think that there are tradeoffs that one has to buy into when using component.

Component introduces simplicity by providing a formal separation between pure and impure code, but also adds complexity in terms of the structure of the application.

The application has to keep a global component graph that tracks the relationships between the resources and pass it explicitly to any code that needs to interact with them. My experience is that this introduces boilerplate and indirection making the overall application structure more complex. Component also requires the developer to adopt a specific workflow to take full advantage of it.

Component can also be rather confusing for beginners and I have avoided using it in Luminus for that reason. However, the problems that component addresses are real and if we're not using component we still need a way to address them. For this reason Luminus uses the mount library to orchestrate the stateful resources in the application.

Using Mount

Unlike component, mount does not require us to structure the application in a specific way or to adopt a particular workflow to use it.

The library leverages the existing namespace hierarchy to resolve the resource dependencies. This approach obviates the need to keep a separate component graph and pass it all over the application.

Mount uses the defstate macro to define stateful resources. The macro allows specifying the code to initialize and tear down the resource by associating it with the :start and :stop keys. In case of a connection we would associate the code that initializes the connection in the :start key and the code that tears it down with the :stop key respectively.

Mount will look for any namespaces that define states with defstate and and compile a set of stateful resources based on that. The resources are started and stopped based on the order of reference of their namespaces.

Mount system is started by calling mount.core/start and stopped using mount.core/stop. This ensures that the lifecycle of the resources is managed in automated fashion and their relationships are all accounted for.

Like component, mount supports the reloaded workflow and even provides ways to restart parts of the application. Mount also supports swapping in alternate implementations for the resources. This provides a simple way to run tests with mock resources without having to restart the REPL.

Structuring the Application

While mount provides us with a solution for managing the lifecycle of the components, we still need a way to ensure that our application is easy to reason about. Since mount does not enforce the separation between pure and impure code, we have to structure the application in such a way that side effects aren't mixed into the core application logic.

Encapsulating Resources

The approach I like to take in my applications is to keep the code that interacts with external resources at the edges of the application. The core business logic should be pure and testable, while anything that deals with side effects and external resources should be pushed to a thin layer around it.

I also find it useful to localize resource management in order to reduce coupling between components. For example, when I'm dealing with a database resource I'll create a namespace to manage it. This namespace will be responsible for handling the connection lifecycle internally and providing the connection to the functions defined in it.

Such a namespace provides an API for interacting with the resource for the rest of the application. Any functions calling this API do not have to be aware of its internal implementation.

My experience is that this approach allows compartmentalizing the application into self-contained components that can be reasoned about individually. When I update the internal implementation of a particular component the rest of the application does not need to be aware of the change.

An example of this would be changing the underlying resource. We may start writing the application by using a database directly, then realize that the functionality can be extracted into a shared service. When the mechanics of communicating with an external resource are internal to the component we can safely update it to use the new type of resource without affecting the rest of the application.

Organizing the Components

The workflows in web applications are typically driven by the client requests. Since requests will often require interaction with a resource, such as a database, we will generally have to access that resource from the route handling the request. In order to isolate the stateful code we should have our top level functions deal with managing the side effects.

Consider a concrete example. Let's say we have a route that facilitates user authentication. The client will supply the username and the password in the request. The route will have to pull the user credentials from the database and compare these to the ones supplied by the client. Then a decision is made whether the user logged in successfully or not and its outcome communicated back to the client.

In this workflow, the code that deals with the external resources should be localized to the namespace that provides the route and the namespace that handles the database access.

The route handler function will be responsible for calling the function that fetches the credentials from the database. The code that determines whether the password and username match represents the core business logic. This code should be pure and accept the supplied credentials along with those found in the database explicitly. This structure can be seen in the diagram below.

            pure code
+----------+
| business |
|  logic   |
|          |
+-----+----+
      |
------|---------------------
      |     stateful code
+-----+----+   +-----------+
|  route   |   |           |
| handlers +---+  database |
|          |   |           |
+----------+   +-----------+

Keeping the business logic pure ensures that we can reason about it and test it without considering the external resources. Meanwhile the code that deals with side effects is pushed to the top making it easy for us to manage it.

Conclusion

Clojure makes it easy to structure the application in such a way that the core of the application logic is kept pure. Doing this is a very good practice and will help you keep your applications manageable as they continue to grow. While it's possible to formalize the handling of stateful resources, using libraries such as component, I personally have not found this to be necessary in my applications.

I hope this post provides a bit of an insight into how Luminus based applications should be structured for long term maintainability.

The Sky Is not Falling

Sat, 28 Nov 2015 00:00:00 +0000

A recent post by Elben Shira boldy proclaims the end of dynamic languages. There was a great followup by Maxime Chevalier-Boisvert that I'd like to expand on a bit in this post.

I think that people often forget that programming is a human endeavor. When all is said and done what really matters is whether you enjoy working with a particular language or not. Of course, different people like different things and hence the plethora of languages available today.

I would not presume to tell people that the way I develop is the one true way. I've found an approach that works for me, I know I'm productive with it, and most importantly I enjoy it.

The truth is that this is the case for everybody else out there as well. Anybody who tells you that they found the one true way is frankly deluded. There's no empirical evidence to show that the typing discipline is the deciding factor of code quality, and everybody out there is using their own anecdotal experience to find the workflow that works for them.

Proponents of static typing accept its benefits as axiomatic. However, I think that's a case of putting the cart before the horse. Let's take a look at the claims from the perspective of a dynamic language user.

The Case for Static Typing

Static typing provides a way to formally track relationships in code and thus allows catching a certain class of errors at compile time. The advantage of this approach is that it becomes possible to guarantee that these types of errors cannot occur at runtime.

Many proponents of static typing argue that this is a common source of errors in dynamic languages and that it's not possible to write and maintain large codebases in absence of static types. It's common to see assertions such as the following:

Programmers have come to embrace dynamically typed languages for prototyping and delivering large and complex systems. When it comes to maintaining and evolving these systems, the lack of explicit static typing becomes a bottleneck.

In practice, very little research has been done to determine whether this is a major source of errors for applications written in dynamic languages, and the few studies that are available don’t show anything conclusive in this regard.

Furthermore, there’s no evidence that real world projects written in statically typed languages produce superior results to their dynamic counterparts. In fact, some of the largest and most robust systems out there are written in dynamic languages such as Erlang:

Erlang is used all over the world in high-tech projects where reliability counts. The Erlang flagship project (built by Ericsson, the Swedish telecom company) is the AXD301. This has over 2 million lines of Erlang.

The AXD301 has achieved a NINE nines reliability (yes, you read that right, 99.9999999%). Let’s put this in context: 5 nines is reckoned to be good (5.2 minutes of downtime/year). 7 nines almost unachievable ... but we did 9.

Erlang is a poster child for robust and fault tolerant systems. However, plenty of large projects have been written in other dynamic languages as well. There's a good chance that a piece of software you rely on daily is written using a dynamic language and it works just fine.

Complexity inherent in static typing

Since static typing sounds like a net win on paper, the obvious questions are why many people prefer dynamic languages and why hasn't static typing been decisively shown to be more effective.

The main drawback of static typing is that you're required to prove what you're stating to the compiler. Anybody who has done proofs knows that stating something is always simpler than proving it. In fact, many things are very simple to state, but are notoriously difficult to prove. Fermat's last theorem is a famous example of this.

Baking a proof into the solution leads to incidental complexity. Once you run into limits of what the type system can easily express then you end up having to write increasingly more convoluted code to satisfy it.

This results in code that’s harder to understand because it compounds the complexity of the problem being solved with the complexity of expressing it using the type system. Effectively, any statement we make in our program has to be accompanied by a proof of correctness to make it possible for the compiler to verify it. The requirement of proving that the code is self-consistent is often at odds with making it simple.

A concrete example of this would be the use of state monad to formally represent mutation in a language like Haskell. Here's what Timothy Baldridge has to say about his experience trying to apply this pattern in Clojure when working on the core.async library:

When I first wrote the core.async go macro I based it on the state monad. It seemed like a good idea; keep everything purely functional. However, over time I've realized that this actually introduces a lot of incidental complexity. And let me explain that thought.

What are we concerned about when we use the state monad, we are shunning mutability. Where do the problems surface with mutability? Mostly around backtracking (getting old data or getting back to an old state), and concurrency.

In the go macro transformation, I never need old state, and the transformer isn't concurrent. So what's the point? Recently I did an experiment that ripped out the state monad and replaced it with mutable lists and lots of atoms. The end result was code that was about 1/3rd the size of the original code, and much more readable.

So more and more, I'm trying to see mutability through those eyes: I should reach for immutable data first, but if that makes the code less readable and harder to reason about, why am I using it?

In a language that forces us to use a particular formalism to represent this problem there would be no alternative solution. While the resulting code would be provably correct, it would be harder for the developer to reason about its intent. Therefore, it's difficult to say whether it's correct in any meaningful sense.

Ultimately, a human needs to be able to understand what the code is doing and why. The more complexity is layered on top of the original problem the more difficult it becomes to tell what purpose the code serves.

As another example, let’s consider what we would need to understand to read an HTTP request using a Haskell web framework such as Scotty. We'll quickly run into ScottyM type that's defined as type ScottyM = ScottyT Text IO. To use it effectively we need to understand the ScottyT. It in turn requires understanding the ReaderT.

Understanding ReaderT relies on understanding of monads, monad transformers and the Reader monad. Meanwhile, to understand the Reader we have to know about functors and applicatives. To understand these we have to understand higher kinded types and constructor classes. This leads us to type classes, type constructors, algebraic datatypes, and so forth.

All of this is needed to satisfy the formalisms of the Haskell type system and is completely tangential to the problem of reading HTTP requests from a client.

Of course, one might argue that Haskell is at the far end of the formal spectrum. In a language with a more relaxed type system you have escape hatches such as casting and unchecked side effects.

However, once you go down that road then it's only a matter of degrees with how relaxed a system you're comfortable with. At this point you've already accepted that working around the type system can make your life easier.

Rendering Reagent on the Server using Hiccup

Tue, 24 Nov 2015 00:00:00 +0000

I recently watched a great talk titled Optimizing ClojureScript Apps For Speed, where Allen Rohner discusses the benefits and hurdles of server-side rendering.

React supports hooking in to server generated HTML at runtime. However, since React is a JavaScript library it becomes problematic to leverage this functionality from Clojure. While the JVM provides a Js runtime with Nashorn, it's extremely slow and requires a lot of twiddling to work for even basic examples.

Another approach is to run an instance of Node.js and farm out React rendering to it. This avoids the limitations of Nashorn, but introduces a host of new problems described in the talk.

Allen then proposes an alternative approach where he implements parts of the Om API and cross-compiles the components that way. You can see how this works in his Foam library.

The main difficulty identified in the talk is in implementing a sufficient amount of Om API in order to generate HTML on the server.

This got me thinking about what it would take to leverage this approach using Reagent. Unlike Om, Reagent has a tiny API and the only part of it used to create components is the Reagent atom implementation. The components themselves are written using plain Hiccup syntax.

Let's see how this could work. We'll start by creating a new Reagent project:

lein new reagent reagent-serverside

Next, we'll add a new namespace in called reagent-serverside.home src/cljc/reagent_serverside/home.cljc. This namespace will house the home page component that we'll pre-render on the server.

All we have to do now is to use a reader conditional to only require the Reagent atom during ClojureScript compilation:

(ns reagent-serverside.home
  #?(:cljs
     (:require [reagent.core :as reagent :refer [atom]])))

We can now write our components as we would normally:

(ns reagent-serverside.home
  #?(:cljs
     (:require [reagent.core :as reagent :refer [atom]])))

(def items (atom nil))

(defn item-list [items]
  [:ul
   (for [item items]
     ^{:key item}
     [:li item])])

(defn add-item-button [items]
  [:button
   {:on-click #(swap! items conj (count @items))}
   "add item"])

(defn home-page []
  [:div [:h2 "Welcome to reagent-serverside"]
   [add-item-button items]
   [item-list @items]])

We'll have the items atom to house a collection of items, an item-list function to render it, and the home-page function that will use the item-list component. We also have a button that lets the user add new items with an :on-click event. This is all standard Reagent code.

Rendering on the Server

Now, let's navigate to the reagent-serverside.handler namespace and reference the reagent-serverside.home we just created.

(ns reagent-serverside.handler
  (:require ...
            [reagent-serverside.home :refer [items home-page]]))

We'll now have to write the functions that will traverse the components and render them as appropriate. We'll attach a :data-reactid key to each one to give it an identifier that React looks for, and inject the result into our Hiccup markup.

(defn react-id-str [react-id]
  (assert (vector? react-id))
  (str "." (clojure.string/join "." react-id)))

(defn set-react-id [react-id element]
  (update element 1 merge {:data-reactid (react-id-str react-id)}))

(defn normalize [component]
  (if (map? (second component))
    component
    (into [(first component) {}] (rest component))))

(defn render
  ([component] (render [0] component))  
  ([id component]
   (cond
    (fn? component)
    (render (component))    

    (not (coll? component))
    component
    
    (coll? (first component))
    (map-indexed #(render (conj id %1) %2) component)
    
    (keyword? (first component))
    (let [[tag opts & body] (normalize component)]
      (->> body
           (map-indexed #(render (conj id %1) %2))
           (into [tag opts])
           (set-react-id id)))
    
    (fn? (first component))
    (render id (apply (first component) (rest component))))))

(reset! items (range 10))

(def mount-target
  [:div#app (render home-page)])

The render function will recursively walk the components evaluating any functions it finds and assigning the React id to each element.

Next, we'll set the items atom to a range of numbers, and then call render inside the mount-target to generate the markup.

Rendering on the Client

Finally, let's navigate to the reagent-serverside.core namespace in the src/cljs source path. We'll update it to reference the home namespace we created and render the home-page component on load.

(ns reagent-serverside.core
    (:require [reagent.core :as reagent :refer [atom]]
              [reagent-serverside.home :refer [items home-page]]))

(defn mount-root []
  (reagent/render [home-page] (.getElementById js/document "app")))

(defn init! []
  (reset! items (range 20))
  (mount-root))

When we load the page we'll immediately see the server generated markup and then it will be updated by Reagent when ClojureScript is loaded. There are a few caveats here that you should be aware of.

Any components you wish to render on the server have to be written in cljc, so you may end up having to add some shims for things like Ajax calls.

The component syntax has to work with both Reagent and Hiccup, so you have to be mindful to use the common subset.

React is fairly picky about the structure and the data-reactid tags. So, it can be tricky to generate a DOM tree that it likes. The example in the post will give a React warning about the DOM being different. Some more work is needed around this.

However, even in the case that React doesn't reuse the DOM, the user will see the page immediately and you'll get the benefits of SEO for your site.

Full source is available on GitHub.

Conclusions

Overall, I'm very happy with the results and it looks like it would be fairly easy to wrap this up into a library. The data focused approach is a huge win for Reagent here in my opinion. Since the components are laid out using regular Clojure data structures there's no need to implement any special API and things just work out of the box.

Evaluating ClojureScript in the Browser

Thu, 12 Nov 2015 00:00:00 +0000

ClojureScript can now compile itself without relying on the Google Closure compiler, and it's now possible to evaluate code straight in the browser. In this post we'll look at how that's accomplished by creating a code editor using CodeMirror, highlight.js, and Reagent. The code entered in the editor will be sent for evaluation and the result displayed to the user.

Let's start by creating a new Reagent project by running the following command:

lein new reagent cljs-eval-example +figwheel

Next, we'll navigate to the project folder and start Figwheel by running:

lein figwheel

Let's open the browser at http://localhost:3449 and navigate to the cljs-eval-example.core namespace in the src/cljs folder. We'll first need to reference cljs.js namespace:

(ns cljs-eval-example.core
  (:require
   ...
   [cljs.js :refer [empty-state eval-str js-eval]]))

We'll create evaluate function that will accept a string and a callback. This function calls cljs.js/eval-str as follows:

(defn evaluate [s cb]
  (eval-str
     (empty-state)
     s
     nil
     {:eval       js-eval    
      :source-map true
      :context    :expr}
     cb))

The eval-str function accepts an initial state, followed by the string representing the form to evaluate, a name, a map with the options, and a callback function for handling the result of the evaluation. We can create an initial state by calling cljs.js/empty-state function. We can now test that our code works by adding a button to our home-page component:

(defn home-page []
  [:div
   [:button
    {:on-click #(evaluate "(println \"hello world!\")" (fn [_]))}
    "let's compile!"]])

When we click the button we should see "hello world!" printed in the browser console. Next, let's add a :textarea to allow entering some text and then send it for evaluation.

(defn home-page []
  (let [input (atom nil)
        output (atom nil)]
    (fn []
      [:div
       [:textarea
        {:value @input
         :on-change #(reset! input (-> % .-target .-value))}]       
       [:div>button
        {:on-click #(evaluate @input (fn [result] (reset! output result)))}
        "let's compile!"]
       [:p @output]])))

At this point we can type some code in our input box, click the button to evaluate it, and see the result. So far so good, now let's make the editor look a bit nicer by replacing it with the CodeMirror version.

We'll open up the cljs-eval-example.handler namespace in the src/clj folder. There, we'll update the include-css and include-js portions of the head to add the respective CSS and Js files for running CodeMirror.

(defn head []
  [:head
   [:meta {:charset "utf-8"}]
   [:meta {:name "viewport"
           :content "width=device-width, initial-scale=1"}]
   (include-css
    (if (env :dev) "/css/site.css" "/css/site.min.css")
    "//cdnjs.cloudflare.com/ajax/libs/codemirror/5.8.0/codemirror.min.css"
    (if (env :dev) "css/site.css" "css/site.min.css"))
   (include-js
    "//cdnjs.cloudflare.com/ajax/libs/codemirror/5.8.0/codemirror.min.js"
    "//cdnjs.cloudflare.com/ajax/libs/codemirror/5.8.0/mode/clojure/clojure.min.js")])

With that in place we'll need to reload the page for the new assets to become available. Since we're using external JavaScript that modifies the DOM, we'll need to use the reagent.core/create-class function to create the editor component.

The create-class function accepts a map keyed on the React lifecycle methods. The methods that we wish to implement are :render and :component-did-mount:

(defn editor [input]
  (reagent/create-class
   {:render (fn [] [:textarea 
                     {:default-value ""
                      :auto-complete "off"}])
    :component-did-mount (editor-did-mount input)}))

The editor component will accept the input atom as the parameter and pass it to the editor-did-mount function. This function will look as follows:

(defn editor-did-mount [input]
  (fn [this]
    (let [cm (.fromTextArea  js/CodeMirror
                             (reagent/dom-node this)
                             #js {:mode "clojure"
                                  :lineNumbers true})]
      (.on cm "change" #(reset! input (.getValue %))))))

The editor-did-mount is a closure that returns a function that accepts the mounted React component, it then calls reagent/dom-node on it to get the actual DOM node mounted in the browser. We'll then call .fromTextArea method on js/CodeMirror and pass it the node along with a map of rendering hints.

Calling .fromTextArea returns an instance of the CodeMirror. As a last step we'll add the change event to this instance to reset the input atom with the updated text whenever the text in the editor is changed.

We can now update the home-page component to use the editor component instead of a plain textarea:

(defn home-page []
  (let [input (atom nil)
        output (atom nil)]
    (fn []
      [:div
       [editor input]
       [:div
        [:button
         {:on-click #(evaluate @input (fn [{:keys [value]}] (reset! output value)))}
         "run"]]
       [:p @output]])))

The editor looks a lot nicer now, but the output doesn't have any highlighting. Let's fix that by running it through highlight.js to generate nicely formatted results.

Once again, we'll need to add the additional CSS and Js files in the cljs-eval-example.handler namespace:

(defn head []
  [:head
   [:meta {:charset "utf-8"}]
   [:meta {:name "viewport"
           :content "width=device-width, initial-scale=1"}]
   (include-css
    (if (env :dev) "/css/site.css" "/css/site.min.css")
    "//cdnjs.cloudflare.com/ajax/libs/codemirror/5.8.0/codemirror.min.css"
    "//cdnjs.cloudflare.com/ajax/libs/highlight.js/8.9.1/styles/default.min.css"
    (if (env :dev) "css/site.css" "css/site.min.css"))
   (include-js
    "//cdnjs.cloudflare.com/ajax/libs/highlight.js/8.9.1/highlight.min.js"
    "//cdnjs.cloudflare.com/ajax/libs/codemirror/5.8.0/codemirror.min.js"
    "//cdnjs.cloudflare.com/ajax/libs/codemirror/5.8.0/mode/clojure/clojure.min.js")])

Back in the cljs-eval-example.core namespace we'll add a reference for [cljs.pprint :refer [pprint]] and write the result-view component that will take care of highlighting the output.

(ns cljs-eval-example.core
  (:require ...
            [cljs.pprint :refer [pprint]]))

...
            
(defn result-view [output]
  (reagent/create-class
   {:render (fn []
              [:pre>code.clj
               (with-out-str (pprint @output))])
    :component-did-update render-code}))

Highlight.js defaults to using <pre><code>...</pre></code> blocks, so we'll generate one in the :render function. Then we'll call the render-code function when the :component-did-update state is triggered. This function will simply pass the node to the .highlightBlock function provided by highlight.js:

(defn render-code [this]
  (->> this reagent/dom-node (.highlightBlock js/hljs)))

Finally, we'll have to update the home-page component to use the result-view component we just wrote:

(defn home-page []
  (let [input (atom nil)
        output (atom nil)]
    (fn []
      [:div
       [editor input]
       [:div
        [:button
         {:on-click #(evaluate @input (fn [{:keys [value]}] (reset! output value)))}
         "run"]]
       [:div
        [result-view output]]])))

Now both the editor and the output should look nicely highlighted, and the output will be formatted as a bonus. The entire code listing is as follows:

(ns cljs-eval-example.core
  (:require
   [reagent.dom :as dom]
   [reagent.core :as reagent :refer [atom]]
   [cljs.js :refer [empty-state eval-str js-eval]]
   [cljs.pprint :refer [pprint]]))

(defn evaluate [s cb]
  (eval-str
     (empty-state)
     s
     nil
     {:eval       js-eval    
      :source-map true
      :context    :expr}
     cb))

(defn editor-did-mount [input]
  (fn [this]
    (let [cm (.fromTextArea  js/CodeMirror
                             (dom/dom-node this)
                             #js {:mode "clojure"
                                  :lineNumbers true})]
      (.on cm "change" #(reset! input (.getValue %))))))

(defn editor [input]
  (reagent/create-class
   {:render (fn [] [:textarea
                    {:default-value ""
                     :auto-complete "off"}])
    :component-did-mount (editor-did-mount input)}))

(defn render-code [this]
  (->> this dom/dom-node (.highlightBlock js/hljs)))

(defn result-view [output]
  (reagent/create-class
   {:render (fn []
              [:pre>code.clj
               (with-out-str (pprint @output))])
    :component-did-update render-code}))

(defn home-page []
  (let [input (atom nil)
        output (atom nil)]
    (fn []
      [:div
       [editor input]
       [:div
        [:button
         {:on-click #(evaluate @input (fn [{:keys [value]}] (reset! output value)))}
         "run"]]
       [:div
        [result-view output]]])))

(defn mount-root []
  (dom/render [home-page] (.getElementById js/document "app")))

(defn init! []
  (mount-root))

A complete example project is available on GitHub.

Troubleshooting With JConsole

Thu, 08 Oct 2015 00:00:00 +0000

It's often useful to be able to tell how much resources your app happens to be using. I've previously discussed how JVisualVM can be used to do some basic profiling for the application. In this post we'll look at how to use another great tool called jconsole that also comes with the JVM. First, let's create and run a new Luminus web app as follows:

lein new luminus guestbook
cd guestbook
lein uberjar
java -jar target/guestbook.jar

We'll run the following command in a separate terminal:

jconsole

We should be greeted by a screen that looks something like the following:

We'll select guestbook and connect to it. Once the connection is established we'll see an overview screen detailing memory, class instances, threads, and CPU usage.

We can also select tabs to drill down into details about each of these as well as the VM summary. The Memory tab is the one of most interest to start. This screen will let us see a graph of memory usage over time and allow us to initiate garbage collection. It also shows the details about application memory usage and how it compares to the overall memory allocated by the JVM.

Let's run the Apache HTTP server benchmarking tool, that comes bundled by default on OS X, and see how that affects our application.

ab -c 10 -n 1000 http://127.0.0.1:3000/
This is ApacheBench, Version 2.3 <$Revision: 1663405 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        undertow
Server Hostname:        127.0.0.1
Server Port:            3000

Document Path:          /
Document Length:        3918 bytes

Concurrency Level:      10
Time taken for tests:   3.544 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      4251000 bytes
HTML transferred:       3918000 bytes
Requests per second:    282.14 [#/sec] (mean)
Time per request:       35.444 [ms] (mean)
Time per request:       3.544 [ms] (mean, across all concurrent requests)
Transfer rate:          1171.26 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       3
Processing:    15   35  27.4     26     252
Waiting:       15   35  26.3     26     226
Total:         15   35  27.5     26     252

Percentage of the requests served within a certain time (ms)
  50%     26
  66%     31
  75%     37
  80%     41
  90%     54
  95%     75
  98%    110
  99%    227
 100%    252 (longest request)

note that 282 req/sec number is running without any warmup while being instrumented

We can see that as the server is getting hammered by requests the memory usage spikes from roughly a 100 megs to about 275. However, once GC is performed the memory usage goes right back down.

This tells us that the application starts using more resources as it serves multiple concurrent requests, but then releases these as the GC runs indicating that no memory leaks are happening. We can also confirm that the threads and class instances are not getting out of hand as the application continues to run using the the respective tabs.

As you can see jconsole is a handy tool that can be used to quickly diagnose the behavior of a Clojure application. Should we find anything that warrants further investigation then it would be time to run a profiler such as jvisualvm to see where specifically the resources are being used.

Building services with Duct and compojure-api

Thu, 01 Oct 2015 00:00:00 +0000

In this post we'll look at writing a RESTful service using Duct and compojure-api. Our service will use a SQLite database and illustrate how to do operations such as adding, remove, and authenticating users.

Prerequisites

Creating the Project

Duct is a minimal web framework with emphasis on using component to manage stateful resources such as database connections. We can create a new Duct application by running the following command:

lein new duct swagger-service +example

This will generate a fresh application and add an example route component to it. Once the application is created we'll have to run the setup task to generate local assets in the root folder of the application:

cd swagger-service
lein setup

We can now test that our application works as follows:

lein run

If everything went well then we should be able to navigate to localhost:3000 and see Hello World displayed on the page. We're now ready to start working on creating our service.

Adding Dependencies

We'll start by adding some dependencies in project.clj that we'll need in order to create our service:

:dependencies
[...
 [crypto-password "0.1.3"]
 [metosin/compojure-api "0.23.1"]
 [org.xerial/sqlite-jdbc "3.8.11.1"]
 [yesql "0.5.0"]
 [migratus "0.8.4"]]

We'll use crypto-password to handle password hashing when we create user accounts and checking passwords during authentication. The compojure-api library will be used to generate the service endpoints. The sqlite-jdbc driver will be used as our data store, we'll access it using yesql, and we'll generate the database using migratus.

Configuring Migrations

Let's add the migratus plugin along with its configuration to our project:

:plugins [[lein-environ "1.0.1"]
          [lein-gen "0.2.2"]
          [migratus-lein "0.1.7"]]

:migratus {:store :database             
           :db {:classname "org.sqlite.JDBC"
                :connection-uri "jdbc:sqlite:service-store.db"}}

We can now run the following commands to generate the migration files for the users table:

mkdir resources/migrations
lein migratus create users

This will produce files for up and down migrations such as:

20151001145313-users.down.sql
20151001145313-users.up.sql

The up migrations file will create the table:

CREATE TABLE users
(id VARCHAR(20) PRIMARY KEY,
 first_name VARCHAR(30),
 last_name VARCHAR(30),
 email VARCHAR(30),
 admin BOOLEAN,
 last_login TIME,
 is_active BOOLEAN,
 pass VARCHAR(100));

Conversely, the down migrations file will delete it:

DROP TABLE users;

We can now run the following command to create the database:

lein migratus migrate

Adding Database Queries

With the database created, we'll need to add some queries to access the database. We'll create a new file called resources/sql/queries.sql and put the following SQL queries in it:

-- name: create-user!
-- creates a new user record
INSERT INTO users
(id, first_name, last_name, email, pass)
VALUES (:id, :first_name, :last_name, :email, :pass)

-- name: get-user
-- retrieve a user given the id.
SELECT * FROM users
WHERE id = :id

-- name: get-users
-- retrieve a user given the id.
SELECT id, first_name, last_name, email FROM users

-- name: delete-user!
-- delete a user given the id
DELETE FROM users
WHERE id = :id

Creating the Database Component

Now, let's create a component that will be used to access it. We'll create a new namespace called swagger-service.component.db then put the following code there:

(ns swagger-service.component.db
  (:require [yesql.core :refer [defqueries]]
            [com.stuartsierra.component :as component]
            [crypto.password.bcrypt :as password]
            [environ.core :refer [env]]))

(defqueries "sql/queries.sql")

(defn create-user-account! [user db]
  (create-user! (update user :pass password/encrypt) db))

(defn authenticate [user db]
  (boolean
   (when-let [db-user (-> user (get-user db) first)]
     (password/check (:pass user) (:pass db-user)))))

(defrecord DbComponent [connection]
  component/Lifecycle
  (start [component]
         (assoc component :connection connection))
  (stop [component]
        (dissoc component :connection)))

(defn db-component [connection]
  (->DbComponent connection))

The namespace will define query functions by calling the defqueries macro and giving it the path to the queries.sql file we just created.

Then we'll add a couple of helper functions to create a user account with a hashed password and to check whether the user and the password match the stored credentials.

Next, we define the DbComponent record that will manage the lifecycle of the database. The start function in the component will associate the given connection settings with the :connection key in the component, and the stop function will remove the connection.

The connection is specified in the swagger-service.config namespace and points to the connection-uri key that is expected to be found in the environment.

(def environ
  {:http {:port (some-> env :port Integer.)}
   :db   {:connection-uri (:connection-uri env)}})

We'll add the actual connection information under the :env key in profiles.clj:

;; Local profile overrides

{:profiles/dev  {:env {:connection-uri "jdbc:sqlite:service-store.db"}}
 :profiles/test {}}

Finally, we have a helper function to instantiate the component called db-component.

Adding a New Component to the System

With the component created we can now add it to the system found in the swagger-service.system namespace:

(ns swagger-service.system
 (:require ...
           [swagger-service.component.db :refer [db-component]]))

...
           
(defn new-system [config]
 (let [config (meta-merge base-config config)]
   (-> (component/system-map
        :db      (db-component (:db config))
        :app     (handler-component (:app config))
        :http    (jetty-server (:http config))
        :example (endpoint-component example-endpoint))
       (component/system-using
        {:http [:app]
         :app  [:example]
         :example []          
         :db []}))))

Creating an HTTP Endpoint Component

The final step is to add the service endpoint that will provide the RESTful interface to the database. We'll create a new namespace called swagger-service.endpoint.service. The namespace will use the compojure-api library to define the service operations. The library requires us to declare the types of request parameters and responses for each endpoint using the schema library.

Let's start by creating the namespace declaration with the following references:

(ns swagger-service.endpoint.service
 (:require [clojure.java.io :as io]
           [ring.util.http-response :refer :all]
           [compojure.api.sweet :refer :all]
           [schema.core :as s]
           [swagger-service.component.db :as db]))

Then we'll create the schema for the User type that matches the user table in our database:

(s/defschema User
 {:id String
  (s/optional-key :first_name) String
  (s/optional-key :last_name) String
  (s/optional-key :email) String
  (s/optional-key :pass) String})

Finally, let's create the service-endpoint component that will define the service routes. The component accepts the config as its parameter. The config will contain the :db key that we added to our system earlier with the database connection.

The routes are created by calling the api macro from compojure-api:

(defn service-endpoint [config]
 (api
   (ring.swagger.ui/swagger-ui
    "/swagger-ui")
   (swagger-docs
    {:info {:title "User API"}})
   (context* "/api" []
             :tags ["users"]

             (GET* "/users" []
                   :return  [User]
                   :summary "returns the list of users"
                   (ok (db/get-users {} (:db config))))
             
             (GET* "/user/:id"  []
                   :return      User
                   :path-params [id :- String]
                   :summary     "returns the user with a given id"
                   (ok (db/get-users {:id id} (:db config))))

             (POST* "/authenticate" []
                    :return         Boolean
                    :body-params    [user :- User]
                    :summary        "authenticates the user using the id and pass."
                    (ok (db/authenticate user (:db config))))
             
             (POST* "/user"      []
                    :return      Long
                    :body-params [user :- User]
                    :summary     "creates a new user record."
                    (ok (db/create-user-account! user (:db config))))
             
             (DELETE* "/user"    []
                    :return      Long
                    :body-params [id :- String]
                    :summary     "deletes the user record with the given id."
                    (ok (db/delete-user! {:id id} (:db config)))))))

Notice that we call ring.swagger.ui/swagger-ui and swagger-docs at the beginning of the definition of api. This will automatically produce the API documentation for the service operations defined within it. Once our service is hooked up, we'll be able to navigate to localhost:3000/swagger-ui and see an interactive page for testing the API endpoints.

As you may have noticed, compojure-api mimics Compojure route definitions with the difference that the route method name has a * after it. The route definition also has some additional keys associated with it.

the :return key specifies the return type for the service operation
the :summary key provides the documentation about the purpose of the operation
the parameters are specified using different keys depending on the parameter type, such as :path-params and :body-params.

Finally, each route will return a response type with the result of calling the handler associated with it.

If we look at the "/users" route we see that it calls the get-users function from the database and passes it the value of the :db key from the config. This will be used to resolve the database connection at runtime.

Adding the Endpoint to the System

With the route added we can now navigate back to the swagger-service.system namespace and add the component there:

(ns swagger-service.system
 (:require ...
           [swagger-service.component.db :refer [db-component]]
           [swagger-service.endpoint.service :refer [service-endpoint]]))
           
...

(defn new-system [config]
 (let [config (meta-merge base-config config)]
   (-> (component/system-map
        :db      (db-component (:db config))
        :app     (handler-component (:app config))
        :http    (jetty-server (:http config))
        :example (endpoint-component example-endpoint)
        :service (endpoint-component service-endpoint))
       (component/system-using
        {:http [:app]
         :app  [:example :service]
         :service [:db]
         :example []          
         :db []}))))

The service component is initialized using the endpoint-component Duct helper. Next, the component relationships have to be described explicitly. We can see that the :service component depends on the :db component, and the :app in turn depends on both the :example and the :service.

We can now restart our app and navigate to localhost:3000/swagger-ui to see the service test page. Using this page we can test all the service operations that we defined such as creating new users, authenticating, and listing users.

The full source for this tutorial is available on GitHub.

Conclusion

As you can see, compojure-api allows us to easily define RESTful services with type assertions, documentation, and a helpful test page. I've found this approach to be extremely effective when creating service APIs as it documents what each endpoint is doing and makes it easy to collaborate with consumers of the service. Meanwhile, Duct provides an excellent base for building services using the component pattern.

Luminus is Moving -> HTTP Kit -> Immutant

Sat, 11 Jul 2015 00:00:00 +0000

Update

After having some discussions with the author of HTTP Kit and doing a deeper evaluation of Immutant I'm switching to it as the default.

It turns out that Immutant addresses all of the concerns as well as HTTP Kit while having the benefit of a larger team maintaining it.

The version 2 of Immutant is modular and provides a minimal runtime that has low overhead. The websocket support works similarly to HTTP Kit and is now documented as well. Unlike Jetty 9 adapters with websocket support, Immutant builds do not require JRE 8 to run.

Finally, Immutant provides many useful pluggable libraries for caching, message queues, and scheduling.

End Update

One of the guiding principles for Luminus has been to provide a great user experience out of the box. Having to go through a tedious setup before you can focus on the problem you actually want to solve should be unnecessary.

Luminus removes the burden or having to find the libraries, configure middleware, and add the common boilerplate. The application generated by the template is ready for deployment out of the box. The only part that's missing is the domain logic for your application.

As the project evolves I'm always looking for new ways to streamline user experience. Clojure web ecosystem is rapidly evolving along with the best practices and tools. Luminus aims to keep abreast of these changes and to provide a reference implementation for Ring based applications.

Recently, Luminus moved to using Migratus for handling database migrations for reasons discussed in this post. This time we'll look at the reasons for moving to HTTP Kit as the default server.

Up to now, Luminus applications would use the version of Jetty packaged by the Ring dependency. The major drawback of the default Jetty adapter is its lack of support for websockets. After evaluating the alternatives I settled on HTTP Kit as the default server for Luminus.

HTTP Kit is built on top of NIO. It combines high performance when handling a large number of connections with low memory overhead per connection. Finally, it provides a Ring/Compojure compatible API for working with websockets that's now part of the official Luminus documentation.

While HTTP Kit is the default, all the major HTTP servers are supported via their respective flags:

+alpeh - Aleph is a stream based server built on top of Netty
+immutant - Immutant is a JBoss based server with many built in features such as messaging, scheduling and caching
+jetty - Jet is the Ring Jetty adapter with websocket support

Another major change is that the lein-ring plugin is no longer used by default. Since the plugin is based on the Jetty based ring-server library, a separate workflow was required for the alternative HTTP servers.

Instead, the template now provides its own core namespace that manages the server lifecycle. This provides a consistent experience regardless of the server being used. The lein-ring plugin is now part of the +war profile used to generate server independent WAR archives for deployment to application servers such as Apache Tomcat.

Data Focused

Tue, 07 Jul 2015 00:00:00 +0000

One interesting part I noticed about working with Clojure is that I practically never look for solutions to problems on Google or StackOverflow. I might google to see if a library exists for accomplishing a particular task, but I rarely end up having to google how to do a specific task.

This got me thinking about why that may be since I used to do that commonly back when I worked with Java. I think the key reason is that Clojure encourages writing code that operates on plain data.

Object Oriented Approach

Object oriented languages, such as Java, encourage encapsulating the data in domain specific abstractions. Rich Hickey discusses this in detail in his excellent Clojure, Made Simple talk. The OO approach ultimately leads to creation of frameworks that provide formalized ways to tie all the different domain abstractions together.

The problem with frameworks is that they impose a particular workflow on the user. However, in most cases there are many valid ways to solve a particular problem. The approach that the framework takes is just one way to do things and not necessarily the best way. If you happen to think about a problem differently from the authors of the framework then using it will not feel intuitive.

When you encounter a problem then you either have to spend the time to understand the internals of the framework and its design or simply memorize how different tasks are accomplished.

Understanding the internals of a complex piece of software is an arduous process that can take a long time and it's often time that you do not have. Conversely, having to understand the internals typically indicates that the abstraction you're working with is a leaky one.

This is where googling comes in. You know what your problem is, you know how you would solve it given the time, but you don't know how the authors of the framework expect you to solve it using their approach.

Since the choice of the solution completely arbitrary, there's no way for you to logically deduce what it is. In many cases the only realistic option is to hope that someone else ran into a similar problem already and see how they solved it within the context of the framework.

Data Centric Approach

Clojure takes the approach of keeping the data and logic separate. Instead of building local abstractions for each domain as we do with objects, all the functions operate on a common set of data structures. When a function is called its output can be used in a new context without any additional ceremony.

Since all Clojure libraries use the same core data structures, it's trivial to take the output of one library and pass it as input to another. Using the REPL we can quickly see how a particular library behaves and what output it generates.

This approach allows the user to find the libraries that fit the problem they're solving and then compose them in the way that makes sense in their particular scenario. The same core pattern of composing data transformations can be applied at different resolutions within a project.

At the lowest level we have functions as our building blocks. We combine these in different ways to transform the data on the small scale.

Once we have a number of related functions that represent a particular domain we combine them into a namespace, and then we pass the data between the namespaces to move data from one domain to another.

Libraries are simply collections of namespaces, and we use the same pattern when transform the data by combining them. A great example of this would be the ring-defaults library that chains a number libraries to achieve complex transformations of HTTP requests and responses.

Finally, at the top level we may have multiple projects passing data between each other in form of services. This approach is becoming increasingly popular in the industry as seen with the micro-services movement.

With Clojure, the focus is always on the data. When solving a problem, all we have to do is figure out how we need to transform the data and then find the appropriate building blocks for our problem.

Focusing on the data helps keep code simple and reusable without introducing unnecessary indirection into the process. I think this is the key reason why it's possible to work with Clojure without having to constantly memorize new patterns to solve different kinds problems.

Luminus is Migrating to Migratus

Mon, 29 Jun 2015 00:00:00 +0000

There was a recent discussion on google groups regarding migrations and handling of database credentials in Luminus. Up to now, Luminus would generate a template where the database credentials were hardcoded in the <app>.db.core namespace and migrations were handled by the ragtime.lein plugin.

This was not ideal for a couple of reasons. First, the hardcoded credentials aren't great for any serious applications. The credentials end up being checked in the code repository and have to be manually updated for each environment the application runs in. Second, you end up with separate sets of database configuration for the application and for the plugin. This is error prone as it puts the burden on the user to keep the credentials in sync.

The proposed approach was to use the profiles.clj instead to keep a single set of credentials for development. The production credentials would then be supplied using environment variables. This is a much cleaner approach to handling credentials as they're no longer part of the code and can be configured in a single place.

In the meantime, Ragtime had a new major version release 0.4.0 that introduces a number of changes. Ragtime is moving away from using a Leiningen plugin, and instead recommends running the commands from the REPL. The other major change is that it no longer allows multiple statements in a single migrations file.

The rationale here is that different SQL databases have different restrictions on the commands that can be sent in a single message. Therefore using a heuristic to split up migrations isn't guaranteed to work correctly across different database engines.

While this is true, in my view it also results in subpar user experience. While it's ok for trivial migrations, such as the ones seen in the examples, it doesn't scale well for larger ones. I think that there is a lot of value in being able to see the entirety of a migration in a single place without having to jump across multiple files.

update: Since the writing of the post, Ragtime has added the ability to use a custom separator, so it should be available in the next release.

At this point I decided to see what other migrations libraries were available and to evaluate if any of them would be a good fit for the workflow that Luminus aims to provide. The one I settled on was Migratus. It provides a workflow that's nearly identical to the original Ragtime based one that Luminus used.

Migrtus elegantly addresses the problem of splitting up statements by using a custom separator --;; to identify individual statements within the file. This removes the ambiguity of having to infer where one statement ends and another begins without forcing the user to split their migrations into multiple files.

Unfortunately, Migratus has not been maintained for the past two years and relied on a deprecated version of the clojure.java.jdbc library. Since Migratus already works well and it's a relatively simple library I decided to see if I could bring it up to date.

This turned out to be a relatively painless process and I ended up making some minor changes and improvements along the way. I contacted Paul Stadig, who is the author of the library, and he graciously agreed to transfer the ownership as he's no longer planning on developing it himself. I've released the updated library to Clojars and the latest version of Luminus uses Migratus to handle migrations.

As I mentioned earlier, using a Leiningen plugin to handle dev migrations requires dupliction of credentials. Instead, Luminus now provides an <app>.db.migrations namespace that manages migrations. This namespace is invoked from the <app>.core/-main when it's passed in migrate or rollback arguments. These arguments can be optionally followed by the migration ids in order to apply specific migrations. So, when previously you would run lein ragtime migrate, you'd now run lein run migrate to apply migrations.

Since this code is now part of the project it can now be run from the packaged uberjar as well. This allows the application to run its migrations on the server without necessitating a separate process for migrating the production database. Complete migrations documentation is available on the offical Luminus site.

Having a straight forward way to run migrations and store credentials securely, taking into account production environments, is an important aspect of providing a solid base for developing web applications.

Using Pulsar

Wed, 17 Jun 2015 00:00:00 +0000

In this post, we'll take a look at a basic usage example for Pulsar and see how to package it for production.

What is Pulsar?

Pulsar is the official Clojure API for the Quasar library that provides lightweight green threads and Erlang style actors for the JVM.

Quasar has a lot of similarities to the popular Akka framewok, but has the advantage of being a library as opposed to a framework that imposes its own workflow. For those interested, a detailed comparison of Quasar and Akka is availble here.

Using Pulsar is very straight forward, however there are a few caveats to be aware of when it comes to packaging it for production. Quasar requires bytecode instrumentation in order to provide suspendanble functions, and this means that the project.clj needs to have additional hints to facilitate it.

Creating the Project

Let's start by creating a new project called pulsar-example:

lein new pulsar-example

Next, we'll add the following dependencies to the project.clj file:

[co.paralleluniverse/pulsar "0.7.2"]
[co.paralleluniverse/quasar-core "0.7.2"]

We'll also have to add a :java-agents key that will invoke the Quasar agent responsible for the instrumentation:

:java-agents [[co.paralleluniverse/quasar-core "0.7.2"]]

Adding Actors

Let's open up the pulsar-example.core namespace and update the ns declaration as follows:

(ns pulsar-example.core
  (:require
   [co.paralleluniverse.pulsar
    [core :refer :all]
    [actors :refer :all]])
  (:refer-clojure :exclude [promise await])
  (:gen-class))

We'll implement one of the official examples where two actors send messages to one another. In the example we have two functions called ping and pong. These are defined using the defsfn macro as opposed to regular defn. This is necessary in order for these functions to be suspendable.

The ping function will accept two parameters consisting of the number representing remaining iterations and the actor to send messages to.

The function checks if there are remaining iterations and notfies pong that the conversation is complete when n is zero. Otherwise, it sends a ping message to the pong actor and waits to receive an acknowledgement before recurring. As you may have guessed, the receive call will block until a message is received.

The @self notation is used to access the actor itself. This is needed to pass it to the other actor as part of the message in order to receive a response.

(defsfn ping [n pong]
  (if (== n 0)
    (do
      (! pong :finished)
      (println "ping finished"))
    (do
      (! pong [:ping @self])
      (receive
        :pong (println "Ping received pong"))
      (recur (dec n) pong))))

Meanwhile, the pong function will wait to receive a message, if the message is :finished then it finishes its run, and if it matches [:ping ping] then it will return the message :ping to the caller and recur:

(defsfn pong []
  (receive
    :finished (println "Pong finished")
    [:ping ping] (do
                   (println "Pong received ping")
                   (! ping :pong)
                   (recur))))

Note that the message can either be a keyword or a vector containing the parameters we wish to pass to the actor. Finally, we'll add a -main function as the entry point to our program. Note that we join our actors to ensure that the application keeps running until the actors exit.

(defn -main []
  (let [a1 (spawn pong)
        b1 (spawn ping 3 a1)]
    (join a1)
    (join b1)
    :ok))

We can now test that everything is working by running it from the REPL or using lein run.

Packaging for Deployment

Once we're ready to package our app for deployment we need to make sure that the Quasar agent can be run to instrument our suspendable functions. To do that we'll have to add a :manifest key to our project that points to the following configuration:

:manifest
 {"Premain-Class" "co.paralleluniverse.fibers.instrument.JavaAgent"
  "Agent-Class" "co.paralleluniverse.fibers.instrument.JavaAgent"
  "Can-Retransform-Classes" "true"
  "Can-Redefine-Classes" "true"}

This will be written out to the META-INF/MANIFEST.MF file in the jar and provide the necessary information about the agent. The project can now be packaged by running lein uberjar. One final thing to be aware of is that the resulting jar must be run with the -javaagent flag as follows:

java -javaagent:target/pulsar-example.jar -jar target/pulsar-example.jar

This is all that needs to be done in order to package and run Pulsar projects using Leiningen. As always, the complete source for the example is available on GitHub.

ClojureScript REPL with Figwheel

Tue, 16 Jun 2015 00:00:00 +0000

update: Figwheel changed recently and the new process of starting the REPL is documented on the official Wiki.

Figwheel provides a fantastic developer experience and if you're not using it already I highly encourage you to give it a shot. I found that in most cases live code reloading is sufficient for my workflow, but there are occasions where I do want to have an actual REPL available.

This mostly comes up when I'm working with code that's not directly tied to rendering UI components and it can quickly devolve into println debugging.

You probably noticed that Figwheel starts a REPL in the terminal when it runs. However, this REPL is not terribly useful in practice. What would be better is to have a REPL that's connected to the editor, such as Cursive or Emacs, so that you can evaluate the code you're working on the same way you would with Clojure.

Luckily, getting this to work turns out to be a pretty simple affair. First thing we need to do is to make sure that the Figwheel config in project.clj has the :nrepl-port key set as seen below:

:figwheel
{:http-server-root "public"
 :server-port 3449
 :nrepl-port 7002 ;;start nREPL on port 7002
 :css-dirs ["resources/public/css"]
 :ring-handler yourapp/handler}

When you run lein figwheel the nREPL server will become available and you can connect your editor to it at localhost:7002, or whatever port you've specifcied. Once the nREPL is connected you'll have to run the following commands there:

user> (use 'figwheel-sidecar.repl-api)
user> (cljs-repl)

You should see the Figwheel REPL start up the same way it did when you ran lein figwheel in the terminal. You should now be able to send any code from the editor to the REPL for evaluation.

Websockets with HTTP Kit

Thu, 11 Jun 2015 00:00:00 +0000

In this post we'll look at working with websocks using Reagent and HTTP Kit. We'll see how to create a multi-user chat server that allows multiple clients to communicate with one another.

First thing to mention is that there are a couple of Clojure/Script libraries for working with websockets, such as Sente and Chord. However, what I'd like to illustrate is that using websockets directly from ClojureScript is extremely straight forward. Let's start by creating a new Luminus project that we'll use as the base for our example. We'll create the project using the +http-kit profile:

lein new luminus multi-client-ws +http-kit +cljs

Once the application is created we'll need to startup the server and Figwheel. To do that, we'll need to run the following commands in separate terminals.

lein run

lein figwheel

The Server

Let's create a new namespace called multi-client-ws.routes.websockets and add the following references there:


(ns multi-client-ws.routes.websockets
 (:require [compojure.core :refer [GET defroutes]]
           [org.httpkit.server
            :refer [send! with-channel on-close on-receive]]
           [cognitect.transit :as t]
           [taoensso.timbre :as timbre]))

Next, we'll create a Compojure route for our websocket handler:

(defroutes websocket-routes
 (GET "/ws" request (ws-handler request)))

Where the ws-handler function will look as follows:

(defn ws-handler [request]
 (with-channel request channel
               (connect! channel)
               (on-close channel (partial disconnect! channel))
               (on-receive channel #(notify-clients %))))

The function accepts the request and passes it to the org.httpkit.server/with-channel macro provided by the HTTP Kit API. The macro creates accepts the request as its argument and binds the value of the :async-channel key to the second paramets representing the name of the channel. The statement following the channel name will be called once when the channel is created. In our case we'll call the connect! function defined below any time a new client connects:

(defonce channels (atom #{}))

(defn connect! [channel]
 (timbre/info "channel open")
 (swap! channels conj channel))

The function will log that a new channel was opened and add the channel to the set of open channels defined above.

When the client disconnects the on-close function will be called. This function accepts the channel along with a handler. The handler should accept the channel and the disconnect status. Our handler will log that the channel disconnected and remove it from the set of open channels.

(defn disconnect! [channel status]
 (timbre/info "channel closed:" status)
 (swap! channels #(remove #{channel} %)))

Finally, we have the on-receive function that's called any time a client message is received. We'll pass it the notify-clients function as the handler. This function will broadcast the message to all the connected clients.

(defn notify-clients [msg]
 (doseq [channel @channels]
     (send! channel msg)))

That's all we need to do to manage the lifecycle of the websocket connections and to handle client communication.

Next, We'll need to add the routes in our multi-client-ws.handler namespace:

(def app
 (-> (routes
       websocket-routes
       (wrap-routes home-routes middleware/wrap-csrf)
       base-routes)
     middleware/wrap-base))

We will also have to update our multi-client-ws.middleware/wrap-base middleware wrapper to remove the wrap-formats middleware as it conflicts with handling websocket requests.

The Client

We'll start by creating a multi-client-ws.websockets in the src-cljs/multi_client_ws folder. The namespace will require the transit library:

(ns multi-client-ws.websockets
 (:require [cognitect.transit :as t]))

Next, we'll define an atom to hold our websocket channel and a couple of helpers for reading and writing the JSON encoded transit messages.

(defonce ws-chan (atom nil))
(def json-reader (t/reader :json))
(def json-writer (t/writer :json))

We'll now create a function to handle received messages. The function will accept the callback handler and return a function that decodes the transit message and passes it to the handler:

(defn receive-transit-msg!
 [update-fn]
 (fn [msg]
   (update-fn
     (->> msg .-data (t/read json-reader)))))

We'll also create a function that sends messages to the socket if it's open:

(defn send-transit-msg!
 [msg]
 (if @ws-chan
   (.send @ws-chan (t/write json-writer msg))
   (throw (js/Error. "Websocket is not available!"))))

Finally, we'll add a function to create a new websocket given the URL and the received message handler:

(defn make-websocket! [url receive-handler]
 (println "attempting to connect websocket")
 (if-let [chan (js/WebSocket. url)]
   (do
     (set! (.-onmessage chan) (receive-transit-msg! receive-handler))
     (reset! ws-chan chan)
     (println "Websocket connection established with: " url))
   (throw (js/Error. "Websocket connection failed!"))))

The UI

We'll now navigate to the multi-client-ws.core namespace and remove the code that's already there. We'll set the ns definition to the following:

(ns multi-client-ws.core
 (:require [reagent.core :as reagent :refer [atom]]
           [multi-client-ws.websockets :as ws]))

Next, we'll create an atom to keep a list of messages and a Reagent component that renders it:

(defonce messages (atom []))

(defn message-list []
 [:ul
  (for [[i message] (map-indexed vector @messages)]
    ^{:key i}
    [:li message])])

We'll now create a message-input component that will allow us to type in a message and send it to the server. This component creates a local atom to keep track of the message being typed in and sends the message to the server when the enter key is pressed.

(defn message-input []
 (let [value (atom nil)]
   (fn []
     [:input.form-control
      {:type :text
       :placeholder "type in a message and press enter"
       :value @value
       :on-change #(reset! value (-> % .-target .-value))
       :on-key-down
       #(when (= (.-keyCode %) 13)
          (ws/send-transit-msg!
           {:message @value})
          (reset! value nil))}])))

We can now create the home-page component that looks as follows:

(defn home-page []
 [:div.container
  [:div.row
   [:div.col-md-12
    [:h2 "Welcome to chat"]]]
  [:div.row
   [:div.col-sm-6
    [message-list]]]
  [:div.row
   [:div.col-sm-6
    [message-input]]]])

We'll also create an update-messages! function that will be used as the handler for the received messages. This function will append the new message and keep a buffer of 10 last received messages.

All that's left to do is mount the home-page component and create the websocket in the init! function:

(defn mount-components []
 (reagent/render-component [#'home-page] (.getElementById js/document "app")))

(defn init! []
 (ws/make-websocket! (str "ws://" (.-host js/location) "/ws") update-messages!)
 (mount-components))

We should now be able to open multiple browser windows and any messages typed in one window should show up in all the open windows.

Conclusion

As you can see, it's very easy to setup basic client-server communication between HTTP Kit and ClojureScript. While you may wish to use one of the libraries mentioned earlier for more sophisticated apps, it's certainly not necessary in many cases. The complete source for the example can be found on GitHub.

Web Development with Clojure 2

Tue, 14 Apr 2015 00:00:00 +0000

First, I'd like to thank all those who purchased the first edition of the book. I was overwhelmed by the response from the readers, and exhilirated to learn that it helped many start developing their web applications using Clojure. I'm excited to announce that I'm working on the second edition of Web Development with Clojure and that I'm expecting to publish it sometime this summer.

What to Expect

The main goal of the book is to provide a no-nonsense introduction to web development using Clojure. As such, I chose to cover tools and libraries that I consider to be beginner friendly.

Clojure web ecosystem has been steadily maturing since the release of the first edition. Last time I illustrated a number of approaches for putting applications together without recommending any one in particular over the others. This time around I'm primarily focusing on building application based on the Luminus stack. Luminus has recently seen a major update and continues to provide a solid foundation for building Clojure web applications according to best practices.

Clojure community favors using composable libraries over monolithic frameworks. This approach offers a number of advantages by giving the developer full control over the structure of the application the components used in it.

However, this approach works best for experienced developers who have a good understanding of the ecosystem and the available libraries. Having to develop this experience presents a significant barrier for newcomers. Luminus mitigates this issue by providing a batteries included template coupled with centralized documentation. This makes it a perfect foundation for a book aimed at beginners.

If you're familiar with the Clojure basics and you're looking to apply it for building web applications then this book is for you. The book aims to provide the reader with the understanding of the core ecosystem and the available tooling.

What's New

Those who are familiar with the first edition will find a number of changes and a number of new topics covered this time around.

At the time of writing of the first edition I felt that ClojureScript was not quite ready for general consumption. While some companies were already using it in production, the tooling around it was often challenging to use. As such, the book only gave it a very brief introduction and focused on traditional server side templating instead.

ClojureScript has matured greatly in the past year and the tooling is rapidly improving, while libraries such as Om and Reagent provide a clear benefit over using plain JavaScript. This time around ClojureScript is front and center with a primary focus on building single page apps using the Reagent library. I chose Reagent over Om for reasons I've discussed here. In short, I find that it's much easier to learn and apply effectively. The main project in the book that guides the reader through developing a multi-user picture gallery application is now developed as a single page application using Reagent.

Another major change is that I no longer recommend using lib-noir for developing new applications. While the library provides a number of helpers for achieving many common tasks found in typical web applications, it also introduces some problems inherent in its design. I've discussed some of these in my last post. Today, there are excellent standalone libraries available for each of the tasks that lib-noir was used for and I recommend using these instead.

The database chapter has been updated to introduce the excellent Yesql library and use the syntax of the latest version of clojure.java.jdbc.

I’m now covering the use of the compojure-api library along side Liberator. I’ve had an excellent experience using this library and I highly recommend trying it out if you haven’t already. The library uses Prismatic/schema to define the service API and allows automatic generation of interactive Swagger documentation such as seen here.

Finally, the book will provide more information on topics such as database migrations and deployment as part of addressing some of the feedback from the previous edition.

My hope is that the book will be useful to both new readers as well as those who purchased the first edition.

Announcing Luminus 2.0

Sat, 28 Feb 2015 00:00:00 +0000

I'm excited to announce the release of Luminus 2.0. This release is a major update and introduces a number of changes to the framework. These changes reflect the evolution of the Clojure web stack and best practices over the past year.

The biggest change is that Luminus is no longer using lib-noir, nor will I be actively developing the library going forward. I intend to continue maintaining it and to provide bug fixes, but I will not be working on additional features. If anybody is interested in taking it over then please leave a comment on GitHub in that regard.

I believe that lib-noir has served an important role providing many useful features such as input validation, access rules, session handling and so on. However, today there are great standalone libraries available for accomplishing each of these tasks. I believe that using simple and focused libraries leads to better overall user experience. The libraries that Luminus currently defaults to are as follows:

access rules - Buddy
cache - core.cache
crypto - Buddy
database - Yesql
sessions/cookies - Ring
validation - Bouncer

Session management is the biggest change from the user perspective. While lib-noir uses a request bound session that can be manipulated anywhere within the scope of the request, Ring requires sessions to be associated with the response explicitly by the handler.

While lib-noir approach is unquestionably convenient it introduces a subtle problem. Since the session is bound to a thread-local variable it's possible to run into race conditions with the in-memory session store. I feel that the Ring approach results in simpler design that’s more explicit about what it’s doing.

The new middleware stack is now based on ring-defaults instead of the app-handler from lib-noir as Noir specific middleware is no longer required.

The move from Korma to Yesql is another major change. While Korma provides a nice DLS for working with SQL, I feel that the sheer simplicity of Yesql is preferable in the long run.

Meanwhile, Buddy is an exciting set of libraries for handling authorization, authentication, and access rules. It has an intuitive API and excellent documentation.

Bouncer is an excellent validation library that works with both Clojure and ClojureScript allowing for shared validation logic between the server and the client.

Some other changes include the +cljs profile updates for Figwheel support and the deprecation of the +site profile. It's been replaced with the +auth profile that sets up Buddy middleware for session based authentication instead.

As always, the framework primarily strives to remove the boilerplate from typical web projects and provide reasonable defaults. If you don’t agree with any of the library choices that it makes it’s trivial to swap them out with your own. The base profile is intentionally kept minimal to provide an unopinionated default.

The template project for Luminus has been completely rewritten as well. The new template cleanly separates different profiles making it much easier to maintain and add new features.

Finally, all the documentation has been updated to reflect the changes with the original made available on GitHub.

A simple plugin system in Clojure

Thu, 15 Jan 2015 00:00:00 +0000

In this post we’ll see how to create a simplistic plugin system where plugins can be supplied as Leiningen dependencies and automatically initialized without any additional code change in the application.

Let’s take a look at Cryogen for a concrete example of how this can be useful. Cryogen started out using Markdown for content encoding, and we recently got a pull request that adds AsciiDoc support.

It’s always great to get additional features, but sometimes features also carry an undesirable cost. It turns out that AsciiDoc support relies on AsciidoctorJ, that in turn relies on JRuby and pulls in a huge amount of additional dependencies. This has a significant impact on the startup time of the application.

For anybody who isn’t using AsciiDoc the new feature simply degrades the user experience. So, ideally we’d like to keep AsciiDoc as a feature, but also avoid impacting users who aren’t using it. The ideal scenario is to be able to split out the parsers into standalone libraries and include the ones we need. This also has the benefit of people being able to write their own custom plugins that add the features they need without having to update the core project.

The approach I took here was to create an init function for each plugin that will take care of any initialization that the plugin needs to do and register it with the system.

All the available parsers are stored in an atom called markup-registry in cryogen-core, and each plugin simply updates the registry when it loads:

(defn init []
  (swap! markup-registry conj (markdown)))

The full code for the Markdown plugin can be seen here.

Next, we need to make our plugins discoverable so that they can be loaded when the application starts. This can be done using a configuration file that can be found on a classpath. Cryogen plugin configuration is stored in resources/plugin.edn using the following format:

{:description "Markdown parser"
 :init cryogen-markdown.core/init}

Using the above information we can load the appropriate namespace and run the initializer function for the plugin.

First, we need to grab all the resources with the name plugin.edn which can done as follows:

(defn load-plugins []
  (let [plugins (.getResources (ClassLoader/getSystemClassLoader) "plugin.edn")]
    (loop []
      (load-plugin (.. plugins nextElement openStream))
      (when (.hasMoreElements plugins)
        (recur)))))

Next, we read the configuration for each resource, require its namespace and then run the initializer functions as seen below:

(defn load-plugin [url]
  (let [{:keys [description init]} (edn/read-string (slurp url))]
    (println (green (str "loading module: " description)))
    (-> init str (s/split #"/") first symbol require)
    ((resolve init))))

With that in place we simply run load-plugins when the applicatin starts and any plugins found on the classpath will be initialized. All the user has to do is select what plugins they want to include in their dependencies to get the functionality they need.

The State of Reagent

Mon, 01 Dec 2014 00:00:00 +0000

I'm happy to report that Dan Holmsand has graciously agreed to move Reagent to a GitHub Organization called the reagent-project. The organization will help spread the work as well as provide a hub for all the related projects. There has been a lot of recent interest in the org and several projects have already migrated under the umbrella.

First, I'd like to mention the reagent-cookbook project that provides many recipes for accomplishing common tasks, such as Js library integration, using Reagent. The project provides clear and simple guidelines for contributing to help ensure that all recipes have a common format that's easy to follow. Next addition is the Historian project that provides drop-in undo/redo functionality for Reagent. World Singles has recently switched from Om to Reagent and Sean Corfield has added reagent-cursor library for Om-like cursor support in the process. Finally, there is my own reagent-forms library for creating data bindings using Reagent.

New Reagent projects can now be easily created and run using the reagent-template as follows:

lein new reagent my-app
cd my-app
lein cljsbuild auto &
lein ring server

The template will generate a Clojure/Clojurescript web app that's designed to get you up and running without any tedious setup. The resulting project is setup to use Figwheel for live code reloading and weasel for the REPL enabling smooth development out of the box.

The template uses sane library and configuration choices across the stack with Reagent for the UI, Secretary for client side routing, and the Ring/Compojure stack on the backend.

The dev server and packaging are handled by lein-ring that will take care of reloading changes during development and producing either an uberjar or an uberwar for running standalone or deploying to a container respectively. The project also contains a Procfile for instant Heroku deployment. For more details, please visit the project page here.

As you can see, much has happened with Reagent in the past month and the future is looking very bright. If you haven't tried Reagent yet, there's never been a better time than now. :)

Moving to Cryogen

Wed, 26 Nov 2014 00:00:00 +0000

The blog has officially been moved over to Cryogen. While, all the content has been migrated over, the links for the posts have changed and the original comments are no longer available since I'm now using Disqus.

Yuggoth was a fun experment and it held up to most traffic storms over the years, but at the end of the day it's hard to beat the simplicity of a static site.

Porting the content from my old blog turned out to be a simple affair. I used Postgres to store all the blog content in Yuggoth. The database contains tables for the posts, the comments, the tags, and the files. All I had to do was extract the data and write it back out using the Cryogen format. First, I extracted the binary data for the files as seen below.

(defn write-file [{:keys [data name]}]
   (with-open [w (clojure.java.io/output-stream
                   (str "resources/templates/files/" name))]
     (.write w data)))

(defn extract-files []
  (doseq [file (sql/query db ["select * from file"])]
    (write-file file)))

The posts table contains the content, while the tags are stored in a separate table. The tags can be aggregated by post using the handy array_agg function. This function will produce a Jdbc4Array as the result, and its contents can then be extracted to a vector.

(defn extract-tags [post]
  (update-in post [:tags] #(vec (.getArray %))))

(defn get-posts []
  (map extract-tags
       (sql/query db
         ["select array_agg(t.tag) as tags,
                  b.id, b.time, b.title, b.content from blog b, tag_map t
           where t.blogid = b.id
           group by b.id, b.time, b.title, b.content"])))

Now, all that's left to do is to generate the post metadata and the file name. Since each post contains a publication date and a title, these can be used to produce a filename in the format expected by Cryogen.

(defn format-post-date [date]
  (let [fmt (java.text.SimpleDateFormat. "dd-MM-yyyy")]
    (.format fmt date)))

(defn format-post-filename [time title]
  (str
    (->> (re-seq #"[a-zA-Z0-9]+" title)
         (clojure.string/join "-")
         (str "resources/templates/md/posts/" (format-post-date time) "-"))
   ".md"))

With that in place we can simply run through all the posts and extract them into appropriate files.

(defn write-post [{:keys [id time tags content title]}]
  (with-open [wrt (clojure.java.io/writer (format-post-filename time title))]
    (.write wrt
            (with-out-str
              (clojure.pprint/pprint
                {:title title
                 :layout :post
                 :tags (vec (.split tags " "))})))
    (.write wrt "\n")
    (.write wrt content)))

(defn extract-posts []
  (doseq [post (get-posts)]
    (write-post post)))

And that's all there is to it. Moral of the story is that we should always keep the data separate from its representation as you never know when you will need it down the road.

Cryogen: static site generation made easy

Thu, 13 Nov 2014 00:00:00 +0000

Cryogen is a new Clojure static site generator by Carmen La. Since there are already many popular site generators such as Jekyll, let's take a look at what makes Cryogen interesting.

In my opinion, the main feature of this project is its simplicity. Cryogen is shipped as a Leiningen template and all you have to do to create a site is run:

lein new cryogen my-blog

This will create an instance of the project with a template site initialized. The site can be run in dev mode using:

lein ring server

Once started, a helpful readme is shown on what to do next, which is a really nice touch.

The server will watch for changes in the resources/templates folder and recompile the site whenever updates are detected. The compiled site is served from resources/public. The static assets generated there can now be copied to over to be served by Nginx or Apache in production.

The layout is handled using Selmer templates, while the content of the posts and the pages is written using Markdown with the help of markdown-clj.

The generator handles all the common things like linking up pages, creating tags, syntax highlighting, sitemap, and RSS our of the box.

While most site generators take the approach of providing numerous configuration options for customization, Cryogen simply gives you the code to customize any way you see fit. You can simply go to the cryogen.compiler namespace and easily change its behaviour to fit whatever it is you're doing. The compiler code is very clean and easy to follow, making it easy to customize.

I definitely recommend taking a look at this project if you're looking to make a static site in the future.

Clojure Cup Results

Wed, 08 Oct 2014 00:00:00 +0000

Clojure Cup results are in and you can see the winning entries here. Kudos to all the teams who participated in the event!

It was exciting to see a wide variety of ideas explored during the event as well as the number of projects that were taken to completion. It's extremely impressive to see the kinds of Clojure apps that can be built from scratch in just a couple of days.

The scope of the competition clearly grew from last year, the prizes were bigger, the teams were larger, and the projects were more ambitious. One major change that I noticed was that a lot more projects were using ClojureScript and many of these used Om and Reagent to build the UI. It's great to see ClojureScript taking off and bringing some sanity to the world of front-end development.

Overall, I think it was an exciting event and I highly recommend browsing through the apps. Many of the projects are hosted on GitHub and it's worth exploring the code to learn some new tricks. :)

I can't wait to see what Clojure Cup 2015 will bring.

Parsing and rendering templates in Clojure & Haskell

Fri, 03 Oct 2014 00:00:00 +0000

A little while back Chris Allen discovered the joys of Haskell. We had a few discussions regarding the merits of static typing and whether Haskell would be a more productive language than Clojure.

Haskell was my first foray into functional programming, and it's the language that made me fall in love with it. However, I also found that non-trivial type relationships would often require a lot of mental gymnastics. Eventually, I ended up moving to Clojure as I realized that I was simply more productive in it.

Chris would have none of this, and repeatedly told me that I was simply lying about having tried Haskell, since nobody could possibly enjoy using a dynamic language after seeing the glory of HM type inference.

These sorts of debates are rather tiring, so instead I proposed that Chris would translate a small piece of Clojure code to Haskell. Then we could discuss the merits of each approach using a concrete example.

On December of 2013, I posted this gist and while it's not trivial, it weighs in at about 70 lines of code. It's a minimal example where I find Haskell starts making things unnecessarily difficult.

The code runs through a stream and looks for either {{...}} or {% if ... %} tags. It is intentionally kept short to limit noise, so it simply reads the tag names and prints them out. Since the code is a dumbed down version of Selmer, which Chris participated on writing, I assumed he would have little difficulty reading and translating it.

To my surprise he started making excuses that if he did translate it then it would give me ammunition to complain about Haskell being difficult. I think that's a rather strange argument to make for somebody making the case that Haskell makes things simpler.

Then he said he'd do it using the Parsec library, which obviously defeats the point. The question is not whether you can figure out a library API in Haskell, but how would you translate specific piece of Clojure code to Haskell.

The power of Clojure is that it allows me to solve hard problems instead of having to rely on somebody else to write libraries that do the hard stuff and then glue them together.

At this point Chris proceeded to ignore the request for over half a year, until he suddenly decided to start messaging me about it on Twitter last night. All of a sudden he couldn't wait a second more and what's more apparently my code was broken! As he so eloquently put it "@yogthos broken software in a unityped language...whodathunkit...". He proceeded to berate me and insisted that the code does not work, that I'm a liar for suggesting that it does, and he must make a blog post regarding this situation immediately.

I was surprised to hear this as I clearly recall the parser example working last I touched it. Since there was no reason to change it, I was fairly confident that it would be in working order. However, not having access to a computer at the time I could not check it myself.

Normally, if an individual contacts me respectfully I would be glad to help and work with them to figure out what the problem is. I make mistakes, we all do, and it's conceivable that there might've been a problem with my code. The approach Chris chosen was frankly insulting.

I explained to Chris that I was not at a computer and I can't look at the code, but evidently the blog post simply could not wait. You can read it in full here.

The problem turned out to be that Chris is apparently incapable of reading code. The parser gets initialized with the render-file function:

(defn render-file [filename args]
  (render (parse filename) args))

This function calls the parse function on the filename and then passes the parsed content to render. Chris tried to call render with:

(parser/render "Hello {{name}}!" {:name "Yogthos"})

Shockingly enough he got an error doing that, at this point he evidently was incapable figuring out on his own what was causing the error and proceeded to throw a tantrum on Twitter.

Of course, if the goal was to actually figure out what the problem was then one would at least look at what parse is doing:

(defn parse [file]
  (with-open [rdr (clojure.java.io/reader file)]

Then it would immediately become obvious that we must pass in something that can be read by clojure.java.io/reader, such as java.io.StringBufferInputStream, and then pass the result of parse to render. Naturally, when called correctly the code does exactly what it's supposed to:

(render
  (parse
    (java.io.StringBufferInputStream. "Hello {{name}}"))
    {:name "Yogthos"})
=>"Hello filter tag value: name"

Since Chris managed to run the render-file function as seen in one the snippets in his blog post, he doesn't seem to understand that I asked him to translate that code to Haskell. For whatever reason, he includes a screenshot of Selmer documentation, which is not the behavior of the parser and was never intended to be. The spec that Chris was asked to translate to Haskell is the code in the gist.

In his post, Chris went ahead and answered the question he would like to have been asked as opposed to the one he was actually asked. I suppose making a straw man is a lot easier than answering the original question.

What I learned from this experience is that some Haskell users like to talk a lot about their favorite language, but when asked to solve a simple problem they will do anything but that. I don't find that to be terribly convincing myself.

Yet Another Clojure Intro Guide

Sat, 27 Sep 2014 00:00:00 +0000

There are already lots of guides introducing Clojure syntax and many of its features, but these guides tend to focus on individual examples as opposed to the broad concepts behind the language.

In my experience, the difficulty in learning Clojure doesn't stem from the syntax, but from having to approach problems with a new mindset. The goal of this guide is to impart such a mindset on the reader. How badly I fail at this task remains to be seen. :P

Without further ado, here's the guide itself, and I hope you'll find it useful if you're starting out with Clojure.

Using ClojureScript REPL from Light Table

Sat, 20 Sep 2014 00:00:00 +0000

I recently discovered that Light Table REPL works with ClojureScript without the need for any specific setup in your project. All you need is the lein-cljsbuild plugin to run the incremental compiler and follow the steps below:

start the ClojureScript compiler using lein cljsbuild auto
start the server using lein ring server
open the Light Table browser connection to the server
navigate to a ClojureScript namespace in your project and start evaluating expressions

I tried other ClojureScript REPL setups before and I always found the process extremely fiddly, with Light Table everything just worked out of the box. I definitely recommend giving it a shot if you haven't yet, especially if you're working with ClojureScript.

Introducing reagent-forms

Sun, 31 Aug 2014 00:00:00 +0000

One thing I’ve always found to be particularly tedious is having to create data bindings for form elements. Reagent makes this much more palatable than most libraries I’ve used. All you need to do is create an atom and use it to track the state of the components.

However, creating the components and binding them to the state atom is still a manual affair. I decided to see if I could roll this into a library that would provide a simple abstraction for tracking the state of the fields in forms.

The usual way to bind inputs to atoms as illustrated in the official Reagent docs can be seen below:

(ns example
  (:require [reagent.core :as reagent :refer [atom]]))

(defn atom-input [value]
  [:input {:type "text"
           :value @value
           :on-change #(reset! value (-> % .-target .-value))}])

(defn shared-state []
  (let [val (atom "foo")]
    (fn []
      [:div
       [:p "The value is now: " @val]
       [:p "Change it here: " [atom-input val]]])))

We create an atom with some state and then pass it to our input component. The component will display the current value and update the state when its :on-change event is triggered.

Normally, we’d have to go through each field in the form and pass the state to it so that it can bind itself to the document we’re generating.

I wanted to be able to specify a template for a form and then pass the entire template to a function that would take care of binding all the fields to an atom representing the document.

This function would need to know what parts of the form need to be bound, how to bind each type of element, and how to uniquely identify it in the document.

My solution was to introduce the :field attribute that would identify the component as a form field, and to use the :id attribute as the unique key for the element.

The binding function would then walk the form and check for any component that contain the :field key in its attribute map. The key would point to the type of component such as text, numeric, list, and so on.

Then it could pass the component to a specific binding function that would be responsible for linking the field with the document and return a bound component. Let’s take a look at how this all works with an example.

We’ll first need to include the library in our project `[reagent-forms "0.1.3"] , then we’ll require the reagent-forms.core/bind-fields` function in our namespace:

(ns myform.core
  (:require [reagent-forms.core :refer [bind-fields]])

Next, we need to create a form template to represent our form:

(defn row [label input]
  [:div.row
    [:div.col-md-2 [:label label]]
    [:div.col-md-5 input]])

(def form-template
  [:div
   (row "first name" [:input {:field :text :id :person.first-name}])
   (row "last name" [:input {:field :text :id :person.last-name}])
   (row "age" [:input {:field :numeric :id :person.age}])
   (row "email" [:input {:field :email :id :person.email}])
   (row "comments" [:textarea {:field :textarea :id :comments}])])

Note that we call helper functions, such as row, eagerly. The bind-fields function will walk the template to construct the actual components that will be used by Reagent.

The . in the :id key indicates nested structure. When we have a key like :person.first-name, then its value will be stored under {:person {:first-name <field-value>}}.

Our form component will then create an atom to represent the document and bind it to the template to produce the actual form:

(defn form []
  (let [doc (atom {})]
    (fn []
      [:div
       [:div.page-header [:h1 "Reagent Form"]]
       [bind-fields form-template doc]
       [:label (str @doc)]])))

That’s all there is to it. Whenever the state of any of the components changes the doc atom will be updated and vice versa.

The bind-fields function also accepts optional events. Events are triggered whenever the document is updated, and will be executed in order they are listed. Each event sees the document modified by its predecessor. The event must take 3 parameters, which are the id, the value, and the document.

The id and the value represent the value that was changed to trigger the event, and the document is the atom that contains the state of the form. Note that the id is in form of a vector representing the path in the document. The event can either return an updated document or nil, when nil is returned then the state of the document is unmodified. The following is an example of an event to calculate the value of the :bmi key when the :weight and :height keys are populated:

(defn row [label input]
  [:div.row
    [:div.col-md-2 [:label label]]
    [:div.col-md-5 input]])

(def form-template
 [:div
   [:h3 "BMI Calculator"]
   (row "Height" [:input {:field :numeric :id :height}])
   (row "Weight" [:input {:field :numeric :id :weight}])
   (row "BMI" [:input {:field :numeric :id :bmi :disabled true}])])

[w/bind-fields
  form-template
  doc
  (fn [[id] value {:keys [weight height] :as doc}]
    (when (and (some #{id} [:height :weight]) weight height)
      (assoc-in doc [:bmi] (/ weight (* height height)))))]

The library provides support for a number of common fields such as inputs, checkboxes, radio buttons, lists, and multi-selects. However, it also makes it easy to add your own custom fields by implementing the reagent-forms.core/init-field multimethod.

The method must take two parameters, where the first parameter is the field component and the second is the options map. The options contain two keys called get and save!. The get key points to a function that accepts an id and returns the document value associated with it. The save! function accepts an id and a value that will be associated with it. Let’s take a look at the :radio field implementation as an example:

(defmethod init-field :radio
  [[type {:keys [field id value] :as attrs} & body]
   {:keys [get save!]}]
  (let [state (atom (= value (get id)))]
    (fn []
      (into
        [type
         (merge {:type :radio
                 :checked @state
                 :class "form-control"
                 :on-change
                 #(do
                    (save! id value)
                    (reset! state (= value (get id))))}
                attrs)]
         body))))

As you can see, the method simply returns a new component that’s bound to the supplied id in the document. For more details please see the documentation on the project page.

The library is still rather new and as such has some rough edges, such as poor error reporting. However, I already find it to be quite useful in my own projects.

File Snooping

Sun, 17 Aug 2014 00:00:00 +0000

I recently needed to watch files for changes and had a chance to play around with using the WatchService functionality in JDK 7. As is generally the case with Java, the API requires you to jump through a number of hoops to do anything, but turns out that it’s possible to wrap it up into something reasonable in the end.

The WatchService can be used to watch directories and provides different types of events, such as create, modify, delete, and overflow. The first three are self-explanatory and the last is a special event that’s triggered when an event might have been discarded or lost.

What we’d like to do is create a function called watch that accepts an input path along with event handlers for each of the above events.

To create a watcher we first need to get an instance of a Path. To do that we have to call (-> path (file) (.toURI) (Paths/get)). Next, we can get an instance of the WatchService by calling (.newWatchService (FileSystems/getDefault))

Now that we have a Path and a WatchService, we can register the service with the path to listen for the events we specify.

To handle this, I ended up creating a map with the keys representing the events and the values being the event handling functions.

{:create event-handler
 :modify event-handler}

When the event is triggered we will receive an instance of the WatchEvent. So, the handler functions should accept it as the parameter.

(defn event-handler [event]
  (println (.kind event) (.context event)))

Next, I created a couple of helpers to map the keywords to the appropriate events:

(defn register-events! [dir watch-service opts]
  (.register dir watch-service
    (-> opts
       (select-keys [StandardWatchEventKinds/ENTRY_CREATE
                     StandardWatchEventKinds/ENTRY_MODIFY
                     StandardWatchEventKinds/ENTRY_DELETE
                     StandardWatchEventKinds/OVERFLOW])
       (keys)
       (into-array))))

(defn rename-event-keys [opts]
  (rename-keys opts
    {:create StandardWatchEventKinds/ENTRY_CREATE
     :modify StandardWatchEventKinds/ENTRY_MODIFY
     :delete StandardWatchEventKinds/ENTRY_DELETE
     :overflow StandardWatchEventKinds/OVERFLOW}))

The transformed map is now ready for use. The WatchService implements closeable, so we can use the with-open macro to manage it:

(defn watch [path opts]
  (let [dir (-> path (file) (.toURI) (Paths/get))
        opts (rename-event-keys opts)]
    (with-open [watch-service
                (.newWatchService (FileSystems/getDefault))]
      (register-events! dir watch-service opts)
      (watch-loop watch-service opts))))

The watch function will register the events we passed in, open the watch service and then call the watch-loop function to do the actual watching.

(defn watch-loop [watch-service opts]
  (loop []
    (let [k (.take watch-service)]      
      (doseq [event (.pollEvents k)]
        (if-let [handler (get opts (.kind event))]
          (handler event)))
      (when (.reset k) (recur)))))

The watch-loop starts each iteration by calling take on the watch service. This method blocks until it receives an event, the service is closed or it’s interrupted.

Once we receive an event we can look it up in our options map and call the handler for the event. Finally, we need to call reset on the key before we start the next iteration.

Since the take function blocks, we probably want to run it in a thread:

(defn start-watcher! [path opts]
  (doto (Thread. #(watch path opts))
    (.setDaemon true)
    (.start)))

The above will start a background watcher thread and return it. The thread is daemonized, so that it doesn’t prevent the application from exiting. Example usage for the above can be to track when files are created or modified in the directory:

(start-watcher! “~/downlads”
  {:create #(println “file created” (-> % (.context) (.toString)))
   :modify #(println “file modified” (-> % (.context) (.toString)))})

That’s all there is to it and the full source for the example can be seen here.

Update

As one of the comments points out, JDK will poll on OS X and the default poll interval is quite large. In order to get around this we can force high sensitivity when we register the WatchService as follows:

(defn register-events! [dir watch-service]
  (.register dir
    watch-service
    (into-array
      [StandardWatchEventKinds/ENTRY_CREATE
       StandardWatchEventKinds/ENTRY_MODIFY
       StandardWatchEventKinds/ENTRY_DELETE
       StandardWatchEventKinds/OVERFLOW])
    (into-array
      [(com.sun.nio.file.SensitivityWatchEventModifier/HIGH)])))

Routing With Secretary

Thu, 14 Aug 2014 00:00:00 +0000

In the last post, we looked at using Reagent for building single page apps. The example app contained a single page with a form in it, which isn't terribly exciting as far as single page apps go.

In this post we’ll see see how to create an app with multiple pages and how we can route between them using the secretary library.

The app will be a guestbook with a page that shows a list of users and another page that allows new users to sign in. We’ll use the project from the last post as the base for this tutorial.

update the tutorial has been updated to the latest version of Luminus, you'll need to create a new project to follow along using lein new luminus guestbook +cljs

First thing that we have to do is to add the [secretary "1.2.3"] dependency to our project.clj. Next, let’s refactor our namespaces in src/cljs as follows:

src
  └ cljs
    └ guestbook
      └ core.cljs
      └ pages
        └ guest.cljs
        └ guest_list.cljs

The core namespace will act as the entry point for the client.
The session will house the global state of the application.
The guest namespace will house the sign-in form.
The guest-list namespace will display the guests.

Since we refactored the namespaces we’ll also need to update our app.html template to reflect that.

<script type="text/javascript">goog.require("guestbook.core");</script>

Session Management

In our example, the session will track the currently selected page and the saved documents.

We’ll use the reagent-utils session. The session is simply a Ragent atom with some helper functions around it.

Listing Guests

Let’s open up the guest-list namespace and add the following code there.

(ns guestbook.pages.guest-list
  (:require [reagent.session :as session]
            [clojure.string :as s]
            [reagent.core :as reagent :refer [atom]]
            [secretary.core :refer [dispatch!]]))

(defn guest-list-page []
  [:div
   [:div.page-header [:h2 "Guests"]]
   (for [{:keys [first-name last-name]}
         (session/get :guests)]
     [:div.row
      [:p first-name " " last-name]])
   [:button {:type "submit"
             :class "btn btn-default"
             :on-click #(dispatch! "/sign-in")}
    "sign in"]])

The namespace will contain a page that lists the guests that are currently in the session. The “sign in” button on the page uses the dispatch! function in order to route to the “/sign-in” page.

Adding Routes

The core namespace will specify the list of routes and provide an init! function to set the current page and render it when the application loads.

(ns guestbook.core
  (:require [reagent.core :as r]
            [reagent.session :as session]
            [secretary.core :as secretary :include-macros true]
            [goog.events :as events]
            [goog.history.EventType :as HistoryEventType]
            [guestbook.ajax :refer [load-interceptors!]]
            [guestbook.pages.guest-list
            :refer [guest-list-page]]
           [guestbook.pages.guest :refer [guest-page]])
  (:import goog.History))

(defn page []
  [(session/get :current-page)])

;; -------------------------
;; Routes
(secretary/set-config! :prefix "#")

(secretary/defroute "/" []
  (session/put! :current-page guest-list-page))

(secretary/defroute "/sign-in" []
  (session/put! :current-page guest-page))

;; -------------------------
;; History
;; must be called after routes have been defined
(defn hook-browser-navigation! []
  (doto (History.)
        (events/listen
          HistoryEventType/NAVIGATE
          (fn [event]
              (secretary/dispatch! (.-token event))))
        (.setEnabled true)))

;; -------------------------
;; Initialize app
(defn mount-components []
  (r/render [#'page] (.getElementById js/document "app")))

(defn init! []
  (load-interceptors!)
  (hook-browser-navigation!)
  (mount-components))

As we can see above, secretary uses Compojure inspired syntax that should look very familiar to anybody who's dabbled in Clojure web development.

In our case the routes will simply set the appropriate page in the session when called. The render function will then be triggered by the atom update and render the page for us.

Signing In

Finally, we’ll add the sign-in form in the guest namespace. The page will keep its local state in an atom and update the session using the callback handler in the save-doc function.

Note that we don’t have to do anything else to update the list of guests once the callback completes. Since the session atom has been updated, it will trigger the guest list to repaint with the new elements.

I found that this behavior largely obviates the need to use core.async since the Reagent atom can act as a sync point between the view and the model. It also makes it trivial to implements the React Flux pattern.

Views--->(actions) --> Dispatcher-->(callback)--> Stores---+
Ʌ                                                          |
|                                                          V
+--(event handlers update)--(Stores emit "change" events)--+

Our view components dispatch updates to the atoms, which represent the stores. The atoms in turn notify any components that dereference them when their state changes.

Using get/set! functions to access the atoms, as we’re doing in this example, allows us to easily listen for changes and hook in event handlers.

(ns guestbook.pages.guest
  (:refer-clojure :exclude [get])
  (:require [reagent.session :as session]
            [reagent.core :as reagent :refer [atom]]
            [secretary.core :refer [dispatch!]]
            [ajax.core :refer [POST]]))

(defn put! [doc id value]
  (swap! doc assoc :saved? false id value))

(defn get [doc id]
  (id @doc))

(defn row [label & body]
  [:div.row
   [:div.col-md-2 [:span label]]
   [:div.col-md-3 body]])

(defn text-input [doc id label]
  [row label
   [:input {:type "text"
            :class "form-control"
            :value (get doc id)
            :onChange #(put! doc id (-> % .-target .-value))}]])

(defn save-doc [doc]
  (POST "/save"
        {:params (dissoc @doc :saved?)
         :handler
         (fn [_]
           (put! doc :saved? true)
           (session/update-in! [:guests] conj @doc)
           (dispatch! "/"))}))

(defn guest-page []
  (let [doc (atom {})]
    (fn []
      [:div
       [:div.page-header [:h1 "Sign In"]]

       [text-input doc :first-name "First name"]
       [text-input doc :last-name "Last name"]

       (if (get doc :saved?)
         [:p "Saved"]
         [:button {:type "submit"
                   :class "btn btn-default"
                   :on-click #(save-doc doc)}
          "Submit"])
       [:button {:type "submit"
                 :class "btn btn-default"
                 :on-click #(dispatch! "/")} "back"]])))

The form code on this page is based on the previous tutorial and should hopefully be self explanatory at this point.

Hooking in Browser Navigation

As a final touch, we can add support for managing history using goog.events to enable more intelligent navigation using the browser.

(ns guestbook.core
 (:require [reagent.session :as session]
           [guestbook.pages.guest-list
            :refer [guest-list-page]]
           [guestbook.pages.guest :refer [guest-page]]
           [reagent.core :as reagent :refer [atom]]
           [secretary.core :as secretary
             :include-macros true :refer [defroute]]
           [goog.events :as events]
           [goog.history.EventType :as EventType]))

(defn hook-browser-navigation! []
  (doto (History.)
    (events/listen
      EventType/NAVIGATE
      (fn [event]
        (secretary/dispatch! (.-token event))))
    (.setEnabled true)))

The function is then run by the init! function when the app loads:

(defn init! []
  (load-interceptors!)
  (hook-browser-navigation!)
  (mount-components))

As usual, the source for the project can be found here.

Final Thoughts

The example in this post is intentionally kept trivial, but hopefully it illustrates a simple way to hook up multiple pages and navigate between them using Reagent and secretary.

I recently rewrote this blog engine to use Reagent and I found that it made the code much cleaner and easier to maintain. I think one of the main benefits of the single page approach is that it enforces a clear separation between the server and the client portions of the application.

If you’d like to see a complete application built using the approach discussed here, don’t hesitate to take a look at the code behind this blog.

Building Single Page Apps with Reagent

Tue, 15 Jul 2014 00:00:00 +0000

Background

I recently started working on a new project that has a significant UI component. I decided that this was a good opportunity to take a look at Angular and React for building the client as a single page application.

After a bit of evaluation, I decided that React was a better fit for the project. Specifically, I found the idea of the virtual DOM very appealing and its component based approach to be a good way to manage the application state.

Once I got a bit deeper into using React I found it lacking in many areas. For example, it doesn't provide an adequate solution for complex data binding and while there are a few libraries such as react-forms, I didn't find them to be a good fit for my needs.

Having heard lots of great things about Om, I decided that this might be a good time to revisit ClojureScript. While I've done some projects in ClojureScript previously, I always ended up going back to JavaScript in the end.

For me, the benefits were not enough to outweigh the maturity of JavaScript and the tooling available for it. One of the things I found to be particularly painful was debugging generated JavaScript. This problem has now been addressed by the addition of source maps.

Trying Om

As I went through Om tutorials, I found that it exposes a lot of the incidental details to the user. Having to pass nil arguments, reify protocols, and manually convert to Js using #js hints are a but a few warts that I ran into. Although, it's worth noting that the om-tools library from Prismatic address some of these issues.

Overall, I feel that Om requires a significant time investment in order to become productive. I found myself wanting a higher level of abstraction for creating UI components and tracking state between them. This led me to trying Reagent. This library provides a very intuitive model for assembling UI components and tracking their state, and you have to learn very few concepts to start using it efficiently.

Differences between Om and Reagent

Om and Reagent make different design decisions that result in different tradeoffs, each with its own strength and weaknesses. Which of these libraries is better primarily depends on the problem you're solving.

The biggest difference between Om and Reagent is that Om is highly prescriptive in regards to state management in order to ensure that components are reusable. It's an anti-pattern for Om components to manipulate the global state directly or by calling functions to do so. Instead, components are expected to communicate using core.async channels. This is done to ensure high modularity of the components. Reagent leaves this part of the design up to you and allows using a combination of global and local states as you see fit.

Om takes a data centric view of the world by being agnostic about how the data is rendered. It treats the React DOM and Om components as implementation details. This decision often results in code that's verbose and exposes incidental details to the user. These can obviously be abstracted, but Om does not aim to provide such an abstraction and you'd have to write your own helpers as seen with Prismatic and om-tools.

On the other hand, Reagent provides a standard way to define UI components using Hiccup style syntax for DOM representation. Each UI component is a data structure that represents a particular DOM element. By taking a DOM centric view of the UI, Reagent makes writing composable UI components simple and intuitive. The resulting code is extremely succinct and highly readable. It's worth noting that nothing in the design prevents you from swapping in custom components. The only constraint is that the component must return something that is renderable.

Using Reagent

The rest of this post will walk through building a trivial Reagent app where I hope to illustrate what makes Reagent such an excellent library. Different variations of CRUD apps are probably the most common types of web applications nowadays. Let's take a look at creating a simple form with some fields that we'll want to collect and send to the server.

I won't go into details of setting up a ClojureScript project in this post, but you can use the reagent-example project to follow along. The project requires Leiningen build tool and you will need to have it installed before continuing.

Once you check out the project, you will need to start the ClojureScript compiler by running lein cljsbuild auto and run the server using lein ring server.

The app consists of UI components that are tied to a model. Whenever the user changes a value of a component, the change is reflected in our model. When the user clicks the submit button then the current state is sent to the server.

The ClojureScript code is found in the main.core under the src-cljs source directory. Let's delete its contents and start writing our application from scratch. As the first step, we'll need to reference reagent in our namespace definition.

(ns main.core
 (:require [reagent.core :as reagent :refer [atom]]))

Next, let's create a Reagent component to represent the container for our page.

(defn home []
  [:div
    [:div.page-header [:h1 "Reagent Form"]]])

We can now render this component on the page by calling the render-component function.

(reagent/render-component [home]
  (.getElementById js/document "app"))

As I mentioned above, the components can be nested inside one another. To add a text field to our form we'll write a function to represent it and add it to our home component.

(defn text-input [label]
  [:div.row
    [:div.col-md-2
      [:span label]]
    [:div.col-md-3
      [:input {:type "text" :class "form-control"}]]])

(defn home []
  [:div
    [:div.page-header [:h1 "Reagent Form"]]
    [text-input "First name"]])

Notice that even though text-input is a function we're not calling it, but instead we're putting it in a vector. The reason for this is that we're specifying the component hierarchy. The components will be run by Reagent when they need to be rendered.

We can also easily extract the row into a separate component. Once again, we won't need to call the row function directly, but can treat the component as data and leave it up to Reagent when it should be evaluated.

(defn row [label & body]
  [:div.row
   [:div.col-md-2 [:span label]]
   [:div.col-md-3 body]])

(defn text-input [label]
  [row label [:input {:type "text" :class "form-control"}]])

We now have an input field that we can display. Next, we need to create a model and bind our component to it. Reagent allows us to do this using its atom abstraction over the React state. The Reagent atoms behave just like standard Clojure atoms. The main difference is that a change in the value of the atom causes any components that dereference it to be repainted.

Any time we wish to create a local or global state we create an atom to hold it. This allows for a simple model where we can create variables for the state and observe them as they change over time. Let's add an atom to hold the state for our application and a couple of handler functions for accessing and updating it.

(def state (atom {:doc {} :saved? false}))

(defn set-value! [id value]
  (swap! state assoc :saved? false)
  (swap! state assoc-in [:doc id] value))

(defn get-value [id]
  (get-in @state [:doc id]))

We can now update our text-input component to set the state when the onChange event is called and display the current state as its value.

(defn text-input [id label]
  [row label
   [:input
     {:type "text"
       :class "form-control"
       :value (get-value id)
       :on-change #(set-value! id (-> % .-target .-value))}]])

(defn home []
  [:div
    [:div.page-header [:h1 "Reagent Form"]]
    [text-input :first-name "First name"]])

Let's add a save button to our form so that we can persist the state. For now, we'll simply log the current state to the console.

(defn home []
  [:div
    [:div.page-header [:h1 "Reagent Form"]]
    [text-input :first-name "First name"]    
    [:button {:type "submit"
              :class "btn btn-default"
              :on-click #(.log js/console (clj->js @state))}
     "Submit"]])

If we open the console, then we should see the current value of the :first-name key populated in our document whenever we click submit. We can now easily add a second component for the last name and see that it gets bound to our model in exactly the same way.

(defn home []
  [:div
    [:div.page-header [:h1 "Reagent Form"]]

    [text-input :first-name "First name"]
    [text-input :last-name "First name"]

    [:button {:type "submit"
              :class "btn btn-default"
              :onClick #(.log js/console (clj->js @state))}
     "Submit"]])

So far we've been using a global variable to hold all our state, while it's convenient for small applications this approach doesn't scale well. Fortunately, Reagent allows us to have localized states in our components. Let's take a look at implementing a multi-select component to see how this works.

When the user clicks on an item in the list, we'd like to mark it as selected. Obviously, this is something that's only relevant to the list component and shouldn't be tracked globally. All we have to do to create a local state is to initialize it in a closure.

We'll implement the multi-select by creating a component to represent the list and another to represent each selection item. The list component will accept an id and a label followed by the selection items.

Each item will be represented by a vector containing the id and the value of the item, eg: [:beer "Beer"]. The value of the list will be represented by a collection of the ids of the currently selected items.

We will use a let binding to initialize an atom with a map keyed on the item ids to represent the state of each item.

(defn selection-list [id label & items]
  (let [selections (->> items (map (fn [[k]] [k false])) (into {}) atom)]    
    (fn []
      [:div.row
       [:div.col-md-2 [:span label]]
       [:div.col-md-5
        [:div.row
         (for [[k v] items]
          [list-item id k v selections])]]])))

The item component will be responsible for updating its state when clicked and persisting the new value of the list in the document.

(defn list-item [id k v selections]
  (letfn [(handle-click! []
            (swap! selections update-in [k] not)
            (set-value! id (->> @selections
                                (filter second)
                                (map first))))]
    [:li {:class (str "list-group-item"
                      (if (k @selections) " active"))
          :on-click handle-click!}
      v]))

Let's add an instance of the selection-list component to our form and see how it looks.

(defn home []
  [:div
    [:div.page-header [:h1 "Reagent Form"]]

    [text-input :first-name "First name"]
    [text-input :last-name "First name"]

    [selection-list :favorite-drinks "Favorite drinks"
     [:coffee "Coffee"]
     [:beer "Beer"]
     [:crab-juice "Crab juice"]]

    [:button {:type "submit"
              :class "btn btn-default"
              :onClick #(.log js/console (clj->js @state))}
     "Submit"]])

Finally, let's update our submit button to actually send the data to the server. We'll use the cljs-ajax library to handle our Ajax calls. Let's add the following dependency [cljs-ajax "0.2.6"] to our project.clj and update our namespace to reference it.

(ns main.core
 (:require [reagent.core :as reagent :refer [atom]]
           [ajax.core :refer [POST]]))

With that in place we can write a save-doc function that will send the current state of the document to the server and set the state to saved on success.

(defn save-doc []
  (POST (str js/context "/save")
        {:params (:doc @state)
         :handler (fn [_] (swap! state assoc :saved? true))}))

We can now update our form to either display a message indicating that the document has been saved or the submit button based on the value of the :saved? key in our state atom.

(defn home []
  [:div
    [:div.page-header [:h1 "Reagent Form"]]

    [text-input :first-name "First name"]
    [text-input :last-name "Last name"]
    [selection-list :favorite-drinks "Favorite drinks"
     [:coffee "Coffee"]
     [:beer "Beer"]
     [:crab-juice "Crab juice"]]

   (if (:saved? @state)
     [:p "Saved"]
     [:button {:type "submit"
              :class "btn btn-default"
              :onClick save-doc}
     "Submit"])])

On the server side we'll simply log the value submitted by the client and return "ok".

(ns reagent-example.routes.services
  (:use compojure.core)
  (:require [reagent-example.layout :as layout]
            [noir.response :refer [edn]]
            [clojure.pprint :refer [pprint]]))

(defn save-document [doc]
  (pprint doc)
  {:status "ok"})

(defroutes service-routes
  (POST "/save" {:keys [body-params]}
        (edn (save-document body-params))))

With the route hooked up in our handler we should see something like the following whenever we submit a message from our client:

{:first-name "Jasper", :last-name "Beardly", :favorite-drinks (:coffee :beer)}

As you can see, getting started with Reagent is extremely easy and it requires very little code to create a working application. You could say that single page Reagent apps actually fit on a single page. :) In the next installment we'll take a look at using the secretary library to add client side routing to the application.

No Free Lunch

Sun, 26 Jan 2014 00:00:00 +0000

On paper static typing sounds strictly superior to dynamic typing. The compiler can track the data flow throughout our code and tell us when we're using the data incorrectly. This clearly eliminates a whole class of errors that are otherwise possible. What's more, types allow us to encode the business logic of our application allowing us to write code that's provably correct.

All this is unarguably correct, however these benefits do not come for free. One major problem is that static typing requires us to declare all the relationships globally. When you have global relationships a change at any level requires global refactoring at every level.

In my opinion, this makes static typing poorly suited for situations where your requirements are incomplete or subject to change, and in reality there are very few scenarios where your requirements are set in stone.

In most cases, we only care about a local contract between any two functions. What we want to know is that the function being called produces the type expected by the caller. A modification of a local contract should not cause global change in our code base.

Another cost of static typing is that it forces us to handle cases that are not part of the application workflow. For example, our code might have an undefined behavior, but the interface does not allow the user to access this state. This is a case of a tree falling in the woods when no one's around.

Regardless of the type system that you use, you will need to do functional testing in order to ensure that the business logic does what's intended. At the end of the day, the goal of the application is to handle the intended use cases as opposed to providing a proof of correctness.

When I look at the GitHub issues for my own projects such as Selemer or markdown-clj, vast majority of them stem from lack of specification. Practically none of these issues would have been caught by the type system. Had I used a statically typed language to write these projects, I would've simply had to jump through more hoops to end up with the same result.

In my opinion the value added by a static type system has to be weighed against the complexity of the problem and the cost of errors. Since it's obviously useful in some cases and provides dubious value in others, an optional type system might provide the right balance. Static typing is a tool and it should be up to the developer to decide how and when to apply it.

With an optional static checker we can add types where it makes sense and leave the rest of the code dynamic. This is precisely the situation CircleCI found themselves in.

Clojure High Performance Programming Review

Mon, 09 Dec 2013 00:00:00 +0000

First thing I'd like to say is that I'm very excited by the shift in nature of Clojure books that are coming out. There are already many excellent books about the language itself. Some of these include The Joy of Clojure, Programming Clojure, and Clojure in Action.

This year we can add Clojure Data Analysis Cookbook, Clojure Cookbook, my own Web Development With Clojure, and Clojure High Performance Programming to the roster. All these books focus on applying the language in the real world.

To me this indicates that developers are moving from simply experimenting with the language to actually using it professionally. The results from the 2013 State of Clojure & ClojureScript, where more than half the respondents reported using Clojure at work, appear to corroborate this idea.

One of the things that makes Clojure attractive is the fact that it's one of the more performant dynamic languages. As can be seen in the recent round of TechEmpower Benchmarks, Clojure web frameworks fare quite well compared to the popular offerings in Ruby and Python. Since performance is a major factor in using Clojure, a book discussing high performance programming is a welcome addition.

The book starts off by introducing the reader to performance use case classification. It does a decent job of explaining the background concepts and the vocabulary that will be used throughout.

Different types of scenarios are discussed and their related performance concerns. For example, when we deal with user interfaces, responsiveness is our main concern. On the other hand if we're doing data-processing then we want to optimize CPU and memory usage.

The author then moves on to discuss how common Clojure idioms impact the performance of the application. Understanding what goes on behind the scenes helps reason about potential pitfalls down the road.

There's a good discussion about the explicit use of loop/recur over higher order functions that illustrates a way to squeeze out additional performance. In another section the author goes on to explain the impact of laziness on performance in functional languages.

There are also tips regarding the use of different data formats. One example compares the benefits of EDN over JSON. EDN can save memory by using interned symbols and keywords, while JSON uses string keys which will not be interned. The author explains that in addition to saving memory, interning also avoids heap usage and this helps minimize garbage collection. This is something you would definitely want to consider if you were working with a high performance application.

The techniques used by some of the popular libraries, such as Nippy, are examined to see how they achieve high performance. These kinds of real world examples are very helpful. Not only do we learn about the theory, but we also get to see how it's applied in practice.

In general, the book covers a wide range of topics, but only offers a superficial overview for many. The reader will most certainly need to do further research in order to apply many of the concepts discussed.

If you're looking for a refresher or a primer on the topics discussed, then it's not a bad place to start. However, if you're looking for a comprehensive discussion on doing high performance programming with Clojure, you'll likely be left wanting.

Making Selmer User Friendly

Thu, 21 Nov 2013 00:00:00 +0000

It's been nearly 5 month since Selmer was released. In that time many bugs have been squashed and lots of new features added. However, there is one aspect that remained shameful and that was error reporting.

When Selmer failed to parse a template it would often produce error messages that were less than useful. For example, given the following template:

<html>
  <body>
    {% blok %}
    {% endblock %}
    <h2>Hello {{name}}</h2>
  </body>
</html>

we'd end up with the following error after trying to render it:

Exception in thread "main" java.lang.Exception: unrecognized tag: :blok - did you forget to close a tag?

While the error indicated the name of the problem tag, it didn't say what template this tag originated from or on what line it appeared.

These types of errors can result in a lot of wasted time and frustration. It would be much better to provide a clear error that contains the actual offending tag along with the name of the template and the line number.

As of version 0.4.8, Selmer has a validator that checks the following cases:

can the tag be parsed successfully
is the filter found in the map of filters
does the tag contain a name
is the tag name found in the map of tags
if a tag is a block tag, is the corresponding closing tag found
is the tag a closing tag for an opening tag that's not present

Here's the error returned by the validator when rendering the above template:

Exception in thread "main" java.lang.Exception: Unrecognized tag: {% blok %} on line 3 for template file:/Users/Yogthos/selmer-test/resources/index.html

This gives us a lot more information as to what went wrong and where. This is a big improvement on the original error, however we still have an ugly stacktrace to look at to figure out what happened.

It would be even better to return a distinct validation error that could be intercepted by some middleware to produce a friendly error page.

This is precisely what Selmer does as of version 0.5.3. The validator will now return ex-info with a key :type that's set to :selmer-validation-error.

It will also contain an error page template that can be rendered using the ex-data attached to the exception. We can now write a simple middleware function to catch these errors and render the error page:

(defn template-error-page [handler]
  (fn [request]
    (try
      (handler request)
      (catch clojure.lang.ExceptionInfo ex
        (let [{:keys [type error-template] :as data} (ex-data ex)]
          (if (= :selmer-validation-error type)
            {:status 500
             :body   (selmer.parser/render error-template data)}
            (throw ex)))))))

Using the above middleware, we'll see the following page whenever the parser fails to compile a template:

We can now immediately tell that an error occurred during the template compilation and see only the information pertaining to the nature of the error.

Of course, we wouldn't want to display this information when running in production. A simple solution would be to set a dev flag and check for it in our middleware.

This is precisely what the latest Luminus template will do using the environ library. The project.clj now contains an :env key under the :dev profile with the :selmer-dev flag set to true:

:dev {:dependencies [[ring-mock "0.1.5"]
                     [ring/ring-devel "1.2.1"]]
      :env {:selmer-dev true}}}

The middleware will check that the key is present and only render the error page in development mode:

(defn template-error-page [handler]
  (if (env :selmer-dev)
    (fn [request]
      (try
        (handler request)
        (catch clojure.lang.ExceptionInfo ex
          (let [{:keys [type error-template] :as data} (ex-data ex)]
            (if (= :selmer-validation-error type)
              {:status 500
               :body (parser/render error-template data)}
              (throw ex))))))
    handler))

When it comes to writing libraries it's easy to forget about the little things like error reporting and documentation. However, these things are just as important as having good code and a clean API.

In the end, this is what makes the difference between a pleasant development experience and one that's fraught with frustration.

Web Development With Clojure Beta

Wed, 28 Aug 2013 00:00:00 +0000

I'm happy to announce that Web Development With Clojure has finally reached beta and it's now available for purchase.

The book covers working with the core Ring/Compojure stack, common libraries as well as techniques for accomplishing common tasks. My main goal is to make the reader comfortable with the Clojure ecosystem and demonstrate how to take an application from inception all the way to deployment.

It's been a long journey to get to this point and I learned much along the way. This book is my way to share that experience with you and hopefully save you time when working on your projects.

There's still some clean up left to do and you might see a few typos here and there. However, there won't be any changes to the core content at this point and I hope that you'll be curious enough to take a look. :)

Why I'm Productive in Clojure

Sun, 18 Aug 2013 00:00:00 +0000

I find that I often get excited about learning a new language, but after I use it for a while it will inevitably lose its lustre. Eventually it becomes just another language in my tool box.

One exception to this rule is Clojure. I still enjoy using the language as much as I did when I first learned it. The reason for this is that it strikes the right balance between power and simplicity.

The Balance of Power

Some languages are simple but they're also verbose. You've probably heard people say that verbosity really doesn't matter. These people will go to great length to point out that all languages are Turing complete and that in certain languages you simply have to write a bit more code.

I think that's missing the point however. The question is not whether something can be expressed in principle. It's how well the language maps to the problem being solved. One language will let you think in terms of your problem domain, while another will force you to translate the problem to its constructs.

The latter is often tedious and rarely enjoyable. You end up writing a lot of boilerplate code and constantly repeating yourself. I hope you'll agree that there is a certain amount of irony involved in having to write repetitive code.

Other languages aren't verbose and they provide many different tools for solving problems. Unfortunately, working in such languages is often akin to trying to decide on a particular set of screwdrivers at a hardware megastore.

You end up comparing this brand against that, checking the number of bits that comes with each set, seeing which one's on sale today, and soon you forget why you wanted a screwdriver in the first place.

The more features there are the more things you have to keep in your head to work with the language effectively. With many languages I find myself constantly expending mental overhead thinking about all the different features and how they interact with one another.

What matters to me in a language is whether I can use it without thinking about it. When the language is lacking in expressiveness I'm acutely aware that I'm writing code that I shouldn't be. On the other hand when the language has too many features I often feel overwhelmed or I get distracted playing with them.

To make an analogy with math, it's nicer to have a general formula that you can derive others from than having to memorize a whole bunch of formulas for specific problems.

This is where Clojure comes in. With it I can always easily derive a solution to a particular problem from a small set of general patterns. The number of things I have to keep in my head is not overbearing.

All you need to become productive is to learn a few simple concepts and a bit of syntax. However, the number of ways that these concepts can be combined to solve all manner of problems appears to be inexhaustible. I've been writing Clojure for years and I discover new ways to combine the things I already know every single day.

Macros are a good example of this. The fact that you can transform the language using itself allows tackling a wide range of problems that would otherwise require a range of specific tools and language features.

Interactive Development

When I'm solving a problem in Clojure I inevitably want to write an elegant solution that expresses the gist of it cleanly and clearly. This is largely due to the fact that the development process is interactive.

When I work with the REPL I can fumble around looking for a solution and make sense of it through experimentation. Once I've internalized the problem I can quickly write a clean solution using my newly gained understanding.

The REPL also helps keep me engaged in trying to find the solution. Being able to try things and get immediate feedback is enjoyable. Even when your code doesn't do what you want you can see the progression and that is often enough of a motivator to keep going.

Another important feature of the REPL is that it encourages refactoring. I'm much more likely to refactor code when I can easily test it without disrupting my workflow.

Finishing Things

Interactivity alone isn't enough however. All the feedback in the world wouldn't make one bit of difference if you couldn't actually solve your problem in a reasonable amount of time.

I find that I have a sharp falloff curve when it comes to staying engaged in a project. You've probably noticed this phenomenon yourself. When you start a project you're excited and you enjoy seeing it take shape.

However, after working on a project for some amount of time the excitement wanes. Eventually, you might even dread having to touch the code again. I find that it's critical to get the core functionality working before I hit this stage.

Once the project solves a particular problem that I have I'll start using it. At this point I get to reap the benefits of having spent the effort on it. This also lets me identify the functionality that I'm missing through usage. There is a lot more incentive to add features to a project that you're actually using.

Most recently I found this to be the case working on Selmer. I was able to implement the base parser in just a couple of days, while cesarbp implemented the logic for the filters.

It took a couple of more days to get the template inheritance logic working. All of a sudden we had a usable library in under a week of effort. I'm already actively using it for actual work and new features are added piecemeal as the need comes up.

Here's the GitHub activity graph for Selmer:

As you can see there's a big initial spike with a very sharp falloff. A similar patten can be seen in my other projects such as clj-pdf and with Luminus:

As the project matures bugs are inevitably found or new features are added, these correspond the occasional spikes in the activity. However, if the initial phase of the project can't be completed in a timely fashion then nothing else can happen.

In my experience, if you can't get something interesting working in a few days the likelihood of actually finishing the project starts rapidly approaching zero.

With Clojure you can get things done fast, fast enough that you get something working while the initial spark of excitement is still present. I would hazard to guess that this is the reason why Clojure has so many great libraries despite being a young language.

Introducing Selmer

Tue, 30 Jul 2013 00:00:00 +0000

Rationale

There are a number of templating engines available in Clojure. Some of the popular ones include Hiccup, Enlive, Laser, Stencil, mustache.clj and Clabango.

As I've mentioned previously, my personal preference is for Clabango syntax. In my opinion it provides the right balance between simplicity and flexibility. Being modeled on Django template syntax it's also very accessible to those who are new to Clojure web development.

However, one major downside to Clabango is that it's slow. On TechEmpower fortunes benchmark Luminus is crawling behind the Compojure results. Yes, you read that right, it's nearly 20 times slower for Clabango to render the results. The difference being that the Compojure benchmark is using Hiccup for rendering the results while Luminus is using Clabango.

The core problem is that Clabango always parses the source files when rendering a template. This is very expensive as it involves disk access and scanning each character in the source file each time a page is served. Dan states that performance has not been a priority.

On top of that, some of the existing behaviours put limitations on how much the performance can ultimately be improved. For example, the child templates aren't required to put their content inside blocks. Clabango parses the templates and creates the AST that's then evaluated. This means that you can put blocks inside the if tags and decide at runtime whether they will be included. If inheritance resolution is pushed to compile time this becomes a problem.

After having some discussions with bitemyapp and ceaserbp we decided that it would be worth writing a fresh impelementation with pefromance as its primary goal. Another reason is that I would like to be able to ensure that the templating engine in Luminus isn't a compromise between speed and convenience. Owning the implementation is the best way to achieve that.

Enter Selmer

All this resulted in Selmer named after the guitar favored by Django Reinhardt whom in turn Django is named after. Selmer aims to be a near drop in replacement for Clabango. The current version is already quite fast keeping pace with Stencil which is one of the faster engines around.

In order to minimize the work that's done at runtime Selmer splits the process into three distinct steps. These steps include preprocessing, compilation and rendering.

First, Selmer will resolve the inheritance hierarchy and generate the definitive template source to be compiled. The extends and include tags will be handled at this time.

The compilation step then produces a vector of text nodes and runtime transformer functions.

The renderer uses these compiled templates to generate its output. The text gets rendered without further manipulation while the transformers use the context map to generate their output at runtime.

Using this approach we minimize the amount of logic that needs to be executed during each request as well as avoiding any disk access in the process.

In order not to have to restart the application when the source templates are changed the renderer checks the last updated timestamp of the template. When the timestamp is changed a recompile is triggered.

Performance Tricks

To our chagrin the first run of the parser ran no better than Clabango. This was rather disappointing considering we took pains to be mindful of the performance issues. However, this mystery was quickly solved by profiling the parser.

Sure enough majority of time was spent in reflection calls. One major problem was that the renderer had to check whether each node was text or a function:

(defn render [template params]  
  (let [buf (StringBuilder.)]
    (doseq [element template] 
      (.append buf (if (string? element) element (element params))))
    (.toString buf)))

Protocols offer an elegant solution to this problem. With their help we can move this work to compile time as follows:

(defprotocol INode
  (render-node [this context-map] "Renders the context"))

(deftype FunctionNode [handler]
  INode
  (render-node ^String [this context-map]
    (handler context-map)))

(deftype TextNode [text]
  INode
  (render-node ^String [this context-map]
    text))

Now our parser can happily run along and call render-node on each element:

(defn render-template [template context-map]
  """ vector of ^selmer.node.INodes and a context map."""
  (let [buf (StringBuilder.)]
    (doseq [^selmer.node.INode element template]
        (if-let [value (.render-node element context-map)]
          (.append buf value)))
    (.toString buf)))

With this change and a few type annotations the performance improved dramatically. Running clojure-template-benchmarks the results are comparable to Stencil. Here are the results from benchmarking on my machine:

Clabango

Simple Data Injection
- Execution time mean : 657.530826 µs
- Execution time std-deviation : 2.118301 µs
Small List (50 items)
- Execution time mean : 2.446739 ms
- Execution time std-deviation : 17.448003 µs
Big List (1000 items)
- Execution time mean : 28.230365 ms
- Execution time std-deviation : 173.518425 µs

Selmer

Simple Data Injection
- Execution time mean : 42.444958 µs
- Execution time std-deviation : 235.652171 ns
Small List (50 items)
- Execution time mean : 209.158509 µs
- Execution time std-deviation : 4.045131 µs
Big List (1000 items)
- Execution time mean : 3.223797 ms
- Execution time std-deviation : 55.511322 µs

Stencil

Simple Data Injection
- Execution time mean : 92.317566 µs
- Execution time std-deviation : 213.253353 ns
Small List (50 items)
- Execution time mean : 290.403204 µs
- Execution time std-deviation : 1.801479 µs
Big List (1000 items)
- Execution time mean : 1.223634 ms
- Execution time std-deviation : 4.264979 µs

As you can see Selmer is showing a large improvement over Clabango and has no trouble keeping up with Stencil.

Obviously, this benchmark is fairly simplistic so you can take it with a grain of salt. If anybody would like to put together a more comprehensive suite that would be great. :)

Current status

The library implements all the functionality offered by Clabango and passes the Clabango test sutie. There are a few minor deviations, but overall it should work as a drop in replacement without the need to change your existing HTML templates.

We also have a few new features such as the Django {{block.super}} tag support and ability to use filters in if statements. In Selmer you can write things like:

(selmer.filters/add-filter! :empty? empty?)

(render 
  "{% if files|empty? %}
   no files available 
   {% else %} 
       {% for file in files %}{{file}}{% endfor %} 
   {% endif %}"
  {:files []})

Switching to Selmer involves swapping the [clabango "0.5"] dependency for [selmer "0.5.3"] and referencing selmer.parser instead of clabango.parser. Selmer provides the same API for rendering templates using the selmer.parser/render and selmer.parser/render-file functions.

One major area of difference is in how custom tags and filters are defined. Defining a filter is done by calling selmer.filters/add-filter! with the id of the filter and the filter function:

(use 'selmer.filters)

(add-filter! :embiginate #(.toUpperCase %))

(render "{{shout|embiginate}}" {:shout "hello"})
=>"HELLO"

Defining custom tags is equally simple using the selmer.parser/add-tag! macro:

(use 'selmer.parser)

(add-tag! :foo
  (fn [args context-map]
    (str "foo " (first args))))

(render "{% foo quux %} {% foo baz %}" {})
=>"foo quux foo baz"

tags can also contain content and intermediate tags:

(add-tag! :foo
  (fn [args context-map content]
    (str content))
  :bar :endfoo)

(render "{% foo %} some text {% bar %} some more text {% endfoo %}" {})
=>"{:foo {:args nil, :content \" some text \"}, :bar {:args nil, :content \" some more text \"}}"

Selmer also supports overriding the default tag characters using :tag-open, :tag-close, :filter-open, :filter-close and :tag-second keys:

(render "[% for ele in foo %]<<[{ele}]>>[%endfor%]"
                 {:foo [1 2 3]}
                 {:tag-open \[
                  :tag-close \]})

This makes it much easier to use it in conjunction with client-side frameworks such as AngularJs.

One limitation Selmer has is the way it handles inheritance. Since the inheritance block hierarchy is compiled before the parsing step, any content in child templates must be encapsulated in block tags. Free-floating tags and text will simply be ignored by the parser. This is in line with Django behavior.

So there you have it. If you like Django template syntax or just want a fast templating engine then give Selmer a try.

As it is a new project there may be bugs and oddities so don't hesitate to open an issue on the project page if you find any. So far I haven't found any problems in switching my application from Clabango to Selmer and the test coverage is fairly extensive at this point.

lib-noir access rule madness

Wed, 12 Jun 2013 00:00:00 +0000

Access rule handling in lib-noir has seen some major rework. James Reeves pointed out that the way the restrict macro worked was not ideal as it wasn't entirely idiomatic and wasn't very composable. For example it didn't take into account the use of the context macro in Compojure.

While there are some breaking changes, it's pretty easy to migrate the old rules and the new approach provides a lot more flexibility.

The first thing that's changed is how the restricted routes are defined. The macro now wraps the handler itself instead of the whole route. So instead of doing

(restricted GET "/private" [] handler)

you would now write:

(GET "/private" [] (restricted handler))

Access rules definitions in the noir.util.middleware/app-handler have been revamped as well. The rules can now be specified by passing either a function representing a single rule or a map representing a group of rules.

When specified as a function, the rule must accept a single parameter that is the request map. Such rules will implicitly redirect to the "/" URI.

The rule group map contains the following keys:

:redirect - the URI string or a function to specify where requests will be redirected to if rejected (optional defaults to "/")
:uri - the URI for which the rules in the map will be activated (optional if none specified applies to all URIs)
:uris - a vector of URI patterns for which the rules in the map will be activated (optional)
:rule - a single rule function for the group
:rules - a vector containing the rule functions associated with the specified :redirect and the :uri
:on-fail - alternative to :redirect allows providing a function that accepts a request and handles the failure case

The :rules key can point to either a vector or a map. If the rules are a vector the default behavior is that every rule in the group must succeed. If rules are specified as a map, you can provide the resolution strategy using the :any and :every keys.

Let's take a look at an example of how this all works below:

(def-restricted-routes

(defroutes app-routes
 ;;restricted routes
 (GET "/restricted" [] (restricted "this page is restricted"))
 (GET "/restricted1" [] (restricted "this is another restricted page"))
 (GET "/users/:id" [] (restricted "howdy"))
 (GET "/admin" [] (restricted "admin route"))
 (GET "/config" [] (restricted "config route"))
 (GET "/super-secret" [] (restricted "secret route"))
 ;;public routes
 (GET "/denied1" [] "denied")
 (GET "/denied2" [] "denied differently"))

(def app 
 (middleware/app-handler 
   [app-routes]
   :access-rules 
   [(fn [req] (session/get :user))

    {:uri "/restricted"
     :redirect "/denied1"
     :rule (fn [req] false)}

    {:redirect (fn [req] 
                 (log/info (str "redirecting " (:uri req)))
                 "/denied2")
     :uri "/users/*"
     :rule (fn [req] false)}

    {:uris ["/admin*" "/config*"]
     :rules {:any [(fn [req] (session/get :admin))
                   (fn [req] (session/get :root))]}}

    {:on-fail (fn [req] "you tried to access the super secret page!")
     :uri "/super-secret*"
     :rules [(fn [req] (session/get :admin))
             (fn [req] (session/get :root))]}

    {:uri "/super-secret*"
     :rules {:every [(fn [req] (session/get :admin))
                     (fn [req] (session/get :root))]
             :any   [(fn [req] (session/get :zeus))
                     (fn [req] (session/get :athena))]}}
]))

The first rule will be activated for any handler that's marked as restricted. This means that all of the restricted pages will redirect to "/" if there is no user in the session.

The second rule will only activate if the request URI matches "/restricted" and will be ignored for other URIs. The "/restricted" route will redirect to the "/denied1" URI.

The third rule will match any requests matching the "/users/" URI pattern. These requests will be redirected to the "/denied2" URI and the URI of the request will be logged.

The next rule group matches both the "/admin*" and the "/config*" patterns and required that either the :admin or the :root keys are set in the session in addition to the :user key specified by the global rule.

Next, we have a rule group that uses :on-fail function that can provide its own handler instead of doing a redirect. It requires that both the :admin or the :root keys are set in the session.

Finally, we have a group that uses a mix of :every and :any keys to specify its rules.

The access-rule macro has been removed in favor of specifying rule groups directly in the handler. This makes it easier to see how all the rules are defined and what routes each set of rules affects.

With this new approach we can create independent rule groups for specific URI patterns as well as easily specify generic rules that affect all restricted handlers.

I found the new rule managing scheme to work better for my projects. I'd be interested on getting feedback whether it works for others as well and I'm always open to suggestions for improvements. :)

what's new in lib-noir

Sat, 25 May 2013 00:00:00 +0000

It's been nearly a year since lib-noir was split out into a stand-alone library. During this time the work on it has continued at a steady pace. There have been numerous bug fixes and many new features have been added to the library.

Many of these come either from user suggestions or contributions. So, if there is something that you'd like to see improved don't hesitate to submit an issue or make a pull request.

In this post I'd like to highlight some of the major new features that have been recently added.

Middleware

The app-handler in noir.util.middleware now accepts optional:middleware and :access-rules parameters.

Since the outer middleware is evaluated first, if you wrap the app-handler in custom middleware it will execute before any of the standard middleware is executed. This is a problem if you wish to get access to things like the session, eg:

(defn log-user-in-session [handler]
  (fn [req]
    (timbre/info (session/get :user))
    (handler req)))

(def app (-> (middleware/app-handler all-routes)
             log-user-in-session))

If we try to run our app with the above handler we'll get the following exception:

java.lang.ClassCastException: clojure.lang.Var$Unbound cannot be cast to java.util.concurrent.Future

This happens due to the fact that noir.session uses the *noir-session* dynamic variable to keep track of the session. This variable is bound by the wrap-noir-session middleware. Since the log-user-in-session executes before it, the session is not yet bound.

The :middleware key allows specifying a vector containing custom middleware to wrap the handler before the standard middleware:

(def app (middleware/app-handler all-routes
          :middleware [log-user-in-session]))

Now, the log-user-in-session will be called after the wrap-noir-session is called and work as expected.

The :access-rules key allows specifying the access rules for the wrap-access-rules middleware. Each set of rules should be specified as a vector with the contents matching the wrap-access-rules arguments:

(defn private-pages [method url params]    
    (session/get :user-id))

(def app (middleware/app-handler all-routes 
          :access-rules
          [[{:redirect "/unauthorized"} private-pages]]))

There's also a new middleware wrapper called wrap-rewrites that allows rewriting URIs based on regex.

The rewrite rules should be supplied as pairs of the regex and the string the matching URL should be rewritten with. The first regex that matches the request's URI will cause it to be replaced with its corresponding string before calling the wrapped handler:

(wrap-rewrites handler #"/foo" "/bar")

Above, all occurances of the/foo URI will be replaced with /bar.

Routes

There's now a noir.util.route/def-restricted-routes macro for creating groups of restricted routes. Where before you had to do something like this:

(defroutes private-routes
  (restricted GET "/route1" [] handler1)
  (restricted GET "/route2" [] handler2)
  (restricted GET "/route3" [] handler3)
  (restricted GET "/route4" [] handler4))

you can now simply do:

(def-restricted-routes private-routes
  (GET "/route1" [] handler1)
  (GET "/route2" [] handler2)
  (GET "/route3" [] handler3)
  (GET "/route4" [] handler4))

The macro will automatically mark all the routes as restricted for you.

Finally, the access rules used to control the restricted routes are more flexible now as well. The redirect target can now point to a function as well as a string, eg:

(def app (middleware/app-handler all-routes 
          :access-rules
          [[{:redirect 
             (fn [] 
              (println "redirecting") "/unauthorized")} 
             private-pages]]))

As always, Luminus provides the latest lib-noir, so all the new features are available there as well.

Packaging a Clojure+ClojureScript jar

Wed, 22 May 2013 00:00:00 +0000

I maintain a Clojure Markdown parser library called markdown-clj. I originally wrote it because I was curious to see just how concise a Clojure Markdown parser would be. Turns out that it's pretty concise. :)

Then I saw a post from Brian Carper that highlighted a problem with having different Markdown parsers on the client and the server.

Since Markdown specification is somewhat loose, most implementations interpret it differently. This means that if you're rendering a preview on the client using a JavaScript library and using a different library, such as pegdown, to render it on the server you may get some surprises.

Since my library was already written in pure Clojure I figured it wouldn't be difficult to cross-compile it to ClojureScript as well.

That turned out to be very easy to do. I split out the element transformers into a separate namespace that's shared between Clojure and ClojureScript cores. However, for the longest time I only packaged it for distribution as a Clojure library.

I finally had a bit of free time to look at ClojureScript packaging over the weekend and I'm happy to report that the dependency now works for both Clojure and ClojureScript out of the box.

While pure ClojureScript libraries compile without any extra work, I found a few gotchas that are specific to cross-compiling.

If you have a project that contains both Clojure and ClojureScript code in it, then only the clj files will be packaged in the jar by default. After some reading of the lein-cljsbuild docs I found the solution.

Here's what I ended up doing to get cljs namespaces to be packaged along with the clj namespaces:

:cljsbuild
{:crossovers [markdown.transformers]
  :crossover-path "crossover"
  :crossover-jar true        
  :builds {:main
           {:source-paths ["src-cljs"]
            :jar true
            :compiler {:output-to "js/markdown.js"
                       :optimizations :advanced
                       :pretty-print false}}
           :dev 
           {:compiler {:optimizations :whitespace
                       :pretty-print true}}}}

I specify the :crossover-path, note that this path has to be different from your :source-paths or the files there will be overwritten.

Next, I added the :corssover-jar true to indicate that I wish the crossover namespaces to appear in the resulting jar.

I also added :jar true to the :main section of the :builds. This is needed to include the namespaces in the src-cljs source directory.

Finally, you also need to include :clojurescript? true in project.clj to indicate that the project contains ClojureScript sources. Here's the complete project file that I'm using:

(defproject markdown-clj "0.9.25"
  :clojurescript? true
  :description "Markdown parser"
   :url "https://github.com/yogthos/markdown-clj"
   :license {:name "Eclipse Public License"
             :url "http://www.eclipse.org/legal/epl-v10.html"}
   :dependencies [[org.clojure/clojure "1.5.1"]
                  [criterium "0.3.1" :scope "test"]]
   :plugins [[lein-cljsbuild "0.3.2"]]
   :hooks [leiningen.cljsbuild]
   :test-selectors {:default (complement :benchmark)
                    :benchmark :benchmark
                    :all (constantly true)}
   
   :cljsbuild
   {:crossovers [markdown.transformers]
    :crossover-path "crossover"
    :crossover-jar true        
    :builds {:main
             {:source-paths ["src-cljs"]
              :jar true
              :compiler {:output-to "js/markdown.js"
                         :optimizations :advanced
                         :pretty-print false}}
             :dev 
             {:compiler {:optimizations :whitespace
                         :pretty-print true}}}})

The resulting jar will contain all your clj and cljs files along with the crossover namespaces.

For me, being able to manage dependencies using Leiningen is a definite killer feature when it comes to ClojureScript.

Clojure Data Analysis Cookbook Review

Wed, 24 Apr 2013 00:00:00 +0000

I was recently asked to review the Clojure Data Analysis Cookbook. Data analysis happens to one of the major niches where Clojure has been gaining popularity. However, the documentation on the subject is far from focused.

The book provides a collection of recipes for accomplishing common tasks associated with analyzing different types of data sets. It starts out by showing how to read data from a variety of sources such as JSON, CSV, and JDBC. The next chapter provides a number of examples of how to sanitize the collected data and sample large data sets. After covering loading and sanitizing the data, the book discusses a number of different strategies for processing it.

Some of the highlights include using the Clojure STM, parallel processing of the data, including useful tricks for partitioning, using reducers, and distributed processing with Hadoop and Casalog.

I found the sections on handling large amounts of data particularly interesting. Often times, it's easy to come up with a solution that works for a small data set, but doesn't scale to handle large amounts of data. One of the techniques the book discusses is the use of lazy sequences. Another example is using heuristics to decide how to partition large data sets data sets effectively.

The book closes with a chapter dealing with the presentation the processed data. First, it covers using Incanter charts and then shows how to display the results in the browser with ClojureScript and NVD3.

For the most part, the book is very much example oriented. The examples are accompanied by explantation of how they all fit together. If you're like me and like to get hands on experience then I think you'll like the style of the book.

The examples are short in size and easy to understand. I found that the best way to work through the book was by following along with a REPL.

The book also introduces the reader to a number of libraries. Some, such as Incanter are well known, while others like parse-ez less so. In my experience, the documentation for many Clojure libraries is often lacking. The recipes in the book serve as a good reference for how to make the most of the tools available.

I would say one missed opportunity in the book is that the examples don't seem to build on each other. You'll see many examples of doing specific tasks, but they will tend to be self contained and don't build up to anything more substantial.

I suspect this was done in order to keep content accessible so that the reader can look at any section without having to have read the others. Conversely, don't expect to see examples of how to structure your projects and build applications end to end.

Overall, I would say this book is aimed at somebody who is already comfortable using Clojure and would like to learn some of the more advanced techinques for working with data processing and analysis. If you're thinking of using Clojure for analyzing your data sets this book will likely save you a lot of time and serve as a handy reference down the road.

Introducing cljs-ajax

Tue, 09 Apr 2013 00:00:00 +0000

I recently started working on a project using ClojureScript and it's turning out to be a really good experience so far. I've been using Domina and Dommy for DOM manipulation and templating. Both libraries are very easy to use and provide all the functionality needed for common operations.

Surprisingly, I didn't find any up to date libraries for handling Ajax. The only one I could find is fetch. Unfortunately, it depends on Noir which is no longer maintained.

I ended up writing a wrapper for goog.net.XhrIo called cljs-ajax. It provides an API similar to clj-http and handles all the nitty gritty details for you.

Currently, the API provides ajax-request, GET, and PUT functions. The ajax-request function accepts the following parameters:

uri - the URI for the request
method - a string representing the HTTP request type, eg: "PUT", "DELETE", etc.
format - a keyword indicating the response format, can be either :json or :edn, defaults to :edn
handler - success handler, a function that accepts the response as a single argument
error-handler - error handler, a function that accepts a map representing the error with keys :status and :status-text
params - a map of params to be sent to the server

The GET and POST are helper functions that accept a URI followed by a map of options containing any of the following keys:

:handler - the handler function for successful operation should accept a single parameter which is the deserialized response
:error-handler - the handler function for errors, should accept a map with keys :status and :status-text
:format - the format for the response :edn or :json defaults to :edn
:params - a map of parameters that will be sent with the request

Here's some example usage:

(ns foo
 (:require [ajax.core :refer [GET POST]]))

(defn handler [response]
 (.log js/console (str response)))

(defn error-handler [{:keys [status status-text]}]
 (.log js/console 
  (str "something bad happened: " status " " status-text)))

(GET "/hello")

(GET "/hello" {:handler handler
               :error-handler error-handler})

(POST "/hello")

(POST "/send-message" 
      {:params {:message "Hello World"
                :user    "Bob"}
      :handler handler
      :error-handler error-handler})

(POST "/send-message" 
      {:params {:message "Hello World"
                :user    "Bob"}
      :handler handler
      :format :json
      :error-handler error-handler})

The latest version of Luminus comes packaged with a ClojureScript example when the +cljs option is selected. Let's create a new project called ajax-example and take a look at how it works:

lein new luminus ajax-example +cljs

The project.clj will contain the dependencies for Domina, Dommy, and cljs-ajax as well as a cljsbuild configuration. The current version of cljsbuild references an old version of ClojureScript, so the latest version is also explicitly included as a dependency.

In order to use the ClojureScript from our page we'll first need to compile it. This is done by running lein cljsbuild once. The resulting artifact will be placed under resources/public/js/site.js as specified in the cljsbuild section of the project.

You'll notice that the build takes a while to run. Luckily, if we run it using lein cljsbuild auto it will run much faster and any time we make changes to any of the ClojureScript namespaces they will trigger an incremental build.

Working with the auto build running is nearly as seamless as working with plain old JavaScript. You make a change in the source, save, and reload the page. The compilation step tends to take under a second, so the intermediate delay is barely noticeable.

Our project has a source directory called src-cljs where ClojureScript namespaces live. It contains a file called main.cljs. This example illustrates using GET and POST calls to interact with the server as well as rendering DOM elements. Let's take a look inside it:

(ns cljs-test.main
 (:require [ajax.core :refer [GET POST]]
           [domina :refer [value by-id destroy-children! append!]]
           [domina.events :refer [listen!]]
           [dommy.template :as template]))

(defn render-message [{:keys [message user]}]
 [:li [:p {:id user} message " - " user]])

(defn render-messages [messages]
 (let [messages-div (by-id "messages")]
   (destroy-children! messages-div)
   (->> messages
        (map render-message)
        (into [:ul])
        template/node
        (append! messages-div))))

(defn add-message [_]
 (POST "/add-message"
       {:params {:message (value (by-id "message"))
                 :user    (value (by-id "user"))}
        :handler render-messages}))

(defn ^:export init []
 (GET "/messages" {:handler render-messages})
 (listen! (by-id "send")
          :click add-message))

Here, we have a couple of functions to render the messages we receive from the server. The render-message function accepts a map with the keys message and user and creates a list item. The render-messages will create a list from the messages and render it using template/node function. The rendered messages will be appended to the div with the id messages using the append! function.

Next, we have a function to add a a new message. It grabs the values from elements selected by their ids and sends them as params named message and user. The server responds with a list of current messages. So we use render-messages as the response handler.

In our init function, we send a GET request to grab the current messages, then we bind the add-message function to the button with the id send.

On the server side we have a ajax-example.routes.cljsexample namespace. It provides the routes to render the page and handle the /messages and /add-message operations.

(ns ajax-example.routes.cljsexample
 (:require [compojure.core :refer :all]
           [noir.response :as response]
           [ajax-example.views.layout :as layout]))

(def messages
 (atom
   [{:message "Hello world"
     :user    "Foo"}
    {:message "Ajax is fun"
     :user    "Bar"}]))

(defroutes cljs-routes
 (GET "/cljsexample" [] (layout/render "cljsexample.html"))
 (GET "/messages" [] (response/edn @messages))
 (POST "/add-message" [message user]
       (response/edn
         (swap! messages conj {:message message :user user}))))

As you can see, the routes simply return EDN responses to the client. Finally, we have the template for the actual example page, that looks as follows:

{% extends "cljs_test/views/templates/base.html" %}


{% block content %}

<br/>
<div id="messages"></div>
<textarea id="message"></textarea>
<br/>
<input type="text" id="user"></input>
<br/>
<button id="send">add message</button>

<!--  scripts -->
<script type="text/javascript" src="js/site.js"></script>
<script type="text/javascript">
	cljs_test.main.init();
</script>

{% endblock %}

The page references the site.js script that will be output by the compiler and calls the init function that we saw above.

Overall, I feel that ClojureScript is rapidly becoming a viable alternative to using JavaScript on the client. There are still some rough edges, but most things work out of the box and you get many of the same benefits associated with using Clojure on the server.

a book is coming

Sat, 23 Mar 2013 00:00:00 +0000

First of all, I'd like to thank all those who've helped with Luminus. Since the original release on Clojars at the end of December there's been over 700 downloads, and the framework site has over 3,500 unique visits!

There's also been many contributions for improved documentation, template fixes, and lib-noir improvements. I'm really thankful for all the help improving the framework and moving it forward. I'd especially like to thank Ed Tsech, who's been toiling on it with me for the last few months. :)

I'm really glad to be able to contribute to popularizing the language and making it more accessible. On that note I have some exciting news. I've recently got signed by the The Pragmatic Programmers to write a book on web development using Clojure.

There is a number of books discussing the fundamentals of Clojure as a language. However, none of the books focus on applying these fundamentals to building real-world solutions. Respondents of the 2012 State of Clojure survey indicated that there still exists a gap in traditional documentation. Specifically, an interest in current tools, libraries, and best practices is not being met. It is my goal to help fill this gap.

I will provide an overview of Clojure as a web development platform, highlighting exactly what makes it so effective. The book will take a tutorial-focused approach to building a production-ready web application from conception to deployment.

The target audience is anyone interested in using Clojure as a web development platform. This includes: those who are currently using the JVM for development; Ruby and Python users who would like to take advantage of the breadth of features and libraries offered by the JVM; and readers simply interested in learning how to develop web applications using Clojure.

I'm quite thrilled about this project and I hope to write the book I wish I had when I spent countless hours googling for tutorials and examples. :)

New Templating Engine in Luminus

Sun, 10 Mar 2013 00:00:00 +0000

I'm happy to announce that Luminus now defaults to using Clabango for HTML templating instead of Hiccup.

I'd like to explain some of the reasoning behind this decision. The primary drive behind Luminus is to make Clojure web development more accessible. This means that the barrier to entry for those who are new to the language should be as low as possible.

Since Clabango is based on the Django templates, it's immediately familiar to anybody who's done templating with other frameworks such as Django, Rails, or JSP. This also makes it easier to migrate exiting sites to use Luminus.

Because the templates are written in plain HTML it's easy to work with designers and other people who aren't versed in Clojure.

Finally, Clabango enforces the separation between the application logic and the presentation. When using Hiccup it's for one to start bleeding into the other if you're not careful.

However, if you are a fan of Hiccup there's nothing to worry about. Everything will work as it did before if you use the +hiccup flag when creating the application.

As always feedback and suggestions are most welcome. :)

New kid on the templating block

Sat, 02 Mar 2013 00:00:00 +0000

Update: Selmer is currently the recommended Django style templating engine

As you may know, there are a few Clojure templating engines floating around. The two most popular ones are probably Hiccup and Enlive.

Hiccup is a nice and simple templating engine. Unfortunately, its biggest advantage is also it's greatest weakness. Since Hiccup templates are written using Clojure data structures, they're simply part of your regular code.

This makes the templates inaccessible to people not versed in Clojure. For example, if you're working with a designer, you can't just give them your template to work with.

Another issue is that it makes it easy for your frontend and backend logic to bleed into each other if you're not careful. Finally, you have to redeploy your site any time you wish to make a change to the layout.

Enlive avoids these problems by using a more traditional approach and using plain HTML markup for its templates. The problem with Enlive lies in its complexity. This spawned Laser, which also allows using pure HTML without any additional markup for its templates. In the words of the author:

Enlive does its job and is the precursor to the way laser does things. However, it is very large and (arguably?) complex compared to laser. laser strives to be as simple as possible.

If you haven't already checked out Laser I certainly urge you to do so!

However, the engine I'd like to focus on in this post is Clabango. It's modeled after Django's templating library and I found that it clicked with me immediately.

Let's take a look at how to convert the example guestbook application from Luminus to use Clabango instead of Hiccup.

We'll first create the project with support for H2 embedded DB by running:

lein new luminus guestbook +h2

We'll then open up our project.clj and add the Clabango [clabango "0.5"] dependency it.

Next, we'll create a templates folder under resources/public. This is where all the Clabango templates will live.

Clabango provides two way to load templates using the clabango.parser namespace. We can either use render-file function to load a template from a file or render to load it from a string.

These functions take two parameters, the template source and a map containing the items that will be populated in the template when it's compiled.

For example, if we had a template such as:

<h2>Hello {{user}}</h2>

We could then render it by calling render as follows:

(render "<h2>Hello {{user}}</h2>" {:user "John"})

Clabango will then replace every occurance of {{user}} with John instead. In case user happens to be a map, we can access its keys using the dot notation:

<h2>Hello {{user.last}}", " {{user.first}}</h2>

The templates provide support for some other useful things like filters, tag definitions, and template inheritance. However, we won't worry about any of that right now.

Let's take a look how to load up the templates using the render function. We won't use render-file since it looks for resources relative to the src folder. We'll use lib-noir.io/slurp-resource to load our templates from the public folder instead. We'll create a helper in our guestbook.util namespace to do that:

(ns guestbook.util  
  (:require ...
            [clabango.parser :as parser]))


(defn render [template params]
  (parser/render (io/slurp-resource template) params))

With that out of the way, let's create the model for our application. We'll open up the guestboook.models.schema namespace and replace create-users-table with create-guestbook table:

(defn create-guestbook-table []
  (sql/with-connection
    db-spec
    (sql/create-table
      :guestbook
      [:id "INTEGER PRIMARY KEY AUTO_INCREMENT"]
      [:timestamp :timestamp]
      [:name "varchar(30)"]
      [:message "varchar(200)"])
    (sql/do-commands
      "CREATE INDEX timestamp_index ON guestbook (timestamp)")))

then update create-tables to call it instead:

(defn create-tables
  "creates the database tables used by the application"
  []
  (create-guestbook-table))

We'll also update the init function in the guestbook.handler to call create-tables if the database isn't already initialized:

(defn init []
  (if-not (schema/initialized?) (schema/create-tables))
  (println "guestbook started successfully..."))

Next, let's open up the guestbook.models.db namespace and replace the code to create and retrieve users with the code to save and load messages:

(ns guestbook.models.db
  (:use korma.core
        [korma.db :only (defdb)])
  (:require [guestbook.models.schema :as schema]))

(defdb db schema/db-spec)

(defentity guestbook)

(defn save-message
  [name message]
  (insert guestbook 
          (values {:name name
                   :message message
                   :timestamp (new java.util.Date)})))

(defn get-messages []
  (select guestbook))

We can test that everything works by calling save-message from the REPL to create some messages and then calling get-messages to see that they're retrieved correctly. If everything works as expected then we're ready to take a look at making our pages.

First, let's create a template for the home page. We'll do this by making a welcome.html file under the resources/public/templates folder.

Here is where we finally get to see Clabango in action. We'll first use it to iterate the messages and create a list from them:

<ul>
{% for item in messages %}
  <li> 
      <blockquote>{{item.message}}</blockquote>
      <p> - {{item.name}}</p>
      <time>{{item.timestamp}}</time>
  </li>
{% endfor %}
</ul>

As you can see above, we use a for iterator to walk the messages. Since each message is a map with the message, name, and timestamp keys, we can access them by name.

Next, we'll add an error block for displaying errors that might be populated by the controller:

{% if error %}
<p>{{error}}</p>
{% endif %}

Here we simply check if the error field was populated and display it. Finally, we'll create a form to allow users to submit their messages:

<form action="/" method="POST">
	<p>Name: <input type="text" name="name" value={{name}}></p>
	<p>Message: <input type="text" name="message" value={{message}}></p>
	<input type="submit" value="comment">
</form>

This takes care of creating the template, now let's take a look at how we populate the templated fields in our controller.

We'll navigate to the guestbook.routes.home namespace and update our home function to render the template when called:

(defn home-page [& [name message error]]
  (layout/common   
    (util/render "/templates/welcome.html" 
                 {:error    error
                  :name     name
                  :message  message
                  :messages (db/get-messages)})))

Above, we simply create a map with all the fields we wish to populate. Then we pass it along with the name of the template file to the render function we defined earlier. Note that we can keep using the Hiccup layout to create the skeleton for the pages. The rest of the code in the home namespace stays the same as it was:

(defn save-message [name message]
  (cond
 
    (empty? name)
    (home-page name message "Some dummy who forgot to leave a name")
 
    (empty? message)
    (home-page name message "Don't you have something to say?")
 
    :else
    (do
      (db/save-message name message)
      (home-page))))

(defroutes home-routes
  (GET "/" [] (home-page))
  (POST "/" [name message] (save-message name message))
  (GET "/about" [] (about-page)))

As you can see, Clabango is very simple to use and allows cleanly separating your markup from your controllers. I think it's an excellent addition to the ever growing Clojure toolbox.

Complete sources for this post are available here.

update

The approach I took with putting templates under the resources folder will not work with template inheritance. So, you're best off simply using render-file from Clabango and keeping your templates under the src folder.

lib-noir updates

Sun, 24 Feb 2013 00:00:00 +0000

I've had a bit of time to hack on lib-noir recently. Specifically, I decided to update the handling of access rules.

Previously, you could use wrap-access-rules by passing one or more rule functions. Each function would accept a method, url, and params and return a boolean indicating whether the rule is satisfied. Using these functions the wrapper would then decide wether the page should be displayed or if the client will be redirected to "/".

This was serviceable for doing some basic restrictions, like making pages private where a rule would check if a user was in the session:

(defn private-page [method url params]
  (session/get :user))

However, it provided no way to redirect to a different URIs based on what rules failed. The update allows using multiple wrap-access-rules wrappers each redirecting to its own redirect URI, as follows:

(-> handler
  (wrap-access-rules rule1)
  (wrap-access-rules {:redirect "/unauthorized"} rule2 rule3))

The first set of rules that fails will redirect to its redirect target, defaulting to "/" if none is provided. This way we can create rule groups each having different behaviours.

Another addition is the noir.util.route/access-rule macro. The macro accepts a URI pattern and a condition. The condition is only checked if the URI of the page being checked matches the pattern.

The macro implicitly defines the method, url, and params variables, so they can be used by the logic in the condition:

(def private-pages
  (access-rule "/private/:id" (= (session/get :user) (first params))))

The above rule will only be triggered for pages matching the "/private/:id" pattern. Hopefully, the new additions will make it easier to work with access rules in lib-noir. Complete documentation for the feature is available at Luminus.

I'm also interested in hearing any feedback and suggestions regarding the current implementation. :)

update

After a bit of discussion with Ed Tsech, we decided that it would be better to make the parameters to the access-rule explicit.

So, now instead of defining access-rule by simply providing the URL pattern and a condition, you would also pass the arguments vector with the method, url, and params:

(def private-pages 
  (access-rule "/private/:id"  [_ _ params] 
    (= (session/get :user) (first params))))

While it's slightly more verbose, it's a lot less magical and there's no risk of the macro masking any variables in scope.

One Ring to rule them all

Fri, 11 Jan 2013 00:00:00 +0000

The latest release of Luminus is no longer using a custom server.clj which starts up Jetty using run-jetty. Instead, it now relies on lein-ring, which in turns uses ring-server to create the server.

Snice you no longer have a -main in the project, you can't use lein run to start it up for development. Instead, use lein ring server, which will run Jetty for you.

If you need to start the server from within a REPL, then you can use the new repl namespace, which provides start-server and stop-server functions.

When you're packaging the application as a standalone, you run would now run lein ring uberjar instead of lein uberjar. The -main will be created by lein-ring for you based on the handler specified in your project.clj.

This means that all the configuration now lives under project.clj and gets picked up consistently both in development and production modes.

The new changes also simplify Heroku deployment. You no longer need to specify +heroku, the application will have all the necessary settings to run on Heroku out of the box.

Finally, I dropped support for Leiningen 1.x as it doesn't have support for profiles. There's no good reason to continue using it instead of upgrading to 2.x.

Luminus progress report

Tue, 08 Jan 2013 00:00:00 +0000

The work on the framework continues steadily, and I've been integrating some of the feedback I got on the initial release.

I quickly discovered that simply using different files for template modules is insufficient. Many features need to update the project.clj with dependencies or other options.

To deal with this I made a util which reads in the project file and injects dependencies, plugins and other options. Now each plugin can add its own project elements independently of others.

I'm considering taking the same approach to managing the layout as well. For example, if bootstrap support was selected, then its js/css would be included in layout/common. Another use case would be to update the application routes if a module provided some new routes of its own.

I'd also like to highlight some of the additions to lib-noir. There are several new namespaces, such as noir.util.cache, noir.io, and noir.util.route. Let's take a look at each of these in turn.

Caching

Basic caching is provided via noir.util.cache. Cache allows wrapping any expr using (cache id expr), and the expr will only be evaluated if it's not found in the cache or if the cache has been invalidated. In case expr throws an exception the current cached value will be kept.

There are a couple of helpers for invalidating the cache. First, there's invalidate-cache!, which takes a key and removes it from the cache. Then, there's clear-cache! which removes all currently cached items.

It's also possible to set the timeout for cached items using set-cache-timeout! and passing it a value in seconds. If an item remains in the cache longer than the timeout, the cache will attempt to refresh the value by running the expr associated with the item.

Finally, you can set the maximum size of the cache by calling set-cache-size!, when the cache grows past the specified size, oldest items will be removed to make room for new ones.

I'm currently using the cache in Luminus for the documentation pages. Luminus fetches the documentation from github as markdown and then translates it to HTML. This is slow enough to be noticeable to the user. On top of that, github is known to have an occasional outage or two. :)

With this scheme, I can keep the docs up to date without having to redeploy the site, and I don't have to worry about the latency or github uptime.

IO

The noir.io namespace provides some helper functions to make it easier to handle static resources.

You can get the absolute path to the public directory of your application by calling resource-path.

If you need to read a file located in the public folder you can get a URL for the resource by calling get-resource and provided the path relative to the public directory.

If the resource is a text file, such as a markdown document, you can use slurp-resource to read it into a string.

Another addition is the upload-file function which saves the file generated by a multipart/form-data form POST to a path relative to the public folder. An example can be seen here:

(ns myapp.upload
  ...
  (:require [noir.io :as io]))
 
(defn upload-page []
  (common/layout
    [:h2 "Upload a file"]
    (form-to {:enctype "multipart/form-data"}
             [:post "/upload"]            
             (file-upload :file)            
             (submit-button "upload"))))
              
(defn handle-upload [file]
  (upload-file "/uploads" file)
  (redirect
    (str "/" (session/get :user) "/" (:filename file))))
   
(defroutes upload-routes
  (GET "/upload" [] (upload-page))
  (POST "/upload" [file] (handle-upload file)))

Access rules

Noir used to have a pre-route macro, which allowed for filtering and redirecting based on some rules.

Now, lib-noir provides a restricted macro which provides similar functionality.

You can define access rules as functions which accept the method, url, and params. The function then returns a boolean to indicate if the rule succeeded or not.

For example, if we wanted to restrict access to a page so that it's only accessible if the id in session matches the id in the page, we could write a rule like this:

(defn user-page [method url params] 
  (and (= url "/private/:id")
       (= (first params) (session/get :user))))

Then we wrap our handler in wrap-access-rules middleware. The middleware accepts one or more access rule functions, and checks if restricted pages match any of the rules provided.

(def app (-> all-routes
             (middleware/app-handler)
             (middleware/wrap-access-rules user-page)))

With that in place, we can restrict access to our page as follows.

(restricted GET "/private/:id" [id] "private!")

Note that, you have to use noir.util.middleware/app-handler for wrap-access-rules to work correctly. Or manually bind the noir.request/*request*, eg:

(defn wrap-request-map [handler]
  (fn [req]
    (binding [noir.request/*request* req]
      (handler req))))

update I've since made wrap-request-map public in lib-noir, so if you need to wrap the request for any reason, you don't need to roll your own.

I hope you find the new features useful, and as always I'm open to feedback and suggestions for improvements as well as new features.

Luminus progress updates

Fri, 28 Dec 2012 00:00:00 +0000

In this post I'd like to give some updates on the progress of Luminus and the direction it's moving in.

I've had some great chats over at #clojure on IRC, and there's been lots of ideas and brainstorming. It's a very friendly and informative place if you haven't yet visited. :)

After talking it over with Raynes we decided that it would be much better to simply add things to lib-noir than to roll a new library. So, lib-luminus is no more, and instead all the updates will be happening in lib-noir now.

All the current helper functions have already been rolled into version 0.3.0 of lib-noir, so definitely switch to it if you're using lib-luminus currently. The good news is that all you need to do is replace [lib-luminus "0.1.5"] with [lib-noir "0.3.0"] in your project.clj`, and update your namespaces to reference it instead. The function names and behaviour haven't changed.

This segues into the next topic of how the line is drawn between what goes into the library and what belongs in the template.

The strategy here is to add functionality to `lib-noir, while putting configuration in the template. This facilitates an easy path for upgrades as the library continues to improve and evolve, while keeping all the customization in the hands of the user. It also means that the template will act as documentation for how to configure your application.

As the template continues to grow, it will be increasingly difficult to please everybody with a single template. For example, somebody might want to use PostreSQL for their db, while another person might like MySQL, and yet another uses CouchDB and doesn't want to see any of the SQL business at all.

As these things tend to be rather polarizing, the approach will be to let people choose the items they want. Luminus aims to be more of a buffet, where you pick what's on your plate, as opposed omakase with the chef telling you what to eat. :)

To this end, the latest release of Luminus provides a base template which can be extended using +feature notation. Currently, there's two features supported, the first is the addition of bootstrap into the project and the second is support for SQLite.

The way this works is if you want to make a basic application, you'd do the same thing you did before.

lein new luminus myapp

But if you wanted to have bootstrap in your app, then you'd simply do this:

lein new luminus myapp +bootstrap

The best part is that you can mix different extensions together, eg:

lein new luminus myapp +bootstrap +sqlite

When you do that, both features will be added to the resulting project. However, if they have any common files between the two features, then the latest one overwrites the former.

Hopefully, this approach will provide an easy way to add extended configuration while keeping things compartmentalized and easy to maintain. The latest documentation and examples are available at the official Luminus site.

Luminus: a web framework for Clojure

Mon, 24 Dec 2012 00:00:00 +0000

Since the retirement of Noir, there aren't any batteries included web frameworks for Clojure. As I mentioned in an earlier post, moving to Compojure is fairly painless. However, you still have to put a lot of things together by hand.

I suspect this isn't a problem for most people who've already been doing Clojure development. However, it can be daunting for beginners and it also means having to write a lot of boiler plate when making a new site.

I decided to see if I could tie some common libraries together to provide a more comprehensive solution for creating web applications in Clojure. This led to the creation of the Luminus framework, which follows in footsteps of Noir in attempting to make web development in Clojure an easy and accessible experience.

The framework consists of two parts, first is the lib-luminus, which provides some useful utility functions, which I found to be helpful when writing applications. The second is the luminus-template, which is used to generate the base application.

The resulting app is ready to be run standalone or deployed as a war. It can also be run on Heroku by following the steps in the official documentation.

The application generated by the template can be easily modified to fit your needs and shouldn't be any more restrictive than a standard Compojure app. This avoids some of the issues with Noir, where things like using custom middleware were problematic.

The documentation site for Luminus is built using the framework in the spirit of eating my own dog food, and its source is available on github as well.

Hopefully this will be useful in helping people get started. I intend to continue working on it and I'm always open to suggestions, patches, and collaboration. :)

Creating Leiningen Templates

Sun, 16 Dec 2012 00:00:00 +0000

If you've used Leiningen before, you've already seen templates in action. When you create a project using lein new myproject, you end up with a project folder with a namespace called myproject and a core.clj inside it.

The templates are really useful if you need to setup some common boilerplate for your project. In the last post I referenced a template for Compojure, which creates a new batteries included project.

Leiningen uses the lein-newnew plugin for this task. All you have to do to create a new template is to run lein new template <template name>. In my case I created a template called compojure-app:

lein new template compojure-app

As all Leiningen projects, it will contain the project.clj, which will contain the description for our project:

(defproject compojure-app/lein-template "0.2.7"
  :description "Compojure project template for Leiningen"
  :url "https://github.com/yogthos/compojure-template"
  :eval-in-leiningen true
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[leinjacker "0.2.0"]])

It looks like a regular project file, except for the eval-in-leiningen key which ~~causes Leiningen to launch a subprocess~~ prevents Leiningen from launching a separate process for the given project during the build time.

The actual template resides under

src/compojure-template/leiningen/new/compojure_app.clj

It looks as follows:

(ns leiningen.new.compojure-app
  (:use [leiningen.new.templates :only [renderer sanitize year ->files]]
        [leinjacker.utils :only [lein-generation]]))

(def project-file
  (if (= (lein-generation) 2)
    "project_lein2.clj"
    "project_lein1.clj"))

(defn compojure-app
  "Create a new Compojure project"
  [name]
  (let [data {:name name
              :sanitized (sanitize name)
              :year (year)}
        render #((renderer "compojure_app") % data)]
    (println "Generating a lovely new Compojure project named" (str name "..."))
    (->files data
             [".gitignore"  (render "gitignore")]
             ["project.clj" (render project-file)]
             ["README.md"   (render "README.md")]
             ["src/{{sanitized}}/handler.clj"      (render "handler.clj")]
             ["src/{{sanitized}}/server.clj"       (render "server.clj")]
             ["src/{{sanitized}}/common.clj" (render "common.clj")]
             ["resources/public/css/screen.css" (render "screen.css")]
             "resources/public/js"
             "resources/public/img"
             "src/{{sanitized}}/models"
             ["test/{{sanitized}}/test/handler.clj" (render "handler_test.clj")])))

The compojure-app function is where all the fun happens, and it's what gets called when we run lein new compojure-app myapp to create an application using this template.

The function is mostly self explanatory. It uses the render function from leiningen.new.templates to take the template files and put them at the specified path. The {{sanitized}} tag ensures that the generated names for the package folders are valid.

Our template files live under

src/compojure-template/leiningen/new/compojure_app

and they don't need to have the same folder structure as the resulting project. As you can see above, we specify the resulting path explicitly in our template.

The template files look exactly like any regular Clojure source file, except for the {{name}} anchor. This will be replaced with the name of the application we specified when creating the project. Here's the common.clj template as an example:

(ns {{name}}.common
  (:use [hiccup.page :only [html5 include-css]]))
       
(defn layout [& body]
  (html5 
    [:head
     [:title "Welcome to {{name}}"]
     (include-css "/css/screen.css")]
    (into [:body] body)))

Every occurrence of {{name}} will be replaced with myapp instead and we'll have our namespace and greeting customized.

Once you've created your template, you'll need to install it using lein install and then add it as a plugin to your profile under ~/.lein/profiles.clj using the following format:

{:user
  {:plugins [[compojure-app/lein-template "0.2.7"]]}}

That's it, you can now use your new template and never have to write boilerplate for this kind of project again.

If you wish to make your template available to others you can publish it to Clojars by running lein deploy clojars from the console.

Any template published on Clojars can be used directly without needing to add it to your plugins in the profiles.clj as shown above.

The complete source for the template discussed in this post is available here.

Moving to Compojure

Sat, 15 Dec 2012 00:00:00 +0000

It was recently announced that Noir is being deprecated. The primary reason cited is that it simply doesn't add a lot of useful functionality over what's already available in Compojure and makes it difficult to integrate other middleware, such as friend.

The useful parts of Noir have been moved to lib-noir. Together, Compojure and lib-noir provide a very similar experience to what you're already used to if you've been using Noir up to now.

There are some differences of course. The main one is that instead of using the defpage macro, you would now declare your routes using defroutes.

So, if you previously had something like the following:

(defpage "/" []
  (common/layout 
    (form-to [:post "/"]                           
              (text-area {:placeholder "say something..."} "message") 
              [:br]
              (text-field {:placeholder "name"} "id")
              (submit-button "post message"))))

(defpage [:post "/"] params
  (common/layout 
    [:p (:id params) " says " (:message params)]))

Noir would then create the GET and POST routes for "/" behind the scenes. With Compojure we'll have to define the routes explicitly using defroutes.

(defroutes app-routes  
  (GET "/" [] (message))
  (POST "/" params (display-message params))
  (route/resources "/")
  (route/not-found "Not Found"))

Then, we'll write the message and display-message functions and put the logic for the pages in them.

(defn message []
  (html5 
    [:body 
     (form-to [:post "/"]                           
              (text-area {:placeholder "say something..."} "message") 
              [:br]
              (text-field {:placeholder "name"} "id")
              (submit-button "post message"))]))

(defn display-message [params]
  (let [form-params (:form-params params)] 
    (html5 
      [:body 
       [:p (get form-params "id") " says " (get form-params "message")]])))

The Noir template comes with a common namespace which defines a layout macro, which we use to wrap our pages so that we don't have to keep typing in the boilerplate. We can easily write a helper function to do the same thing.

(ns myapp.common
  (:use [hiccup.def :only [defhtml]] 
        [hiccup.page :only [include-css]]))
       
(defhtml layout [& body]  
  [:head
   [:title "Welcome to myapp"]
   (include-css "/css/screen.css")]
  (into [:body] body))

The next difference is that our request map contains the complete request as opposed to just the form params as is the case with the one in defpage.

This means that we have to grab the :form-params key from it to access the form parameters. Another thing to note is that the parameter keys are strings, meaning that we can't destructure them using :keys.

This problem is also easily addressed by a macro which will grab the form-params and keywordize them for us. Note that the original request map will still be available as request in the resulting function.

(defmacro page [f form-params & body]
  `(defn ~f [~'request]
     (let [~form-params 
           (into {} (for [[k# v#] (:form-params ~'request)] 
                      [(keyword k#) v#]))]
       ~@body)))

Now, we can rewrite our app as follows:

(page message []
  (layout
    (form-to [:post "/"]                           
             (text-area {:placeholder "say something..."} "message") 
             [:br]
             (text-field {:placeholder "name"} "id")
             (submit-button "post message"))))

(page display-message {:keys [id message]}
  (layout
      [:p id " says " message]))

(defroutes app-routes  
  (GET "/" [] (message []))
  (POST "/" params (display-message params))
  (route/resources "/")
  (route/not-found "Not Found"))

update

Turns out Compojure already provides the functionality provided by the page macro, and to get the form params, we can destructure them as follows:

(defn display-message [id message]
  (layout [:p id " says " message]))

(defroutes app-routes  
  (POST "/" [id message] (display-message id message))
  (route/not-found "Not Found"))

Big thanks to James Reeves aka weavejester on setting me straight there. :)

This is starting to look very similar to the Noir style apps we're used to. Turns out that migrating from Noir to Compojure is fairly painless.

If you use lib-noir when converting your existing Noir application, then the changes end up being minimal. You can continue using noir.crypt, noir.validation, and etc. as you did before. The only caveat is that you now have to remember to add the appropriate wrappers to your handler, eg:

(-> handler
  (wrap-noir-cookies)
  (session/wrap-noir-session 
    {:store (memory-store session/mem)})
  (wrap-noir-validation))

One thing which Noir provided was a nice batteries included template. I created a similar one called compojure-app.

To use the template you can simply run:

lein new compojure-app myapp

The template sets up a project with a main, which can be compiled into a standalone using lein uberjar or into a deployable WAR using leing ring uberwar. The project is setup to correctly handle loading static resources located in resources/public and correctly handle the servlet context.

When run with lein run the project will pickup the dev dependencies and use the wrap-reload, so that changes to source are picked up automatically in the running app.

This should get all the boiler plate out of the way and let you focus on making your app just as you did with Noir. :)

ClojureScript Adventures

Fri, 26 Oct 2012 00:00:00 +0000

I finally got a chance to play around a bit more with ClojureScript. When I was updating markdown-clj to compile to it, the extent of interaction was to accept a markdown string and return the corresponding HTML.

This time around I decided to dive into doing interop with JavaScript and actual interaction with the page. I wrote a silly Tetris game a while back, and it seemed like a perfect fit for the task.

So, let's see what's involved in porting Clojure to ClojureScript and Canvas. First, I had to separate the pure Clojure code from any code which relies on Java interop. The original code can be seen here.

After, splitting it up, I ended up with a game namespace which contains the bulk of the game logic, and a core namespace containing all the logic pertaining to the UI and input. The split turned out to be fairly painless since I already had the game logic separated from UI in the original design.

Now it's time to add some ClojureScript to the project. First, we need to create a new source folder for the ClojureScript namespaces. In my project I called this folder src-cljs. Then we need some way to compile our script.

The easiest way to do that is to use the lein-cljsbuild plugin. With it you can specify the ClojureScript sources, Clojure namespaces you'd like to reference, and the output Js files to produce.

In my case the project file looks as follows:

(defproject tetris "0.1.0-SNAPSHOT"
  :description "a simple Tetris game"
  :url "https://github.com/yogthos/Clojure-Tetris"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  
  :dependencies [[org.clojure/clojure "1.4.0"]]
  :plugins [[lein-cljsbuild "0.2.9"]]
  
  :source-paths ["src"]
  :main tetris.core

  :cljsbuild {:crossovers [tetris.game]
              :builds
              [{:source-path "src-cljs"
                :compiler
                {:output-to "js/tetris.js"
                 :optimizations :advanced
                 :pretty-print false}}]})

All that's needed to include ClojureScript compilation is to add the lein-cljsbuild in the plugins and specify the options for the cljsbuild. The crossovers section specifies a vector of Clojure namespaces which will be included during compilation.

Once the project file is setup, we have two options for invoking ClojureScript compilation. We can either run lein cljsbuild once or lein cljsbuild auto. When using the auto option, the build will watch for changes in the source and automatically recompile the Js files as needed. This takes much less time than when compiling using the once option, and turns out to be quite handy for development.

The ClojureScript version of the UI, which uses the canvas, can be seen here .

Interacting with JavaScript turns out to be pretty simple and the syntax is similar to Java interop in Clojure. However, there are some differences which are worth mentioning.

Any standard Js functions can be accessed using the js namespace, for example if we want to make a logger which logs to the console we can write something like the following:

(defn log [& items]
  (.log js/console (apply str items)))

This works exactly like the Java interop, where we denote methods by using the . notation and pass in the object as the first argument.

Exporting functions so that they're visible from JavaScript is also quite simple. Instead of denoting them with - as we do when we interop with Java, we use ^:export annotation:

(defn ^:export init []
  (log "Hello ClojureScript!"))

One thing that's not obvious is the interaction with JavaScript object properties. To access these we use (.-property obj) notation. Where - indicates that we're referencing a property and not a function. Writing properties is accomplished by calling the set! function. Here's an example:

(defn ^:export init []
  (let [canvas (.getElementById js/document "canvas")
        ctx    (.getContext canvas "2d")
        width  (.-width canvas) 
        height (.-height canvas)]
 
    (log "width: " width ", height: " height)

    ;;set a property
    (set! (.-fillStyle ctx) "black")

    (.fillRect ctx 0 0 width height)))

Another quirk I ran into is that :use doesn't seem to work in the namespace declaration as it expects a collection.

For example, if you have the following setup:

(ns foo)

(defn bar [])

(ns hello
 (:use foo))

(defn ^:export init []
	(js/alert "Hello from ClojureScript!"))

the compiler throws the following error:

java.lang.UnsupportedOperationException: nth not supported on this type: Symbol

Fortunately, both (:use [foo :only [bar]]) and (:require foo) work as expected.

Finally, to make a timer, it's possible to use js/setTimeout and simply pass it the function to call after the timeout:

(declare game-loop)
(defn game-loop [state]
  (if (not (:game-over state))
    (js/setTimeout
      (fn [] (game-loop (update-state state)))
      10)))

Everything else appeared to work exactly as it does in Clojure itself. The only caveat when porting code is that it cannot contain any Java interop or use libraries which do so. In case of the game, I simply put the game logic into a shared namespace and wrote separate UI logic for both Java and JavaScript versions.

To try out the ClojureScript version simply grab tetris.js and tetris.html which expects the tetris.js file to be in the js folder relative to it.

One thing to note is that ClojureScript is definitely chunky compared to writing JavaScript by hand. The game weighs in at a hefty 100k. That said, it should be noted that jQuery weighs in about that as well, and nobody would claim that it's outrageous for a site to include it.

I feel that the benefits of ClojureScript offers far outweigh the downsides of its size. You get a much nicer language without all the quirks of working in JavaScript, immutability, persistent data structures, and the ability to easily share code between the server and the browser.

The good news is that ClojureScript is under active development, and performance and size are both targets for future improvement. As it stands I find it very usable for many situations.

making reporting easy

Wed, 17 Oct 2012 00:00:00 +0000

There are a bunch of reporting options out there, JasperReports is one popular example. While it's got a ton of features, it often involves a disproportionate amount of effort to create even the simplest of reports. Here's what's involved in simply printing out some fields from the database to a PDF.

Let's see if we can make things easier with Clojure. We'll produce the same report as the one in the linked example.

First, we'll create our database connection using java.jdbc.

(def db {:classname "org.postgresql.Driver"
           :subprotocol "postgresql"
           :subname "//localhost:5432/testdb"
           :user "user"
           :password "secret"})

then we'll make an employee table and populate it with the sample data

(defn create-employee-table []
  (sql/create-table
    :employee
    [:name "varchar(50)"]
    [:occupation "varchar(50)"]
    [:place "varchar(50)"]
    [:country "varchar(50)"]))

(sql/with-connection 
  db
  (create-employee-table)
  (sql/insert-rows 
    :employee
    ["Babji, Chetty", "Engineer", "Nuremberg", "Germany"]
    ["Albert Einstein", "Engineer", "Ulm", "Germany"]
    ["Alfred Hitchcock", "Movie Director", "London", "UK"]
    ["Wernher Von Braun", "Rocket Scientist", "Wyrzysk", "Poland (Previously Germany)"]
    ["Sigmund Freud", "Neurologist", "Pribor", "Czech Republic (Previously Austria)"]
    ["Mahatma Gandhi", "Lawyer", "Gujarat", "India"]
    ["Sachin Tendulkar", "Cricket Player", "Mumbai", "India"]
    ["Michael Schumacher", "F1 Racer", "Cologne", "Germany"]))

finally we'll write a function to read the records from the table.

(defn read-employees []
  (sql/with-connection db 
    (sql/with-query-results rs ["select * from employee"] (doall rs))))

Let's run read-employees to make sure everything is working as expected, we should see something like the following:

(clojure.pprint/pprint (read-employees))

({:country "Germany",
  :place "Nuremberg",
  :occupation "Engineer",
  :name "Babji, Chetty"}
 {:country "Germany",
  :place "Ulm",
  :occupation "Engineer",
  :name "Albert Einstein"}
  ...)

You'll notice that the result is simply a list of maps where the keys are the names of the columns in the table.

We're now ready to generate our report, clj-pdf provides a template macro, which uses $ to create anchors which are populated from the data using the keys of the same name.

The template returns a function which accepts a sequence of maps and applies the supplied template to each element in the sequence. In our case, since we're building a table, the template is simply a vector with the names of the keys for each cell in the row.

(def employee-template 
  (template [$name $occupation $place $country]))

if we pass our data to the template we'll end up with the following:

(employee-template (take 2 (read-employees)))

(["Babji, Chetty" "Engineer" "Nuremberg" "Germany"] 
 ["Albert Einstein" "Engineer" "Ulm" "Germany"])

All that's left is to stick this data into our report:

(pdf
 [{}
  (into [:table 
         {:border false
          :cell-border false
          :header [{:color [0 150 150]} "Name" "Occupation" "Place" "Country"]}]
        (employee-template (read-employees)))]
 "report.pdf")

here's the result of running the above code, which looks just as we'd expect:

It only took a few lines to create the report and we can see and manipulate the layout of the report in one place. Of course, the template we used for this report was completely boring, so let's look at another example. Here we'll output the data in a list, and style each element:

(def employee-template-paragraph 
  (template 
    [:paragraph 
     [:heading $name]
     [:chunk {:style :bold} "occupation: "] $occupation "\n"
     [:chunk {:style :bold} "place: "] $place "\n"
     [:chunk {:style :bold} "country: "] $country
     [:spacer]]))

when writing the report, we can mix the templated elements with regular ones:

(pdf 
  [{:font {:size 11}}      
   [:heading {:size 14} "Employees Test"]
   [:line]
   [:spacer]
   (employee-template-paragraph (read-employees))]
  "report.pdf")

here's the new report with the fancy formatting applied to it:

I think that this approach provides a lot of flexibility while keeping things concise and clear. In my experience there are many situations where all you need is a simple well formatted report, and the effort to create that report should be minimal.

Making services with Liberator

Sun, 09 Sep 2012 00:00:00 +0000

Liberator is a recent Clojure library for writing RESTful services. Its primary feature is that it puts strong emphasis on decoupling the front end and the back end of the application.

Conceptually, Liberator provides a very clean way to reason about your service operations. Each request passes through a series of conditions and handlers defined in the resource. These map to the codes specified by the HTTP rfc, such as 200 - OK, 201 - created, 404 - not found, etc. This makes it very easy to write standards compliant services and to group the operations logically.

While the official site has some fairly decent documentation, I found there were a few areas where I had to dig around and look through the source to figure out what to do.

In this post I'll walk you through the steps to create a simple application which serves static resources, provides basic session management, and JSON operations.

Our application will be structures as follows:

src/liberator_service
         server.clj
         resources.clj
         static_resources.clj
         ui.clj
resources/public
         site.js
project.clj

Our project.clj will look as follows:

(defproject liberator-example "0.1.0-SNAPSHOT"
  :description "Example for the Liberator library"
  :url "https://github.com/yogthos/liberator-example"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.4.0"]
                 [compojure "1.0.2"]
                 [liberator "0.5.0"]
                 [sandbar "0.4.0-SNAPSHOT"]
                 [org.clojure/data.json "0.1.2"]
                 [ring/ring-jetty-adapter "1.1.0"]]
  :dev-dependencies [[lein-ring "0.7.3"]]
  :ring {:handler liberator-service.server/handler}
  :main liberator-service.server)

Now we'll take a look at the service namespace, in it we'll add the required libraries and create an atom to hold the session information.

(ns liberator-service.server
  (:use [liberator.representation :only [wrap-convert-suffix-to-accept-header]]
        [ring.middleware.multipart-params :only [wrap-multipart-params]]
        ring.middleware.session.memory
        sandbar.stateful-session
        compojure.core 
        [compojure.handler :only [api]]
        liberator-service.ui
        liberator-service.resources
        liberator-service.static-resources)
  (:require
   [ring.adapter.jetty :as jetty]))

(defonce my-session (atom {}))

Next we will define the routes which our application responds to. In our case we've defined routes for serving the home page, our services, and static content:

(defn assemble-routes []
  (routes
    (GET   "/" [] home)
    (POST "/login" [] login)
    (POST "/logout" [] logout)
    (GET   "/resources/:resource" [resource] static)))

we'll also need to create a handler for the application:

(defn create-handler []
  (fn [request]
    ((->
       (assemble-routes)
       api
       wrap-multipart-params
       (wrap-stateful-session {:store (memory-store my-session)})
       (wrap-convert-suffix-to-accept-header
         {".html" "text/html"
          ".txt" "text/plain"
          ".xhtml" "application/xhtml+xml"
          ".xml" "application/xml"
          ".json" "application/json"}))
      request)))

The session handling in our handler is provided by the wrap-stateful-session from the sandbar library. The wrap-convert-suffix-to-accept-header is used by the Liberator to decide what types of requests it will accept. Finally, we'll create a main to run our service:

(defn start [options]
  (jetty/run-jetty
   (fn [request]
     ((create-handler) request))
   (assoc options :join? false)))

(defn -main
  ([port]
     (start {:port (Integer/parseInt port)}))
  ([]
     (-main "8000")))

Next let's write a resource which will display a login page:

(ns liberator-service.ui
  (:use hiccup.page
        hiccup.element
        hiccup.form
        sandbar.stateful-session
        [liberator.core :only [defresource]]))

(defresource home
  :available-media-types ["text/html"]
  :available-charsets ["utf-8"]
  :handle-ok (html5 [:head (include-js
                             "http://ajax.googleapis.com/ajax/libs/jquery/1.8.0/jquery.min.js"
                             "/resources/site.js")]
                    [:body
                       [:div#message]
                       [:div#login
                        (text-field "user")
                        (password-field "pass")
                        [:button {:type "button" :onclick "login()"} "login"]]]))

Here we get a glimpse at how Liberator works. We use defresource to define the handler for the home route we specified earlier in our service. The resource specifies what media types it provides as well as the encoding for the content. If the handler is invoked successfully then the :handle-ok handler is called and its output is set as the body of the response. In our site.js we'll define login and logout functions which will use POST to call login and logout operations on the server:

function login() {
	$("#message").text("sending login request");
	$.post("/login", 
	       {user: $("#user").val(), pass: $("#pass").val()}, 
    	       function({window.location.reload(true);},
    	       "json")
         .error( function(xhr, textStatus, errorThrown) {       			 
      			 $("#message").text(textStatus + ": " + xhr.responseText);
   	 });
}

function logout() {
    $.post("/logout", 
           function() {window.location.reload(true);});					  	
}

Since we reference a local JavaScript file, we'll need to create a handler to serve it. We'll create a static-resources namespace for this purpose:

(ns liberator-service.static-resources  
  (:use [liberator.core :only [defresource]]
        [ring.util.mime-type :only [ext-mime-type]])
  (:require [clojure.java.io :as io]))

(let [static-dir  (io/file "resources/public/")]
  (defresource static
    :available-media-types
    #(let [file (get-in % [:request :route-params :resource])]       
       (if-let [mime-type (ext-mime-type file)]
         [mime-type]
         []))

    :exists?
    #(let [file (get-in % [:request :route-params :resource])]       
       (let [f (io/file static-dir file)]
         [(.exists f) {::file f}]))
    
    :handle-ok (fn [{{{file :resource} :route-params} :request}]                 
                 (io/file static-dir file)))

    :last-modified (fn [{{{file :resource} :route-params} :request}]                                                               
                     (.lastModified (io/file static-dir file))))

When our home page requests /resources/site.js, this resource will set the mime type to "text/javascript" based on the extension of the file. It will check if the resource exists and the last modified time, and finally serve the resource in :handle-ok as needed.

Now let's create a resource which the client can call to login and create a session on the server. We'll put it in the resources namespace:

(ns liberator-service.resources
  (:use clojure.data.json                        
        sandbar.stateful-session
        [liberator.core :only [defresource request-method-in]]))

For our testing, we'll simply create a dummy list of users and a helper to check if one matches our login params:

(def users [{:user "foo" 
             :pass "bar"
             :firstname "John"
             :lastname "Doe"}])

(defn valid-user [user]
  (some #(= user (select-keys % [:user :pass])) users))

and now we'll create the login resource itself:

(defresource login
  :available-media-types ["application/json" "text/javascript"]
  :method-allowed? (request-method-in :post)  
  :authorized?     (fn [{{user :params} :request}]                 
                     (or (session-get :user) (valid-user user)))
  
  :post! (fn [{{{:keys [user]} :params} :request :as ctx}]
           (session-put! :user user))
  
  :handle-unauthorized (fn [ctx] (:message ctx))  
  :handle-created      (json-str {:message "login successful"}))

Again, the above is fairly straight forward. We specify the media types the handler responds to, set it to allow the POST request type , check if the supplied user params are valid, and either create the user or return an error based on whether the :authorized? handler succeeds.

As I mentioned above, each handler responds to a specific HTTP code. For example, if :authorized? returns false then the code will be set to 401, which will cause :handle-unauthorized handler to be invoked. If :authorized? it true then the :post! handler gets called, and if it succeeds then subsequently:handle-created. Next we need a logout resource, and it looks as follows:

(defresource logout
  :available-media-types ["application/json" "text/javascript"]
  :method-allowed? (request-method-in :post)  
  :post!           (session-delete-key! :user)
  :handle-created  (json-str {:message "logout successful"}))

You might have noticed that Liberator is pretty flexible regarding what you can supply as the handler. It can either be a callable function, an evaluated expression, or a value.

Now that we have a way for the user to login and logout, let's revisit our UI handler and update it to render different content based on whether there is a user in the session:

(ns liberator-service.ui
  (:use hiccup.page
        hiccup.element
        hiccup.form
        sandbar.stateful-session
        liberator-service.resources
        [liberator.core :only [defresource]]))

(defn get-user []
  (first (filter #(= (session-get :user) (get-in % [:user])) users)))

(def login-page 
  [:body
   [:div#message]
   [:div#login
    (text-field "user")
    (password-field "pass")
    [:button {:type "button" :onclick "login()"} "login"]]])

(defn home-page [] 
  [:body
   (let [{firstname :firstname lastname :lastname} (get-user)] 
     [:div#message (str "Welcome " firstname " " lastname)])
   [:div#logout 
    [:button {:type "button" :onclick "logout()"} "logout"]]])

(defresource home
  :available-media-types ["text/html"]
  :available-charsets ["utf-8"]
  :handle-ok (html5 [:head (include-js
                             "http://ajax.googleapis.com/ajax/libs/jquery/1.8.0/jquery.min.js"
                             "/resources/site.js")]
                    (if (session-get :user) (home-page) login-page)))

That's all there is to it. We have a page which checks if there is a user in the session, if there is then it dsiplays the content of the home-page and if not then the login-page content is displayed. The page interacts with the service by calling login and logout resources via Ajax.

Complete source for the example is available here.

Overall, I definitely think that Liberator makes writing RESTful applications easy and natural. This is a fairly different approach from Noir, where you think in terms of pages and simply implement the UI and the backend portion for each one.

While the Noir approach can easily result in tight coupling between the UI and the backend, the Liberator ensures that we're always thinking in terms of service operations whenever any interaction between the service and the client is happening.

Noir tutorial - part 7

Mon, 03 Sep 2012 00:00:00 +0000

In the last part of the tutorial we saw how we can use a request handler wrapper to fix the redirect URLs. There is another option for doing this that I'd like to mention.

As we've seen, the defpage params only contain form parameters, but there is a way to access the complete parameter map provided by ring using the noir.request/ring-request helper.

If the application is running on a servlet its context will show up in this map and we can use it in our redirects. We can write a simple macro called local-redirect which will do this for us:

(defmacro local-redirect [url]
  `(noir.response/redirect 
     (if-let [context# (:context (noir.request/ring-request))]
       (str context# ~url) ~url)))

The advantage to this approach is that we do not try to infer if the redirect is supposed to be local or not. If we want to redirect to the local servlet context we can do it explicitly, and if we wish to do an absolute redirect then we can use the noir.response/redirect directly.

With that out of the way, I'd like to cover using the servlet init function and accessing files located on the classpath of the servlet. This allows us to run a function once when our serlvet starts up.

For example, we might want to read in a configuration file and setup some environment parameters based on it. To do that we'll open up our project.clj and add an :init key to our map or ring parameters:

(defproject my-website "0.1.0-SNAPSHOT"
            :description "my Noir website"
            :dependencies [[org.clojure/clojure "1.4.0"]
                           [noir "1.3.0-beta3"]
                           [org.clojure/java.jdbc "0.2.3"]
                           [postgresql/postgresql "9.1-901.jdbc4"]
                           [joda-time "2.0"]]
            :dev-dependencies [[lein-ring "0.7.3"]]
            :ring {:handler my-website.server/handler

                   ;;initialization function which will be run 
                   ;;once when the servlet is loaded
                   :init my-website.config/init-config}
            :main my-website.server)

update: with Leiningen 2.0 you will need to use :plugins instead of :dev-dependencies to get lein-ring to work correctly:

(defproject my-website "0.1.0-SNAPSHOT"
            :description "my Noir website"
            :dependencies [[org.clojure/clojure "1.4.0"]
                           [noir "1.3.0-beta3"]
                           [org.clojure/java.jdbc "0.2.3"]
                           [postgresql/postgresql "9.1-901.jdbc4"]
                           [joda-time "2.0"]]

            ;;lein 2
            :plugins [[lein-ring "0.7.5"]]
            ;;lein 1
            :dev-dependencies [[lein-ring "0.7.3"]]

            :ring {:handler my-website.server/handler

                   ;;initialization function which will be run 
                   ;;once when the servlet is loaded
                   :init my-website.config/init-config}
            :main my-website.server)

Now we'll create a new namespace which the :init key is pointing to, and create an init-config function in it:

(ns my-website.config
  (:use clojure.java.io))

(defn init-config [] 
  (println "servlet has been initialized"))

If you build and deploy the application, the "servlet has been initialized" message is printed in the server log once after deployment. Now, let's add a configuration file in our resources folder:

touch my_webapp/resources/site.config

When we run lein ring uberwar this file will be packaged under /WEB-INF/classes/ path in the servlet. To access this file we'll need to add the following function to our config namespace:

(def config-file "site.config")

(defn load-config-file []
  (let [url (.. 
              (Thread/currentThread) 
              getContextClassLoader 
              (findResource config-file))] 
    (if (or (nil? url) 
            (.. url 
              getPath 
              (endsWith (str "jar!/" config-file))))
      (doto (new java.io.File config-file) 
        (.createNewFile))
      url)))

The load-config-file function will get the context class loader and attempt to find the resource by name. If the resource is found we will get back a URL pointing to it. Unfortunately, if we're running as a standalone jar, we cannot modify the resource inside it. So, in case the URL is nil, meaning that the file was not found, or if it ends with "jar!/site.config" we will create a new file instead. When running standalone, the file will be created in the same folder as the jar.

Now that we have a function to read the configuration, let's load it so we can actually use it. To do that we will add an atom to hold the configuration, and update our init-config function as follows:

(def app-config (atom nil))

(defn init-config []
  (with-open
    [r (java.io.PushbackReader. (reader (load-config-file)))]
    (if-let [config (read r nil nil)]
      (reset! app-config config)))
  (println "servlet has been initialized"))

In our log-stats namespace the path to the logs is currently hard coded. Let's change it to read the path from our config file. We'll open our resources/config and add the following to it:

{:log-path "logs/"}

Then in our log-stats namespace we'll change all references to "logs/" to (:log-path @app-config) instead:

(ns my-website.views.log-stats
  ...
  (:use ... my-website.config))

(defpage [:post "/get-logs"] params  
  (response/json 
    (hits-per-second 
      (read-logs (last-log (:log-path @app-config))))))

To ensure that the application still runs correctly standalone we will have to call init-config in our -main in the server namespace:

(ns my-website.server
  (:use my-website.config)
  ...)

(defn -main [& m]
  (let [mode (keyword (or (first m) :dev))
        port (Integer. (get (System/getenv) "PORT" "8080"))]
    (init-config)
    (server/start port {:mode mode
                        :ns 'my-website})))

Now the log path can be specified in the config file without having to rebuild and redeploy the application each time. Complete source for this section is available here.

Noir tutorial - part 6

Sun, 02 Sep 2012 00:00:00 +0000

In the first part of the tutorial we've already seen how to run our application in standalone mode. Here we will look at what we need to do to deploy it on an application server such as Glassfish, Tomcat, jBoss, or Immutant which is a modification of jBoss geared specifically towards Clojure.

There are numerous reasons as to why you might want to do this. For example, an application server lets you run multiple applications at the same time. Another advantage is that the application server can take care of the configuration details, such as handling database connections.

When building real world applications, you will likely have separate dev/staging/prod configurations. Instead of having different builds for our application, we can instead configure our application servers appropriately for each environment. Then we can have a single build process which is much less error prone in my opinion. We can also configure CI, such as Jenkins to build our application and automatically deploy it to the server ensuring that we always have the latest code running.

Finally, if you plan on using a hosting provider, you may end up deploying on a shared application server as opposed to being able to run your application standalone.

Let's go over the prerequisites for building our application into a WAR and deploying it to a server. You will need to setup an application server of your choice for this section. I will be using Tomcat, but the steps will be similar for other servers as well. If you will be using Tomcat, then download the latest version. To start up the server you simply unpack the archive, navigate to the resulting directory, and run:

chmod +x bin/catalina.sh
bin/catalina.sh start
Using CATALINA_BASE:   apache-tomcat-7.0.29
Using CATALINA_HOME:   apache-tomcat-7.0.29
Using CATALINA_TMPDIR: apache-tomcat-7.0.29/temp
Using JRE_HOME:        /Library/Java/Home
Using CLASSPATH:       apache-tomcat-7.0.29/bin/bootstrap.jar:apache-tomcat-7.0.29/bin/tomcat-juli.jar

Your Tomcat should now be up and running and you can test it by navigating to localhost:8080:

With the application server running, let's prepare our application for deployment. First, we must ensure that the server namespace requires all the namespaces in our views package, and has the gen-class directive specified:

(ns my-website.server
  (:require [noir.server :as server]
            [my-website.views 
             common
             files
             log-stats
             users
             welcome])     
  (:gen-class))

This will ensure that the server and the views are compiled during the build step, which is needed for them to be picked up by the application server when the application is deployed. Next, we will change server/load-views call to server/load-vews-ns:

(server/load-views-ns 'my-website.views)

If you used Leiningen 2 to create the project template, then load-views-ns should already be set correctly.

Finally, we have to add a handler which will be used instead of the -main when running on the application server:

(def base-handler 
  (server/gen-handler 
    {:mode :prod, 
     :ns 'my-website 
     :session-cookie-attrs {:max-age 1800000}}))

It is possible to chain different handlers together. In our case, we will need a wrapper for our handler to prepend the servlet context to all the requests coming to our servlet. This is a workaround for a bug in the current version of Noir, which ignores it. Without this fix none of the redirects will work correctly as they will be routed to the base application server URL instead.

Each wrapper is a function which accepts the current handler, and returns a function which accepts a request, does something to it, and then return the result of calling the handler on it. The result is in turn a handler itself, so we can chain as many wrappers together as we like. In our case we will override the resolve-url function in noir.options with one of our own making:

(defn fix-base-url [handler]
  (fn [request]    
    (with-redefs [noir.options/resolve-url 
                  (fn [url]                    
                    ;prepend context to the relative URLs
                    (if (.contains url "://") 
                      url (str (:context request) url)))]
      (handler request))))

Above, we will check that the URL contains "://", if not then we treat it as a local URL and prepend the servlet context to it. Now we have to hook it up with our initial handler to produce the final request handler for our servlet:

(def handler (-> base-handler fix-base-url))

Now that we've created our handler, we need to point our project.clj to it:

(defproject my-website "0.1.0-SNAPSHOT"
            :description "my Noir website"
            :dependencies [[org.clojure/clojure "1.4.0"]
                           [noir "1.3.0-beta3"]
                           [org.clojure/java.jdbc "0.2.3"]
                           [postgresql/postgresql "9.1-901.jdbc4"]]
            :dev-dependencies [[lein-ring "0.7.3"]]
            :ring {:handler my-website.server/handler}
            :main my-website.server)

We've also added lein-ring plugin to our dev-dependencies, this is required for generating the WAR artifact from our build. Under the :ring key we set the :handler to the one we defined above.

Let's test that our project builds correctly and produces a working WAR by running the following commands from the temrinal:

lein deps
Copying 29 files to Noir-tutorial/lib
[INFO] snapshot thneed:thneed:1.0.0-SNAPSHOT: checking for updates from clojars
[INFO] snapshot thneed:thneed:1.0.0-SNAPSHOT: checking for updates from central
Copying 5 files to Noir-tutorial/lib/dev

lein ring uberwar
Compiling my-website.server
Compilation succeeded.
Created Noir-tutorial/my-website-0.1.0-SNAPSHOT-standalone.war

If we have our application server running, then we should be able to simply drop this WAR in its deployment folder and the server will take care of the rest. If we're using Tomcat, then we have to copy it to the webapps folder:

cp my-website-0.1.0-SNAPSHOT-standalone.war ../apache-tomcat-7/webapps/my-website.war

Make sure to replace the ../apache-tomcat-7 above with the location of your Tomcat server. We can now take a look at our server log and see that the application was deployed successfully:

tail -f logs/catalina.out
...
INFO: Deploying web application archive apache-tomcat-7.0.29/webapps/my-website.war

Now let's navigate to localhost:8080/my-website and we should see our application running:

One last thing to note is that any Ajax calls in our pages will have to use the servlet context to be resolved correctly. A workaround for this issue is to use noir.request/ring-request to check if a context is present and set it as a hidden field on the page:

(ns my-website.views.log-stats
  (:require [my-website.views.common :as common]
            [noir.request :as request]
            [noir.response :as response])
  (:use clojure.java.io hiccup.page hiccup.form noir.core)
  (:import java.text.SimpleDateFormat))

(defpage "/access-chart" []
  (common/basic-layout
    (include-js "/js/site.js")
    (hidden-field "context" (:context (request/ring-request)))
    [:div#hits-by-time "loading..."]))

Then we can check this value and prepend it to the URL when making our Ajax query:

$(document).ready(function(){	
    var context = $('#context').val();
    var url = '/get-logs';
    if (context) url = context + url;
    var options = {xaxis: { mode: "time", 
                            minTickSize: [1, "minute"]}};
	$.post(url, function(data){
	    $.plot($("#hits-by-time"), [data], options);
	    });		
});

As usual, the complete code for this section is available here.

Noir tutorial - part 5

Sat, 01 Sep 2012 00:00:00 +0000

In this section we will learn how to add some JavaScript to the application and how to use Ajax to query the service. We'll use the flot jQuery library to display the usage statistics for our site. When the page loads it will call the service which will parse today's access log and return a JSON response which will be used to generate the chart.

First, let's generate some sample usage data in the apache combined log format:

(defn gen-log-line [[cur-time]] 
  (let [new-time (doto (new java.util.Date) (.setTime (+ (.getTime cur-time) (rand-int 5000))))
        browsers ["\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:15.0) Gecko/20100101 Firefox/15.0\""
                  "\"Mozilla/5.0 (Linux; U; Android 2.2; en-gb; LG-P500 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko)\""
                  "\"Mozilla/5.0 (X11; Linux i686) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11\""]]
    [new-time
     (->>
       (concat
         (interpose "." (take 4 (repeatedly #(rand-int 255))))
         [" - - [" (.format (new java.text.SimpleDateFormat 
                                  "dd/MMM/YYYY:HH:mm:ss ZZZZ") new-time) "]"]
         [" \"GET /files/test.jpg\" " 200 " " (rand-int 4000) " \"http://my-website/files/test.jpg\" " (first (shuffle browsers))])
       (apply str))]))
 
(defn gen-test-logs [size]
  (->> (gen-log-line [(new java.util.Date)])
    (iterate gen-log-line)
    (take size)
    (map second)
    (interpose "\n")
    (apply str)))

(spit "test-log.txt" (gen-test-logs 500))

If you run the above in the REPL, you will end up with test-log.txt file which should have the contents that look roughly like the following:

120.138.220.117 - - [31/Aug/2012:21:06:47 -0400] "GET /files/test.jpg" 200 3989 "http://my-website/files/test.jpg" "Mozilla/5.0 (Linux; U; Android 2.2; en-gb; LG-P500 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko)"
201.59.151.159 - - [31/Aug/2012:21:06:49 -0400] "GET /files/test.jpg" 200 1729 "http://my-website/files/test.jpg" "Mozilla/5.0 (Linux; U; Android 2.2; en-gb; LG-P500 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko)"
122.39.249.88 - - [31/Aug/2012:21:06:51 -0400] "GET /files/test.jpg" 200 1650 "http://my-website/files/test.jpg" "Mozilla/5.0 (Linux; U; Android 2.2; en-gb; LG-P500 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko)"
...

Now that we have a log file with some access logs in it, we'll parse those logs into structured data to make them easier to analyze:

(defn round-ms-down-to-nearest-sec [date]
  (let [date (.parse 
               (new SimpleDateFormat 
                    "dd/MMM/yyyy:HH:mm:ss zzzzz") 
               date)] 
    ( * 1000 (quot (.getTime date) 1000))))

(defn parse-line [line]
  {:ip (re-find #"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b" line) 
   :access-time (round-ms-down-to-nearest-sec 
                  (second (re-find #"\[(.*?)\]" line))) })

(defn read-logs [file] 
  (with-open [rdr (reader file)] 
    (doall (map parse-line (line-seq rdr)))))

Above, we simply return a map containing the ip and the access-time for each line in the logs. Using this map we can aggregate the logs by IP to get unique hits, and then group them by time to see hits per second:

(defn hits-per-second [logs]
  (->> logs 
    (group-by :ip)
    (map #(first (second %)))
    (group-by :access-time)    
    (map (fn [[t hits]] [t (count hits)]))
    (sort-by first)))

(hits-per-second (read-logs "test-log.txt"))
=>([1346460948000 2] [1346460949000 1] [1346460954000 1] ...)

We now have a list where each element has a time rounded down to the nearest second with a number of unique hits associated with it. This happens to be the exact format that flot time series is expecting. We can serve the this data as JSON by using noir.response/json:

(defpage [:post "/get-logs"] params
  (response/json (hits-per-second (read-logs "test-log.txt"))))

Finally, we will have to create the page with a placeholder where our chart will be displayed and reference a Js file which will create shortly:

(defpage "/access-chart" []
  (common/basic-layout
    (include-js "/js/site.js")
    [:div#hits-by-time "loading..."]))

We will also have to add the CSS to set the height and width of the chart as well as the margin:

#hits-by-time {
        margin: 25px;
	width: 400px;
	height: 400px;
}

All that's left to do is to add the JavaScript which will get the stats and display them. To do that we'll have to download flot, and put jquery.flot.min.js in the resources/public/js folder.

Then we will include it and jQuery in the header of our page. This can be done using include-js from Hiccup. We'll open up our common namespace and modify the basic-layout as follows:

(defpartial basic-layout [& content]
  (html5
    [:head
     [:title "my-website"]
     (include-css "/css/reset.css")
     (include-js "http://code.jquery.com/jquery-1.7.2.min.js"
                 "/js/jquery.flot.min.js"
                 "/js/site.js")]
    [:body content]))

Now let's create a site.js file in resources/public/js and add the following to it:

$(document).ready(function(){	
    var options = {xaxis: {mode: "time", 
                           minTickSize: [1, "minute"]}};
	$.post('/get-logs', function(data){
	    $.plot($("#hits-by-time"), [data], options);
	    });		
});

If all went well, then when we start up our site and browse to localhost:8080/access-chart. we should see something like this:

Finally, here's some fun daily stats for the blog generated using the above approach. The sources for this part of the tutorial can be found here.

Noir tutorial - part 4

Sat, 25 Aug 2012 00:00:00 +0000

Securing Pages

This part of the tutorial will focus on controlling page visibility, form validation, and handling complex form parameters. In the last section we added support for uploading files, it would make sense to make the upload page private. This way only registered users can access it.

Noir provides a pre-route macro for handling this. However, we will not be using it for a couple of reasons.

First, there is currently a bug in Noir, where pre-route ignores the servlet context, meaning that unless our application is deployed to "/" the routing will not work as expected. The second reason is that you have to remember to add a pre-route entry for each page that you want to make private.

A better solution, in my opinion, is to simply write a macro which will behave the same way as defpage, but will check if there is a user in session and redirect to "/" otherwise. With this approach we make pages private right in their definition. Let's open up our common namespace and add the macro:

(defmacro private-page [path params & content]
  `(noir.core/defpage 
     ~path 
     ~params 
     (if (session/get :user) 
      (do ~@content) 
      (resp/redirect "/"))))

As you can see it has exactly same signature as defpage and calls it internally as you normally would, but only adds the content if the session contains a user, otherwise the page will redirect to "/".

Now, we'll go to our files namespace and mark all the pages as private:

(common/private-page "/upload" {:keys [info]}
  ...)

(common/private-page [:post "/upload"] {:keys [file]}
  ...)

(common/private-page "/files/:name" {:keys [name]} 
  ...)

Let's test that it works by navigating to localhost:8080/upload without logging in. We should be redirected right back to "/".

Site navigation

Since we now have a couple of pages that we will be navigating we can add a navigation menu in our common namespace:

(defn menu []
  [:div.menu
   [:ul
    [:li (form-to [:post "/logout"] (submit-button "logout"))]    
    [:li (link-to "/upload" "my files")]
    [:li (link-to "/" "home")]]])

(defpartial layout [& content]
  (html5
    [:head
     [:title "my-website"]
     (include-css "/css/reset.css")]
    [:body               
     (if-let [user (session/get :user)]
       [:div
        (menu)
        [:h2 "welcome " user]]
       [:div.login
        (login-form) [:p "or"] (link-to "/signup" "sign up")])     
     content]))

Now, if a user logs in, they will see the navigation menu and can either select home or their files page. To keep things clean we'll also move the logout link into our menu. This is functional, but it's rather ugly, so let's add some CSS to make it a bit nicer. We'll open up our stock resources/public/css/reset.css which was generated for our site and add the following to it:

.menu ul {
    list-style: none;
    margin: 0;
    padding-left: 0;
}
.menu li {
    float: right;
    position: relative;
    margin-right: 20px;
}

Things should look much better now:

Input Validation

Next, let's reexamine our sign up page, previously we didn't bother doing any validation when creating a new user, so let's add some now. Noir provides a simple way to validate input fields via the noir.validation namespace. Let's open the users namespace and add it in:

(ns my-website.views.users
  (:use [noir.core]
        hiccup.core hiccup.form)
  (:require [my-website.views.common :as common]
            [my-website.models.db :as db]
            [noir.util.crypt :as crypt]
            [noir.session :as session]
            [noir.response :as resp]
            [noir.validation :as vali]))

Next we will create our validation function:

(defn valid? [{:keys [handle pass pass1]}]
  (vali/rule (vali/has-value? handle)
             [:handle "user ID is required"])
  (vali/rule (vali/min-length? pass 5)
             [:pass "password must be at least 5 characters"])  
  (vali/rule (= pass pass1)
             [:pass "entered passwords do not match"])
  (not (vali/errors? :handle :pass :pass1)))

The function will check that all the fields conform to the rules, such as user id being provided, minimum password length, and that retyped password matches the original. The rules have the following form:

(rule validator [:filed-name "error message"])

where the validator must return a boolean. We'll also need a helper for displaying the error on the page:

(defpartial error-item [[first-error]]
  [:p.error first-error])

Next we will update our signup page to show the errors generated by the validator:

(defpage "/signup" {:keys [handle error]}
  (common/layout
    [:div.error error]
    (form-to [:post "/signup"]
             (vali/on-error :handle error-item)
             (label "user-id" "user id")
             (text-field "handle" handle)
             [:br]
             (vali/on-error :pass error-item)
             (label "pass" "password")
             (password-field "pass")             
             [:br]
             (vali/on-error :pass1 error-item)
             (label "pass1" "retype password")
             (password-field "pass1")             
             [:br]
             (submit-button "create account"))))

All we have to do here is add on-error statements for each field we're validating. Finally, we'll have to update the POST part of the page, to call the validator and return the errors:

(defpage [:post "/signup"] user
  (if (valid? user)
    (try 
      (db/add-user (update-in (dissoc user :pass1) [:pass] crypt/encrypt))
      (resp/redirect "/")
      (catch Exception ex
        (render "/signup" (assoc user :error (.getMessage ex)))))
    (render "/signup" user)))

This should give you a basic idea of how to validate input using Noir, and more details about validation can be found on the official site.

One thing you'll notice that when we navigate to the signup page, we still see the login option as well as the link to sign up:

This is because our layout adds these to every page. We can fix this by splitting layout in the common namespace as follows:

(defpartial basic-layout [& content]
  (html5
    [:head
     [:title "my-website"]
     (include-css "/css/reset.css")]
    [:body content]))

(defpartial layout [& content]  
  (basic-layout 
    [:div
     (if-let [user (session/get :user)]      
       [:div
        (menu)
        [:h2 "welcome " user]]
       [:div
        [:div.login
         (login-form) 
         [:p "or"] 
         (link-to "/signup" "sign up")]])
     content]))

Then we simply change:

(defpage "/signup" {:keys [handle error]}
  (common/layout
  ...)

(defpage "/signup" {:keys [handle error]}
  (common/basic-layout
  ...)

Another clean up item is to make our form items aligned, to do that we can use the following bit of CSS:

label {
    margin-left: 10px;
    width:120px; 
    float:left;
}

The sign up page should now look as follows:

Complex Form Items

Now that we've cleaned up our singup page, we'll turn our attention back to file management. We'll add the ability for the user to filter files by their type. To do that we will first create a function in our db namespace to get all the file types from our database:

(defn file-types []
  (map :type (db-read "select distinct type from file")))

Then in our files namespace we will create a new helper called select-files-by-type:

(defn select-files-by-type []  
  (let [file-types (db/file-types)] 
    (form-to [:post "/show-files"]
             "select file types to show"
             (into 
               (with-group "file-types")
               (for [type file-types]
                 [:div 
                  type
                 (check-box type)]))
             (submit-button "show files"))))

which we will add to our "/upload" page:

(common/private-page "/upload" {:keys [info]}
  (common/layout       
    [:h2.info info]
    (select-files-by-type)
    (list-files)
    (form-to {:enctype "multipart/form-data"}
             [:post "/upload"]
             (label :file "File to upload")
             (file-upload :file)
             [:br]
             (submit-button "upload"))))

This function will read the file types from the database and create a checkbox group from them. When we hit submit we'll see something like the following in our params:

{"image/png" "true", "image/jpeg" "true"}

Where the value of each selected checkbox will appear as a key in the params map with the value of "true". We will now have to update our list-files function to accept optional file type restriction and in turn pass it to list-files in db namespace:

(defn list-files [& [types]]  
  (into [:ul]
        (for [name (db/list-files types)]             
          [:li.file-link (link-to (str "/files/" name) name) 
           [:span "  "] 
           [:div.file]])))

The following changes will have to be made to retrieve files based on type:

(defn params-query [params]
  (apply str (interpose ", " (repeat (count params) "?"))))

(defn list-files [& [types]]
  (map :name 
       (if types
         (apply (partial db-read (str "select name from file where type in (" (params-query types) ")")) types)
         (db-read "select name from file"))))

The params-query helper will create an appropriate WHERE clause based on the number of types we pass in, and list-files will now check if types have been passed in and create the appropriate query. Finally, we'll add a new page which will display the selected files:

(common/private-page [:post "/show-files"] params
  (let [file-types (keys params)] 
    (common/layout 
      [:h2 "showing files types " 
       (apply str (interpose ", " file-types))]
      (list-files file-types)
      (link-to "/upload" "back"))))

The "/upload" page should now look as follows:

When we select some files and hit "show files" button we should see our new "show-files" page:

Summary

In this section we covered the following topics:

restricting access to pages
creating a navigation menu
input validation
handling inputs from multi part items such as check boxes

The complete code for this section is available here.

Noir tutorial - part 3

Wed, 22 Aug 2012 00:00:00 +0000

Last time we created a database to store users, and created pages allowing users to create new accounts and login. This time we'll look at how we can allow users to upload files to the server and how to serve them back using the proper content type. To make things easy, we'll stick our files in the database, so let's design a table to hold them:

(defn create-file-table []
  (sql/with-connection 
    db
    (sql/create-table
      :file
      [:type "varchar(50)"]
      [:name "varchar(50)"]
      [:data "bytea"])))

if we run the above in the REPL a file table should be created. We'll now need a few helper functions to read the list of files and add new files to the table:

(defn to-byte-array [f]  
  (with-open [input (new java.io.FileInputStream f)
              buffer (new java.io.ByteArrayOutputStream)]
    (clojure.java.io/copy input buffer)
    (.toByteArray buffer)))

(defn store-file [{:keys [tempfile filename content-type]}]
  (sql/with-connection 
    db
    (sql/update-or-insert-values
      :file
      ["name=?" filename]
      {:type content-type 
       :name filename
       :data (to-byte-array tempfile)})))

(defn list-files []
  (map :name (db-read "select name from file")))

(defn get-file [name]
  (first (db-read "select * from file where name=?" name)))

The first helper is used by store-file to copy the file out of the input stream into a byte array and then store it in the table. The other two functions simply read the file columns from our database.

Uploading Files

We'll create a new namespace called files under views, and make a page facilitating the uploads:

(ns my-website.views.files
  (:use hiccup.util
        noir.core
        hiccup.core
        hiccup.page
        hiccup.form
        hiccup.element)
  (:require [my-website.views.common :as common]
            [my-website.models.db :as db]
            [noir.response :as resp]))

(defpage "/upload" {:keys [info]}
  (common/layout    
    [:h2.info info]
    (form-to {:enctype "multipart/form-data"}
             [:post "/upload"]
             (label :file "File to upload")
             (file-upload :file)
             [:br]
             (submit-button "upload"))))

There shouldn't be anything too surprising here, we create an "/upload" page with a an info header and a form. On the form we set enctype to multipart/form-data, then we use file-upload function from hiccup.form to create the file upload dialog and add a submit button. As a note, all Hiccup helper functions also accept a map of attributes as an optional first parameter, these attributes will be merged with the ones already provided by the helper.

Now we'll have to make its POST counterpart to handle the upload request on the server:

(defpage [:post "/upload"] {:keys [file]}
  (render "/upload"
    {:info 
      (try
        (db/store-file file) 
        "file uploaded successfully"
        (catch Exception ex
          (do
            (.printStackTrace ex)
            (str "An error has occured while uploading the file: "
              (.getMessage ex)))))}))

Here we accept the params, grab the file and pass it to store-file function we created earlier in the db namespace. The file is a map containing the following keys:

:tempfile - the file itself
:filename - the name of the file being uploaded
:content-type - the content type of the file being uploaded
:size - size of the file in bytes

eg:

{:size 422668, 
 :tempfile #<File /var/folders/0s/1vrmt9wx6lqdjlg1qtgc34600000gn/T/ring-multipart-3157719234459115704.tmp>, 
 :content-type "image/jpeg", 
 :filename "logo.jpg"}

We can now test that file uploading works correctly by navigating to localhost:8080/upload and uploading a file.

once we hit upload we should see the following:

Serving Files

At this point it might be nice to be able to see what files we have on the server, so let's update our "/upload" page to display a list of files and allow downloading them:

(defn list-files []
  (into [:ul]
        (for [name (db/list-files)]             
          [:li.file-link (link-to (str "/files/" name) name) 
           [:span "  "] 
           [:div.file]])))

(defpage "/upload" {:keys [info]}
  (common/layout    
    [:h2.info info]
    (list-files)
    (form-to {:enctype "multipart/form-data"}
             [:post "/upload"]
             (label :file "File to upload")
             (file-upload :file)
             [:br]
             (submit-button "upload"))))

(defpage "/files/:name" {:keys [name]}
  (let [{:keys [name type data]} (db/get-file name)]
    (resp/content-type type (new java.io.ByteArrayInputStream data))))

Above, list-files reads the file names from the database, using the helper function we defined earlier and then sticks them into an unordered list. Notice, that Hiccup allows literal notation for any HTML tags, the syntax is as follows:

[:tag {:attr "value"} content]

So, if we don't have a helper function for a particular tag, or we need to make a custom tag, we can always just make a vector and set the attributes we care about.

The new "/files/:name" page we defined uses resp/content-type function to set the appropriate content type when returning the file. It accepts the content type string and an input stream as parameters.

If we reload the page after making the above changes we should see the following:

and when we click on the file link it should display the image in the browser:

Summary

In this section we learned the following:

storing files in the database
setting custom attributes on Hiccup elements
using multipart/form-data form to upload a binary file
serving a file with a custom content type

The complete code for this section is available here.

In the next section we'll talk about creating private pages, form input validation, and handling multi-select form parameters, such as multi checkbox set.

Reflecting on performance

Tue, 21 Aug 2012 00:00:00 +0000

I'm going to take a short break from Noir tutorials and do a post on optimization and profiling instead. I was playing around with rendering Metaballs and I stumbled on a neat visualization.

To get the above effect we simply calculate the influence of each metaball on each point on the screen based on the distances to each ball's center and radius:

i_mp = r_mp / √ (δx_mp² + δy_mp²)

where r is the radius of the metabll and δx_mp and δy_mp are the x and y distances from the center of the metaball m to the point p. The resulting color c_p of the point is the sum of all the influences:

c_p = Σ i_mp

The algorithm is on the order of n², given a small number of metaballs relative to the number of pixels and a square viewport. Unsurprisingly, this runs quite slowly without optimizations, so let's see if we can do anything about that.

The code to accomplish this is as follows:

;;compute influence of each metaball
(defn influence [{:keys [x y radius]} px py]
  (let [dx (- x px)
        dy (- y py)]
    (/ radius (Math/sqrt (+ (* dx dx) (* dy dy))))))

;;compute the resulting r g b values based on influence
(defn compute-color [x y [red-cur green-cur blue-cur] ball]   
  (let [influence (influence ball x y)
        [r g b] (:color ball)] 
    [(+ red-cur (* influence r))
     (+ green-cur (* influence g))
     (+ blue-cur (* influence b))]))

...
;;reverse direction if we hit the edge of the screen
(defn direction [p v]
  (if (or (> p SIZE) (neg? p)) (- v) v))

;;compute the position and velocity of the ball
(defn move [{:keys [x y vx vy radius color]}]
  (let [vx (direction x vx)
        vy (direction y vy)]
    {:x (+ x vx)
     :y (+ y vy)
     :vx vx
     :vy vy
     :radius radius
     :color color}))

;;for each x,y coordinate compute the color
(reduce (partial compute-color x y) [0 0 0] balls)

;;run this in a loop where we move the 
;;balls around and render them
(loop [balls (take 2 (repeatedly metaball))]      
      (draw canvas balls)
      (recur (map move balls)))

complete code can be seen here

First thing to do is to time our our loop:

(loop [balls (take 2 (repeatedly metaball))]      
      (time (draw canvas balls))
      (recur (time (map move balls))))

"Elapsed time: 250.345 msecs"
"Elapsed time: 0.004 msecs"
"Elapsed time: 171.136 msecs"
"Elapsed time: 0.005 msecs"
"Elapsed time: 212.646 msecs"
"Elapsed time: 0.004 msecs"

As can be expected the draw function eclipses the move function. So we'll focus on what's happening in our rendering code and see where the CPU time is being spent. Instead of guessing, let's profile the application using VisualVM, which should already be bundled with your JVM, and see what's happening.

We can see that the vast majority of the CPU time is being spent in the color function, and that reflection is the culprit. So, let's see why reflection is happening by setting the *warn-on-reflection* flag to true:

(set! *warn-on-reflection* true)

Reflection warning, metaballs.clj:32 - call to java.awt.Color ctor can't be resolved.
Reflection warning, metaballs.clj:40 - call to setColor can't be resolved.
Reflection warning, metaballs.clj:40 - call to fillRect can't be resolved.
Reflection warning, metaballs.clj:52 - reference to field getBufferStrategy can't be resolved.
Reflection warning, metaballs.clj:53 - reference to field getDrawGraphics can't be resolved.
Reflection warning, metaballs.clj:64 - reference to field dispose can't be resolved.
Reflection warning, metaballs.clj:65 - reference to field contentsLost can't be resolved.
Reflection warning, metaballs.clj:66 - reference to field show can't be resolved.

Now we know precisely which spots are causing us trouble. Let's see if adding some annotations will improve things. First warning we hit happens when we create a new instance of Color:

(defn color-in-range [c]
  (int
    (cond 
      (< c 0) 0
      (> c 255) 255
      :default c)))

(defn color [r g b]
  (new Color (color-in-range r) (color-in-range g) (color-in-range b)))

what's happening here is that even though we cast the result into int inside color-in-range, color is not aware of it and uses reflection to resolve the constructor for Color. So we should be doing the cast inside color instead:

(defn color [r g b]
  (new Color (int (color-in-range r)) 
             (int (color-in-range g)) 
             (int (color-in-range b))))

The rest of the warnings simply require annotations for the classes in the function arguments:

(defn paint-square [g color x y size]
  (doto g
    (.setColor color)
    (.fillRect x y size size)))

becomes

(defn paint-square [^Graphics g ^Color color x y size]
  (doto g
    (.setColor color)
    (.fillRect x y size size)))

and so on. Finally, we'll cast our distances to doubles when we compute the influence:

(defn influence [{:keys [x y radius]} px py]
  (let [dx (double (- x px))
        dy (double (- y py))]
    (double (/ radius (Math/sqrt (+ (* dx dx) (* dy dy)))))))

optimized version can be seen here

Now that we've annotated our code let's see if performance is any better:

"Elapsed time: 55.424 msecs"
"Elapsed time: 55.399 msecs"
"Elapsed time: 55.373 msecs"
"Elapsed time: 55.482 msecs"

Indeed it is, we went from ~200ms to ~55ms a 4X improvement in speed! Let's see what the profiler has to say now:

From here we can clearly see that majority of the time is spent in the paint-square function, meaning that our code performs as it should. Turns out the the only real factor on performance is reflection.

We could've spent time doing random optimizations here and there, but it's clear from profiling which functions are actually eating up the resources and need optimizing. While this is a toy project, the technique is equally effective for large projects where it might be much more difficult to guess which functions need tuning.

P.S. try setting a negative radius for some of the metaballs in the scene :P

Noir tutorial - part 2

Sat, 18 Aug 2012 00:00:00 +0000

This is the second part of the Noir tutorial, where we'll continue to cover the basics of building a website. In the comments for part 1, somebody suggested that Noir might be abandoned. This is absolutely not the case, I've contacted Chris Granger and this is what he has to say:

Hey Dmitri,

Light Table actually uses Noir, so it's certainly still alive. I'm not the primary one driving things day to day right now, Raynes has been helping out with that.

Cheers,

Chris.

Hopefully, this should put any fears regarding the health of the project to rest. And with that out of the way, lets continue building our site. In the previous section of the tutorial we setup a basic project and learned how to add pages to it. This time let's look at how to persist data to a database, create sessions, and do some basic user management.

Database Access

There are several Clojure libraries for dealing with relational databases, such as SQLKorma, ClojureQL, Lobos, and [clojure.data.jdbc])(http://clojure.github.com/java.jdbc/doc/clojure/java/jdbc/UsingSQL.html). In this tutorial we'll be using clojure.data.jdbc to keep things simple, but I do encourage you to take a look at the others.

Setting up the DB connection

First, we'll need to define our database connection, this can be done by either providing a map of connection parameters:

(def db {:subprotocol "postgresql"
         :subname "//localhost/my_website"
         :user "admin"
         :password "admin"})

by specifying the JNDI name for a connection managed by the application server:

(def db {:name "jdbc/myDatasource"})

I personally like this option, because it completely separates the code in the application from the environment. For example, if you have dev/staging/production servers, you can point the JNDI connection to their respective databases, and when you deploy your application it will pick it up from the environment.

Finally, you can provide a JDBC data source, which you configure manually:

(def db
  {:datasource
    (doto (new PGPoolingDataSource)
     (.setServerName   "localhost")
     (.setDatabaseName "my_website")
     (.setUser         "admin")
     (.setPassword     "admin")
     (.setMaxConnections 10))})

At this point you should setup a database and create a schema for this tutorial called my_website. I will be using PostgreSQL so if you use a different DB there might be slight syntactic differences in your SQL. Once you have the DB up and running, we'll need to add the clojure.data.jdbc and JDBC driver dependencies to project.clj:

(defproject my-website "0.1.0-SNAPSHOT"
  :description ""my Noir website""
  :dependencies [[org.clojure/clojure "1.4.0"]
                 [noir "1.3.0-beta3"]
                 [org.clojure/java.jdbc "0.2.3"]
                 [postgresql/postgresql "9.1-901.jdbc4"]]
  :main my-website.server)

Using to the Database

Next, let's create a new namespace called my-website.models.db in the models directory of our project, and open it up. Here we'll first need to add a require statement for clojure.data.jdbc:

(ns my-website.models.db
  (:require [clojure.java.jdbc :as sql]))

now let's create a connection:

(def db 
  {:subprotocol "postgresql"
   :subname "//localhost/my_website"
   :user "admin"
   :password "admin"})

we'll add the following function which will allow us to create the users table:

(defn init-db []
  (try
  (sql/with-connection
    db
    (sql/create-table
      :users
      [:id "SERIAL"]
      [:handle "varchar(100)"]
      [:pass   "varchar(100)"]))
  (catch Exception ex
    (.getMessage (.getNextException ex)))))

Here's you'll notice that the create-table needs to be wrapped in a with-connection statement which ensures that the connection is cleaned up correctly after we're done with it. The only other thing to note is the use of "SERIAL" for the id field in the table, which is PostgreSQL specific way to create auto incrementing fields. It's also possible to use keywords such as :int, :boolean, and :timestamp for field types as well as the corresponding SQL string as is done in the above example. The whole statement is wrapped in a try block, so if we get any errors when it runs we'll print the error message.

In the REPL we'll run:

(init-db)

If your DB is configured correctly, then you should now have a users table. We'll now write a function to add a user to it:

(defn add-user [user]
  (sql/with-connection 
    db
    (sql/insert-record :users user)))

now test that the function works correctly:

(add-user {:handle "foo" :pass "bar"})
=>{:pass "bar", :handle "foo", :id 1}

finally we'll need a way to read the records from the database, I wrote the following helper function to do that:

(defn db-read [query & args]
  (sql/with-connection 
    db
    (sql/with-query-results 
      res 
      (vec (cons query args)) (doall res))))

the function accepts an SQL string and optional parameters:

(db-read "select * from users")
({:pass "bar", :handle "foo", :id 1})

(db-read "select * from users where id=?" 1)
({:pass "bar", :handle "foo", :id 1})

we'll write another helper function to fetch the user by handle

(defn get-user [handle]
  (first 
    (db-read "select * from users where handle=?" handle)))

at this point we've got a user table and helper functions to create and query users. Let's hook that up to our pages and provide the functionality to create user accounts and allow users to login.

Creating a Registration Page

Noir provides a very simple way to manage sessions using noir.ession namespace. Let's update our site to allow a user to create an account. First we'll create a new namespace called my-website.views.users and add the following code to it:

(ns my-website.views.users
  (:use [noir.core]
        hiccup.core hiccup.form)
  (:require [my-website.views.common :as common]
            [my-website.models.db :as db]
            [noir.util.crypt :as crypt]
            [noir.session :as session]
            [noir.response :as resp]))

(defpage "/signup" {:keys [handle error]}
  (common/layout
    [:div.error error]
    (form-to [:post "/signup"]
             (label "user-id" "user id")
             (text-field "handle" handle)
             [:br]
             (label "pass" "password")
             (password-field "pass")             
             [:br]
             (submit-button "create account"))))

(defpage [:post "/signup"] user
  (try 
    (db/add-user (update-in user [:pass] crypt/encrypt))
    (resp/redirect "/")
    (catch Exception ex
      (render "/signup" (assoc user :error (.getMessage ex))))))

You'll notice that we've required a few new namespaces which we'll be using shortly. Otherwise, we see a similar setup to what we did in the first part of the tutorial, except when we accept the post from the form, we actually add the user to the database.

We will encrypt the user password using noir.util.crypt and then attempt to store the user in the database. If we fail to add the user we'll render our signup page again, but this time with an error message.

create user page

error displayed when user creation fails

Notice that we pass the user fields back to the defpage displaying the form, so if we get an error we don't have to make the user retype all their information.

Session Management

At this point we need to provide the users with the ability to login with their accounts. Let's go to the common namespace and add a way for users to login. We'll need to add noir.session to our :require statement:

(ns my-website.views.common
  ...
  (:require [noir.session :as session])

then we'll go back to users namespace and create a page to handle logins:

(defpage [:post "/login"] {:keys [handle pass]}
  (render "/" 
          (let [user (db/get-user handle)] 
            (if (and user (crypt/compare pass (:pass user)))
              (session/put! :user handle)
              {:handle handle :error "login failed"}))))

We'll use noir.crypt to validate the password against the one we have in the database, and if the password matches we'll stick the user handle into the session. The syntax for updating the session is fairly straightforward, and the documentation page explains it well. We'll be using get, put!, and clear! functions, notice that put! and clear! have an exclamation mark at the end indicating that they mutate the data in place.

The users will also need a way to logout, so let's add a page to handle that as well:

(defpage [:post "/logout"] []
  (session/clear!)
  (resp/redirect "/"))

When the user logs out, we'll simply clear the session and send them back to the homepage. We will now go to our common namespace and add the noir.session and hiccup.form in our namespace:

(ns my-website.views.common
  (:use [noir.core :only [defpartial]]
        hiccup.element 
        hiccup.form
        [hiccup.page :only [include-css html5]])
  (:require [noir.session :as session]))

then add a helper function to create the login form:

(defn login-form []
  (form-to [:post "/login"]           
           (text-field {:placeholder "user id"} "handle")                        
           (password-field {:placeholder "password"} "pass")                        
           (submit-button "login")))

and finally add it to our layout:

(defpartial layout [& content]
            (html5
              [:head
               [:title "my-website"]
               (include-css "/css/reset.css")]
              [:body               
               (if-let [user (session/get :user)]
                  [:h2 "welcome " user 
                    (form-to [:post "/logout"] (submit-button "logout"))]
                  [:div.login
                   (login-form) [:p "or"] (link-to "/signup" "sign up")])
               
               content]))

At this point our main page should look like the following:

and after we sign up and login, we should see:

The logout button should take us back to the login page by clearing the user session. We now have a complete website with some basic user management, the only thing left to add is actual content. :)

Summary

In this section we learned the following:

how to setup the database and do basic queries
do basic authentication using noir.crypt
use sessions to store user information

Hopefully this is enough to get you started using Noir and making your sites with it. If I omitted anything important let me know in comments and I'll be glad to go over it.

The complete source for this part of the tutorial is available here. Also, for an example of a complete real world site you can see the source for this blog here.

In the next section we'll talk about setting content types and doing file uploads and downloads.

Noir tutorial - part 1

Fri, 17 Aug 2012 00:00:00 +0000

Background

Clojure web stack consists of Ring, which is the base HTTP library akin to Python's WSGI and Ruby's Rack. On top of Ring lives Compojure, which provides some basic routing, and that in turn is leveraged by Noir to provide a simple framework for developing websites. Here's we'll see how to use Noir to build a basic website.

Setting Up a Noir Project With Leiningen

The easiest way to get Noir setup is to use Leiningen 2, which has become the de facto build tool for Clojure. Once you have Leiningen installed, you can simply do the following to get a template site created:

lein new noir my-website
cd my-website

Alternatively, if you're using Counterclockwise with Eclipse, then all you need to do is make a new Leiningen project and put noir in the "Leiningen Template to use:" field.

Project Structure

The template site will have the following structure:

/my-website
  project.clj
  --src/
     --my_website/
       server.clj
       --models/
       --views/common.clj
               welcome.clj
  --test/my_website
  --resources/public/
                  --css/reset.css
                  --img/
                  --js/

The skeleton application contains a few files in it. The project.clj file is used for building the application and managing dependencies by Leiningen. Under the src folder, we have the folder called my_website which contains server.clj. This file contains the entry point to our application. It loads up all the views and provides a main function which can be used to start the application.

The models folder is used to keep the data layer of the application, such as code for the database access and table management. The views folder contains the namespaces describing the pages of our application and their supporting code. The template contains common.clj which provides a basic layout and any code shared between the pages. The welcome.clj is the namespace where an example page is defined.

Dependency Management

Let's first look at the project.clj file:

(defproject my-site "0.1.0-SNAPSHOT"
            :description "FIXME: write this!"
            :dependencies [[org.clojure/clojure "1.4.0"]
                           [noir "1.3.0-beta3"]]
            :main my-site.server)

The file is fairly self explanatory, and currently only contains dependencies for Clojure and Noir.

Running the Project in Development Mode

At this point we should be able to start up our website:

lein run

Starting server...
2012-08-16 09:39:22.479:INFO::Logging to STDERR via org.mortbay.log.StdErrLog
Server started on port [8080].
You can view the site at http://localhost:8080
#<Server Server@2206270b>
2012-08-16 09:39:22.480:INFO::jetty-6.1.25
2012-08-16 09:39:22.521:INFO::Started SocketConnector@0.0.0.0:8080

Let's point the browser to localhost:8080 and make sure everything is working as expected. We should be greeted with a Noir help page since we haven't defined one for "/" route yet. At this point we can start editing our pages and any changes we make should be reflected immediately.

Creating Pages

Noir provides two primary way to manipulate pages. One useful macro is defpartial which simply wraps the body in html function from Hiccup, which will generate the resulting HTML string from our content:

(defpartial foo [content]
  [:p content])

(foo "some stuff")
"<p>some stuff</p>"

The other is defpage, this macro will create a Compojure route for the specified URL. It has the following syntax:

(defpage url params content)

By default defpage is expected to return an HTML string. How that string is generated is up to you. In this tutorial we'll be using Hiccup, but you could just as easily use something like Enlive to create your templates using actual HTML. Noir itself is completely agnostic in this regard.

Now, let's look at the parameters that defpage accepts. First we have a URL which supports the following formats:

a simple string such as "/welcome"
a RESTful path such as "/welcome/:user" where the key :user will be appended to the params map with the value provided when the URL is accessed
a vector specifying the request type which the page responds too : [:post "/welcome"]

Next, we have params, which is simply a map of keywords and their associated values generated from the request parameters. Any keys from the URL will also appear in this map:

(defpage "/welcome/:user" {:keys [user]}
  (html [:html [:body "hello " user]]))

Finally, we add the actual page content to be rendered. As I mentioned above the result must be a string, so generally we'll wrap the contents of each page in (common/layout ...) which was provided by the template. The official documentation for defpage with lots of other examples and details is available here.

Handling Form Input

When making pages with forms the general pattern is to create a defpage for the GET request which will contain the UI, and another for POST which contains the server component. To test that out, let's change welcome.clj to look like the following:

(ns my-website.views.welcome
  (:require [my-website.views.common :as common]
            [noir.content.getting-started])
  (:use [noir.core :only [defpage]]
        hiccup.core hiccup.form))

(defpage "/welcome" {:keys [greeting]}
  (common/layout
    (if greeting [:h2 greeting])
    (form-to [:post "/welcome"]
      (label "name" "name")
      (text-field "name")
      (submit-button "submit"))))

(defpage [:post "/welcome"] {:keys [name]}
  (noir.core/render "/welcome" 
    {:greeting (str "Welcome " name)}))

As can be seen above, the page which responds to GET creates a form and submits it to its POST counterpart. It in turn generates a greeting and renders the page with it. Note that the names for fields used in the form get translated into keys in the params map when we submit it.

before submit

and after submit

This covers the basic model for creating pages and interacting with them. Now, let's look at how we can package our website into a standalone application.

Packaging and Running Standalone

To package our project we need to change our server to compile into a class, we can do this by simply adding gen-class to its namespace like so:

(ns my-website.server
  (:require [noir.server :as server]) 
  (:gen-class))

Now we can build and run our project:

lein uberjar
java -jar my-website-0.1.0-SNAPSHOT-standalone.jar

Starting server...
2012-08-16 20:12:47.846:INFO::Logging to STDERR via org.mortbay.log.StdErrLog
2012-08-16 20:12:47.846:INFO::jetty-6.1.x
2012-08-16 20:12:47.882:INFO::Started SocketConnector@0.0.0.0:8080
Server started on port [8080].
You can view the site at http://localhost:8080

Summary

To recap, in this section of the tutorial we learned the following:

how to create a new Noir project
manage dependencies
create pages
handle submits from forms
create a standalone instance of our application

Next time we'll look at how to do session management and database access.

continue to part 2

The source for the tutorial is available here.

Easy PDF reports with clj-pdf

Thu, 16 Aug 2012 00:00:00 +0000

A few months ago I was tasked with generating reports for one of the applications I was working on. I looked around for some off the shelf libraries for doing this sort of thing. The most popular library in the Java world appears to be iText. It's a mature library with lots of features, but it takes entirely too much code to produce anything useful with it. On top of that, the latest version licensed under LGPL2 is 2.1.7 which, while serviceable, is full of quirks and odd behaviors.

After spending a bit of time playing with it I decided that it would make more sense to have a declarative API for describing the PDF document. I really like the way Hiccup allows generating HTML using nested vectors, and decided that something similar could be done for generating PDF documents.

This lead to the creating of clj-pdf, which allows describing the document using this approach. Each vector represents a different element, such as a paragraph, a list, or a table. Internally, I leverage iText to produce the actual PDF document, but the API is completely declarative. The library attempts to abstract away any of the quirks as well as provide useful elements such as headings, spacers, page breaks, etc.

Let's look at how this all works in practice. A document is simply a vector which contains metadata describing it followed by one or more inner elements:

[{:title "My document"} "some content here..."]

In the spirit of Hiccup, each element is represented by a vector, where the first item must be a tag describing the type of the element, followed by optional metadata, and finally the content of the element. For example if we wanted to create a paragraph we'd do the following:

[:paragraph "a fine paragraph"]

to set the font style we could add the following metadata:

[:paragraph
  {:style :bold :size 10 :family :halvetica :color [0 255 221]}
  "Lorem ipsum dolor sit amet, consectetur adipiscing elit."]

any metadata in an element will propagate to its children:

[:paragraph
  {:style :bold :size 12 :family :halvetica :color [0 255 221]}
  "Lorem ipsum dolor sit amet, consectetur adipiscing elit."
  [:phrase "some text here"]]

here the phrase will inherit the font style of its parent paragraph. However, the child element is always free to overwrite the parent metadata:

[:paragraph
  {:style :bold :size 12}
  "Lorem ipsum dolor sit amet, consectetur adipiscing elit."
  [:phrase {:style :normal :size 10} "some text here"]]

This provides us with a lot of flexibility, while allowing to specify defaults for the entire document. The library attempts to provide reasonable behavior out of the box, so that adding metadata should not be necessary in most cases.

Some reports might include things like usage statistics. And to that end I leveraged the excellent JFreeChart library to provide a simple charting API:

[:chart {:type :line-chart 
         :title "Line Chart" 
         :x-label "checkpoints" 
         :y-label "units"} 
  ["Foo" [1 10] [2 13] [3 120] [4 455] [5 300] [6 600]]
  ["Bar" [1 13] [2 33] [3 320] [4 155] [5 200] [6 300]]]

At this time bar charts, line charts, time series, and pie charts are supported. And because a chart is just an image, all the image styling, such as scaling and alignment, can be applied to it as well.

Since the API is completely declarative, it doesn't actually have to be encoded in Clojure structures. We could instead encode it in something like JSON, which is exactly what I ended up doing next. I created a service which would accept POST requests containing JSON encoded documents and return PDF documents. The service can be accessed by any application regardless of what language its written in, and can even be called by JavaScript from a browser as can be seen here.

Documentation and examples are available on the github project page.

Blogging with Noir

Tue, 14 Aug 2012 00:00:00 +0000

Writing a blog engine in Noir turned out to be a very pleasant experience. The engine which is currently powering this blog supports all the features one would expect from a blogging engine, some of which include:

content caching
RSS feed
tags
markdown in posts and comments with live preview
syntax highlighting
file uploads and management through web UI
captchas for comments
latest comments view
controlling post visibility

All that weighs in at under 1K lines of Clojure, and some 50 lines of Js. I've outlined some of the quirks I ran into previously. Now, I'd like to talk about what went right and what facilitated writing a complete functional website in only a few hundred lines of code.

I used clojure.java.jdbc for database access. The library is very easy to use and provides all the basic functionality you'd expect with minimal fuss. You can define a database either using a map:

(def mysql-db {:subprotocol "postgresql"
               :subname "//127.0.0.1:3306/clojure_test"
               :user "clojure_test"
               :password "clojure_test"})

by providing a JNDI name and configuring a JNDI data source on the app server:

(def my-db {:name "jdbc/myDatasource"})

or by simply instantiating a data source as I do in the blog:

(def db 
  {:datasource 
    (doto (new PGPoolingDataSource)
       (.setServerName   (:host blog-config))
       (.setDatabaseName (:schema blog-config))
       (.setUser         (:user blog-config))
       (.setPassword     (:pass blog-config))
       (.setMaxConnections 10))})

Calling SQL commands is straightforward as well, all statements must be wrapped with a with-connection statement. This ensures that any result sets are cleaned up and the connection is closed once you're done with it. I found the library to be very intuitive and easy to work with. The documentation is available on github and covers most use cases. All of my db interaction ended up fitting in just under 250 lines, which makes it easy to keep on top of.

Noir has been indispensable in making things concise and easy to manage. Noir and Hiccup make it trivial to organize the pages and their controllers into self contained chunks.

Because it encourages making things stateless, it's easy to add functionality in isolated chunks. This means that you can add a particular feature, such as RSS support, without having to worry how it might interact with existing code. I find this especially important when writing side projects as it means that you have a very short ramp up time when you come back to a project after not having touched it for a while.

I'm personally a fan of using Hiccup for generating HTML, as it allows using the full power of Clojure for templating. However, some people have concerns about not having the actual HTML that designers can then style. Fortunately, there's nothing inherent to Noir that ties it to Hiccup. A defpage simply has to return an HTML string, how that string gets generated is entirely up to you. And there's a great guide for using Noir with Enlive, which is designed for HTML based templating. Again, I have to point out the thoughtfulness of design which separates creating routes and serving pages from the libraries which deal with actually generating them.

For Markdown parsing I dredged up an old library of mine, and with a few tweaks it's been doing its job as far as far as this blog is concerned. One advantage of this particular library is that it compiles to both Clojure and ClojureScript, so I can do previews in the browser and guarantee that they will be rendered the same by the server.

I added the ability to add language hinting using github style markdown, eg: ```clojure, to output tags compatible with the syntax highlighter, which I then use to do code highlighting in the browser.

I also didn't find any readily available libraries for generating RSS from Clojure, so I proceeded to make clj-rss, which turned out to be very easy thanks to the excellent XML support in the standard library and a few macros.

For my captcha needs I turned to a Java library called jlue. Thanks to the excellent Java interop, using it is quite seamless:

(defn gen-captcha []
  (let [text (gen-captcha-text)
        captcha (doto (new Captcha))]
    (session/put! :captcha 
                  {:text text 
                   :image (.gen captcha text 250 40)})))

(defpage "/captcha" []
  (gen-captcha)
  (resp/content-type 
    "image/jpeg" 
    (let [out (new ByteArrayOutputStream)]
      (ImageIO/write (:image (session/get :captcha)) "jpeg" out)
      (new ByteArrayInputStream (.toByteArray out)))))

Finally, all of the building and packaging is handled by Leiningen, which makes it trivial to track dependencies and package up the resulting application. In my case I'm deploying the blog to Tomcat, and so I simply build a WAR using:

lein ring uberwar

The resulting WAR can be dropped on any Java application server. If you wanted to deploy to Heroku, you simply have to add a Procfile to the root directory of the project with the following contents:

web: lein trampoline run -m yuggoth.server

Overall, I experienced very few issues and found the experience to be overwhelmingly positive. In my opinion the current tools and libraries available in Clojure allow writing web sites just as easily, if not more so, as most of the established languages out there.

Why be conservative

Sat, 11 Aug 2012 00:00:00 +0000

Steve Yegge has made a post introducing the idea of liberalism and conservatism in programming languages. While it is an entertaining read I have to question the usefulness of the proposed metric.

In my opinion the language either gets out of your way and makes it easy to do what you need to do or it doesn't. I don't really care how it does it as long as at the end of the day I enjoy using it and I'm productive in it.

It can certainly be argued that Clojure is conservative in some ways. As has been pointed out in the results of the 2012 State of Clojure survey, some people find the process for contributing to the language too restrictive. Rich Hickey is very cautious about adding new features and about the way they're added to the language.

But I would argue that this is in fact a good thing and the end result is a cleaner and more consistent language. Destructuring is a concrete example of this. At one point people were asking for named arguments for functions and Rich resisted the idea of adding them. Instead, we got destructuring which is a more powerful and general purpose tool. It can be used for naming arguments in functions, but it can also be used for many other things as well.

Let's consider what the result would have been if Clojure was more liberal about adding features, and named arguments were in fact added. There would now be two separate ways to do the same thing, each with its own quirks. Different code bases would use different rules for naming function parameters and you would have to make adapters to make them work together.

The more eagerly features get accepted into a language, the more likely they it is that the solution won't be elegant or general purpose. Which means that inevitably a new feature needs to be added to cover the case which isn't adequately addressed by the original attempt.

In my opinion this quickly leads to having a crufty syntax, and requires a lot of mental overhead to work with code written by others. Since, some people will prefer this or that particular style of doing things you have to be aware of every quirk and their interactions.

Fact of the matter is that Lisp is already phenomenally powerful, more so than most languages out there. It would seem prudent not to be rash about trying to improve it.

Setting up Eclipse for Clojure

Tue, 07 Aug 2012 00:00:00 +0000

The Results of the 2012 State of Clojure survey are out, and they look very exciting indeed. More people are using Clojure, the community is growing, and for the most part things appear to be progressing well. However, one notable problem that people are reporting is actually getting started with Clojure.

I'd like to spend some time here to help people actually get up and running with the language. First, I'll cover setting up the development environment. Many Clojure users gravitate towards Emacs, which is a natural choice for Lisp development. But if you're new to Clojure and you haven't used Emacs before, I would strongly suggest against learning both Emacs and Clojure at the same time.

The reason being is that Emacs is fairly arcane in many ways, and it behaves very differently from traditional IDEs, such as NetBeans or Eclipse. Learning a new language, which has very different syntax from languages you might be used to, and requires learning a new programming paradigm is enough to keep one busy without having to learn a quirky IDE on the side.

My recommendation would be to grab a copy of Eclipse and install the Counterclockwise plugin. Installing the plugin is incredibly simple, once you have Eclipse running follow the following steps:

navigate to the "Install new Software" tab under the help menu
paste in the CCW update URL: http://ccw.cgrand.net/updatesite in the "Work with:" text field
check the "Clojure Programming" checkbox and hit the "Next" button

Counterclockwise takes care of setting up Clojure and Leiningen for you. And once the plugin is installed, you will be able to create a new Clojure project or a new Leiningen project. I would recommend making Leiningen projects, since they allow easily managing dependencies by updating the project.clj file in the project directory. I'll touch more on this later.

At this point, I'll assume that you have Eclipse with CCW up and running. So, navigate to File->new->project in Eclipse menu. Then select Leiningen->Leiningen project. Here you'll see the default Leiningen Template filled in. And only thing you have to do is provide a project name. Let's call our project "clojure-test" and hit the finish button.

You should now see a new project in your Package Explorer view on the left. The project template will have a src folder which will contain the package folder named clojure_test. Since Java cannot use dashes in names, all the dashes in package folders for namespaces get converted to underscores. The pckage will contain a core.clj file, and its contents should look like the following:

(ns clojure-test.core)

(defn -main
  "I don't do a whole lot."
  [& args]
  (println "Hello, World!"))

Let's open it and then hit the run button. You should see a REPL pop up momentarily on the bottom of the IDE. If all went well, your project should be ready to work on. The code that's in the file will have already been loaded up in the REPL when we hit run, and we should now be able to call our -main function.

To do that, let's write the code which calls main below it:

(-main)

Then navigate the cursor inside the call body and hit CTRL+ENTER on Linux/Windows or CMD+ENTER on OS X. You should see "Hello, World!" printed in the REPL view on the bottom. We can now change the behavior of the -main function and after it is reloaded the new behavior will be available next time it's called.

I would also recommend enabling the "strict/paredit" mode under Preferences->Clojure->Editor section. This will allow the editor to keep track of balancing the parens for you. It might seem odd at first, but I highly encourage you to stick with it.

Another useful feature of the editor is the ability to select code by expression. If you navigate inside a function and press ALT+SHIFT+UP (use CMD instead of ALT in OS X), then inner body of the expression will be selected, pressing it again, will select the expression, and then the outer body, and so on. Conversely pressing ALT+SHIFT+DOWN will narrow the selection. This allows you to quickly navigate nested structures, and select code by chunks of logic as opposed to simply selecting individual lines.

I've also mentioned the project.clj file in your project folder earlier. This file should look like the following:

(defproject clojure-test "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.3.0"]])

You can add new dependencies to your project by simply sticking them in the dependencies vector. For example, if we wanted to add an HTTP client, we'd go to http://clojuresphere.herokuapp.com/ click on clj-http link. From there select the Clojars link and copy the following:

[clj-http "0.5.2"]

now we'll simply paste it under dependencies in our project.clj:

:dependencies [[org.clojure/clojure "1.3.0"]
               [clj-http "0.5.2"]]

In our package explorer view on the left we should be able to expand "Leiningen dependencies" and see the clj-http jar included there. We will now have to kill our current REPL, to do that navigate to the terminal view next to it and press the stop button. When we start a new instance of the REPL, the library will be available for use. In the core file we can now add it to the namespace:

(ns clojure-test.core
 (:require [clj-http.client :as client]))

and test using the client by typing

(client/get "http://google.com")

and running it as we did earlier. This should cover all the basics of using Clojure with Counterclockwise, and allow you to get hacking on your project.

I'd also recommend visiting the following sites:

4Clojure is an excellent site for practicing small exercises in Clojure. Be sure to make an account and follow some of the top users. When you solve a problem, you'll be able to see how others solve it and get a taste for idiomatic Clojure code.
Clojure - Functional Programming for the JVM is a very comprehensive introduction to Clojure aimed at Java programmers.
ClojureDocs is an excellent documentation site for Clojure which contains many examples on using the functions in the standard library.
Noir is an great Clojure framework for making web apps, in fact this blog is built on top of it with source available here.

There are many other great Clojure sites that I failed to mention here, but the above should provide a good starting point.

Serving RSS with Clojure

Sat, 04 Aug 2012 00:00:00 +0000

I recently got invited to join Planet Clojure, which is an excellent place for keeping up with what people are up to in Clojure world. As part of being syndicated I had to add an RSS feed to my blog. A cursory Google search came up with lots of tutorials for parsing RSS, but nothing regarding generating it. Turns out that it's very straight forward and it takes less than a 50 lines of code to create a proper RSS feed for your site.

First, a bit of background about RSS. Essentially, it's a very simple syndication format designed to allow pushing out notifications about frequently updated content such as blog posts. RSS is served as XML and each feed has to consist of a channel tag with some metadata and item tags, each one describing a specific update such as a new blog post.

All we have to do to create our RSS feed is to structure the data accordingly and serialize it to XML. Clojure standard library provides a simple way to output XML using the emit function in the clojure.xml namespace. It accepts data in the following format:

{:tag :tag-name :attrs attrs-map :content [content]}

The content in the above can contain a mix of strings and tags. One thing to be aware of is that any other content will result in a null pointer exception, so it's one of rare cases where that doesn't get handled gracefully by default. Once we've constructed a proper tag we can serialize it to XML as follows:

(with-out-str 
  (clojure.xml/emit 
    {:tag :channel :attrs nil :content []}))

which results in

<?xml version='1.0' encoding='UTF-8'?>
<channel>
</channel>

Note that emit needs to be wrapped in with-out-str to capture its output into a string. RSS also specifies the format in which time should be output, so we'll make a helper function to handle that:

(defn format-time [time] 
  (.format (new java.text.SimpleDateFormat 
                "EEE, dd MMM yyyy HH:mm:ss ZZZZ") time))

Writing out the tags by hand gets tedious, so I wrote a macro to output the tags for us:

(defmacro tag [id attrs & content]
  `{:tag ~id :attrs ~attrs :content [~@content]})

I covered macros briefly in an earlier post. The only new syntax used here is the ~@ notation, which simply says that the items in content should be inserted into the enclosing structure, eg:

(tag :foo nil "foo" "bar" "baz")
{:tag :foo, :attrs nil, :content ["foo" "bar" "baz"]}

Armed with this macro let's write the function to describe an individual post. The function accepts the site, the author and a map describing the post as parameters, then generates the appropriate tags as per RSS specification.

(defn item [site author {:keys [id title content time]}]
  (let [link (str site "/" id )] 
    (tag :item nil
         (tag :guid nil link)
         (tag :title nil title)
         (tag :dc:creator nil author)
         (tag :description nil content)
         (tag :link nil link)
         (tag :pubDate nil (format-time time))
         (tag :category nil "clojure"))))

Let's test that it does what we expect:

(item "http://yogthos.net"
      "Yogthos" 
      {:id 1 
       :title "Test post" 
       :content "Some content" 
       :time (new Date)})

{:content
 [{:content ["http://yogthos.net/1"], :attrs nil, :tag :guid}
  {:content ["Test post"], :attrs nil, :tag :title}
  {:content ["Yogthos"], :attrs nil, :tag :dc:creator}
  {:content ["Some content"], :attrs nil, :tag :description}
  {:content ["http://yogthos.net/1"], :attrs nil, :tag :link}
  {:content ["Sat, 04 Aug 2012 18:16:03 -0400"],
   :attrs nil,
   :tag :pubDate}
  {:content ["clojure"], :attrs nil, :tag :category}],
 :attrs nil,
 :tag :item}

If we pass the above to xml/emit we'll get the corresponding XML. Next we'll need a function which will will create the representation of the channel:

(defn message [site title author posts]
  (let [date (format-time (new Date))] 
    (tag :rss {:version "2.0"
               :xmlns:dc "http://purl.org/dc/elements/1.1/"
               :xmlns:sy "http://purl.org/rss/1.0/modules/syndication/"}
         (update-in 
           (tag :channel nil
                (tag :title nil (:title (first posts)))
                (tag :description nil title)
                (tag :link nil site)
                (tag :lastBuildDate nil date)
                (tag :dc:creator nil author)
                (tag :language nil "en-US")
                (tag :sy:updatePeriod nil "hourly")
                (tag :sy:updateFrequency nil "1"))
           [:content]
           into (map (partial item site author) posts)))))

Again, this is fairly straight forward, the function takes the site url, blog title, the author and the posts. Then it creates the necessary tags to describe the channel and inserts the formatted posts into it. We should now be able to generate valid RSS content by calling it with some data:

(message "http://yogthos.net" "My blog" "Yogthos" 
         [{:id 1 
           :title "Test post" 
           :content "Some content" 
           :time (new Date)}])

{:content
 [{:content
   [{:content ["Test post"], :attrs nil, :tag :title}
    {:content ["My blog"], :attrs nil, :tag :description}
    {:content ["http://yogthos.net"], :attrs nil, :tag :link}
    {:content ["Sat, 04 Aug 2012 18:23:06 -0400"],
     :attrs nil,
     :tag :lastBuildDate}
    {:content ["Yogthos"], :attrs nil, :tag :dc:creator}
    {:content ["en-US"], :attrs nil, :tag :language}
    {:content ["hourly"], :attrs nil, :tag :sy:updatePeriod}
    {:content ["1"], :attrs nil, :tag :sy:updateFrequency}
    {:content
     [{:content ["http://yogthos.net/blog/1"], :attrs nil, :tag :guid}
      {:content ["Test post"], :attrs nil, :tag :title}
      {:content ["Yogthos"], :attrs nil, :tag :dc:creator}
      {:content ["Some content"], :attrs nil, :tag :description}
      {:content ["http://yogthos.net/blog/1"], :attrs nil, :tag :link}
      {:content ["Sat, 04 Aug 2012 18:23:06 -0400"],
       :attrs nil,
       :tag :pubDate}
      {:content ["clojure"], :attrs nil, :tag :category}],
     :attrs nil,
     :tag :item}],
   :attrs nil,
   :tag :channel}],
 :attrs
 {:version "2.0",
  :xmlns:dc "http://purl.org/dc/elements/1.1/",
  :xmlns:sy "http://purl.org/rss/1.0/modules/syndication/"},
 :tag :rss}

Finally, we'll write a function which converts the message to XML:

(defn rss-feed [site title author posts]
  (with-out-str (emit (message site title author posts))))

We can confirm that we're generating valid content by pasting it to W3C Feed Validation Service. This is all that's needed to create a valid RSS message. It can now be served over HTTP using your favorite library or framework.

Complete code for the example can be found here.

Updates

I've since rolled all of the above into a (hopefully :) friendly clj-rss library.

Noir tricks

Thu, 02 Aug 2012 00:00:00 +0000

This blog is built on top of Noir, which is quite excellent for the most part. However, I did run into one problem which I spent a bit of time on. I'd like to share my workarounds to save others time.

First issue I noticed is that response/redirect doesn't respect the servlet context. This means that if you're not deploying your app to the root context, your redirects will not work properly.

After some digging and questions on the Google groups I found out that the offending function is resolve-url in noir.options namespace. When it builds the URL string it doesn't check for the context and as such the resulting URL ends up redirecting to the root of the app server regardless of what context the servlet was deployed at.

My workaround for this is a bit of a hack, and if anybody has a better solution I'd love to know, but it works well for most purposes. In my server.clj I added a new handler wrapper, which redefines the offending function with one that checks if the URL is relative and prepends the context to it as needed.

(defn fix-base-url [handler]
  (fn [request]
    (with-redefs [noir.options/resolve-url 
                  (fn [url] 
                    ;prepend context to the relative URLs
                    (if (.contains url "://")
                      url (str (:context request) url)))]
      (handler request))))

A related issue is that pre-route doesn't respect the context either. I decided to simply write a macro for defining private pages:

(defmacro private-page [path params & content]
  `(noir.core/defpage 
     ~path 
     ~params 
     (if (session/get :admin) 
       (do ~@content) (resp/redirect "/"))))

An added advantage of the macro is that I don't have to remember to update pre-routes when I want to make a page private.

Also, there are a couple of things to be aware of if you wish to make a WAR. Make sure that all your views are required in your server namespace, :gen-class is set and that server/load-views-ns is used instead of server/load-views:

(ns yuggoth.server
  (:require 
   ...
   [yuggoth.views archives auth blog comments common profile rss upload])
   (:gen-class))

(server/load-views-ns 'yuggoth.views)

In your project.clj add the following:

:ring {:handler yuggoth.server/handler}

With the above in place you can build an uberwar with

lein ring uberwar

The resulting WAR should deploy on any app server such as Tomcat or Glassfish without problems. Aside from the above quirks, I haven't run into any other issues with Noir, and I'm absolutely in love with it.

open access

Tue, 31 Jul 2012 00:00:00 +0000

Sometimes you might run into a situation where you're using a library which defines a certain function in a way that might not work the way you need it to in a particular context. To make things worse, this function might be used by the library internally, so you can't simply write your own version and use it.

In some languages it's possible to use monkey patching to get around this problem. This approach allows you to simply redefine the offending function at runtime with your own version. The downside of this approach is that the change is global and as such might interact poorly with other code which expects the original version.

In Clojure it's possible redefine an existing function in a particular context using with-redefs. This approach gives us the ability to make runtime modifications in a safer fashion where we know exactly what code is affected.

Let's look at an example where we have a get-data function defined in namespace foo which is used by display-results in namespace bar. When we write tests for bar we would like to use preset test data instead of calling out to the database:

(ns foo)

(defn get-data []
  ;gets some data from a db
  )

(ns bar
 (:require foo))

(defn display-results []
  (apply str (interpose ", " (foo/get-data))))

(ns tests
  (:use clojure.test)
  (:require foo bar))

(deftest display-results-test
  (with-redefs [foo/get-data (fn[] ["Doe", "John"])]
    (is (= "Doe, John" (bar/display-results)))))

Now any code that references foo/get-data inside the with-redefs scope will get ["Doe", "John"] as a result.

a look at macros

Fri, 27 Jul 2012 00:00:00 +0000

Lisp macros can be rather confusing and especially so for newcomers. In fact, the rule of thumb is not to use macros if you can avoid them. That said, macros can be an incredibly powerful tool and have innumerable uses. I'd like to give a concrete example of a macro that I'm using in this blog engine.

I wanted to be able to cache page content in memory, so that the page doesn't need to be generated for every single request. This means that before rendering a page I want to check if the page is in my cache, and if the cache hasn't expired then serve the cached page, otherwise render a new version and cache it.

First I created an atom which would store the cached content:

(def cache (atom {}))

Next I wrote the cache logic for the /blog page:

(defpage "/blog/:postid" {:keys [id]}
  (let [last-updated (:time (get @cache id))
        cur-time     (.getTime (new java.util.Date))]

    (if (or (nil? last-updated)
            (> (- cur-time last-updated) 10000))
      (swap! cache assoc id {:time cur-time 
                             :content (entry (db/get-post id))}))
    (:content (get @cache id)))

Obviously, we don't want to repeat this logic each time we wish to cache something, and we'd like an easy way to modify existing functions to allow caching. Here's where macros come in. One property of macros is that, unlike functions, they do not execute the s-expressions which are passed in. Let's look at how this works in practice:

(defn foo [] (println "foo was called"))

(defn bar [f])

(bar (foo))

=>foo was called

Here foo is executed as we would expect and "foo was called" is printed, but what happens if we make bar a macro instead?

(defmacro bar [f])
(bar (foo))
=>

This time nothing is printed! In a macro the parameters are treated as data and are not evaluated unless we explicitly choose to do so:

(defmacro bar [f] f)
(bar (foo))
=>foo was called

A macro allows us to change code before it is compiled, and at compile time it is replaced with its output. We can check this by running macroexpand:

(macroexpand '(bar (foo)))
=>(foo)

We can see that (bar (foo)) simply gets replaced with (foo) which is what our macro is returning. While the previous version would evaluate to nil, and foo would never be executed.

As you might have guessed by now, we can pass any s-expression to a macro and then decide inside the macro whether we'd like to evaluate it. So, let's see how we can use this to make our caching macro:

(defmacro cache [id content]
  `(let [last-updated# (:time (get @cached ~id))
         cur-time#     (.getTime (new java.util.Date))]

      (if (or (nil? last-updated#)
              (> (- cur-time# last-updated#) 10000))
        (swap! cached assoc ~id {:time cur-time#
                                 :content ~content}))
      (:content (get @cached ~id))))

We can move the logic which checks if we should use a cached value into our macro and pass in the id and the s-expression to run if cache needs to be updated. The code looks very similar to our original version, except for a few new symbols. First thing you'll notice is that we used ` in front of our let expression, this quotes the body of the expression. The # at the end of the binding names ensures that the names are unique and won't collide with other symbols at compile time. Finally ~ says that the next expression should be unquoted.

Let's run macroexpand again to make sure our macro is outputting something reasonable:

(pprint (macroexpand '(cache postid (entry (get-post postid)))))

(let*
 [last-updated__1294__auto__
  (:time (clojure.core/get @agents/cached postid))
  cur-time__1295__auto__
  (.getTime (new java.util.Date))]
 (if
  (clojure.core/or
   (clojure.core/nil? last-updated__1294__auto__)
   (clojure.core/>
    (clojure.core/- cur-time__1295__auto__ last-updated__1294__auto__)
    10000))
  (clojure.core/swap!
   agents/cached
   clojure.core/assoc
   postid
   {:content (entry (get-post postid)), :time cur-time__1295__auto__}))
 (:content (clojure.core/get @agents/cached postid)))

This definitely looks like the logic we're expecting. Any time we use this macro, it will be replaced with the code similar to the above, where the s-expression is inside the if block, and only gets called if cache needs to be updated. Now we can easily cache any s-expressions with minimal change to the original code and all the caching logic sits in one convenient place:

(defpage "/blog/:postid" {:keys [postid]}
  (cache postid (entry (db/get-post postid))))

As I've mentioned before, there are many other uses for macros, but I hope this gives a clear example of a concrete situation where a macro facilitates cleaner code and provides an easy way to avoid repetition.

perfection

Fri, 20 Jul 2012 00:00:00 +0000

There's a quote by Antoine de Saint-Exupery that says: "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away". I think any experienced programmer can relate to that. You always strive to find the most elegant solution which describes the problem simply and clearly.

A lot of novice programmers have a habit of writing clever code which uses some esoteric properties of the language, or other tricks to get the job done. An experienced programmer knows that the real cleverness lies in being able to solve a problem with very simple code, that might even seem obvious in retrospect.

Eventually one develops an intuition for coming up with solutions which do not involve kludges, avoid edge cases, and forgo cleverness in favor of simplicity. Sometimes, however, this can lead to paralysis, where you don't yet know the elegant solution and you are unwilling to write down the one you know to be imperfect.

I find that REPL development is a great tool for overcoming this dilemma. You can quickly start experimenting with your problem, and through the experimentation gain the understanding necessary to implement it properly. At this point you can easily refactor your existing ugly solution into something beautiful.

less is more

Fri, 20 Jul 2012 00:00:00 +0000

An expressive language has many benefits. The most obvious one is that you have to write less code to solve your problem. The reason you write less code is often not because the syntax is more terse, but because you're using better abstractions. For example, instead of writing a loop, you can use an iterator function to do the work:

(loop [count 0
         [head & tail] items]
    (if tail
      (recur (+ count head) tail)
      (+ count head)))

(reduce + items)

One non-obvious benefit of having less code is that it makes it much easier to throw code away. In a verbose language where you have to write a lot of code to solve simple problems, you tend to become attached to that code. In a language where you can express complex things in a relatively few lines, it's not a big issue to replace those with a few different lines. This encourages refactoring as you go, instead of waiting until you have a mountain of code accumulated and you really need to do something about it.

all things being equal

Fri, 13 Jul 2012 00:00:00 +0000

You might have heard terms such as anonymous functions, first class functions, higher order functions, and closures. These might sounds mathy and imposing, but they're very simple ideas. In fact, I'll argue that they make the language simpler and more consistent.

In some languages there's a distinction between a function and a variable. You can assign variables, pass them in as parameters, and return them. Yet when it comes to functions, all you can do is define them and call them.

If you take a moment to think about it, I think you'll agree that this distinction is fairly arbitrary. There's no practical reason why we shouldn't be able to do all the things we do with variables with functions.

Let's look at some things that become possible once this distinction is erased. Sometimes we like to use values inline and not assign them to a variable, we usually do this because the value is only going to appear once, and we don't want to go through the ceremony of naming it.

If our language supports anonymous functions, we can do the same thing with a small piece of logic. If it's only needed in a single situation then we can make an anonymous function and call it directly:

    ((fn [x] (* 2 x)) 5)
    10

Here we created an anonymous function which takes a value and multiplies it by 2, and we passed 5 to it as a parameter. Just as we name values which we reuse in multiple places, so can we name functions:

    (def times-2 (fn [x] (* 2 x)))

and then call them by their name instead

    (times-2 5)
    10

The other thing we said that we can do with variables is pass them as parameters to functions. By being able to pass functions to other functions, we're able to decompose our logic into smaller chunks.

If we takes our times-2 function and pass it in as a parameter to an iterator function such as map, it in turn can apply it to each element in a collection:

    (map times-2 '(1 2 3 4))
    (2 4 6 8)

You might recognize this as the strategy pattern from OO. Turns out that all the complexity in the pattern comes from the idea of treating functions as second class citizens. Which brings us to the idea of a first class function. All that means is that a function is treated no differently than a variable. The only other thing we haven't defined is the higher order function, map in the above example is such a function. Once again, there's nothing complicated about the concept. Any function which can accept another function as a parameter is a higher order function.

Finally, what happens if functions can return functions as output. There are many uses for this, but I'd like to focus on one that will be familiar from OO. When we create a class we often use a constructor to initialize some data that will be available to the methods of the instantiated object.

In a functional language we can achieve this by having a function which takes some parameters and returns another function. Because the inner function was defined in scope where the parameters are declared it too can access them. Here's an example:

    (defn foo [x]
      (fn [y] (* x y)))

    ((foo 2) 5)
    10

Function foo accepts parameter x and returns an anonymous function which in turn accepts a parameter y and multiplies them together. Function foo is said to close over its parameters, and hence it's called a closure. Unlike a constructor a closure does not introduce any special cases. It's just a function that returns a result which itself happens to be a function.

Treating functions as first class citizens makes the language more uniform. Instead of having special constructs for specific cases, we have a general purpose tool that we can apply in many situations.

limits of mutation

Thu, 12 Jul 2012 00:00:00 +0000

When you start learning functional programming you will quickly notice that you can't simply mutate data in place as you might be used to. Initially you might find this odd and restrictive, but it turns out there are also some tangible benefits to this approach.

Mutable data structures are very simple in nature. They reference a location in memory where some value can be stored, when that value changes the old one is simply replaced with the new.

Persistent data structures create revisions of the data when changes are made. We pay a small penalty in performance compared to in place mutation, but we gain a history of changes that exists as long as its referenced somewhere.

This means that if a function accepts some data as a parameter, you don't have to worry if anybody else is referencing that data when you work with it. Any time you change the data you get a new version without paying the penalty of copying it. By contrast, we always have to be aware if a reference may be used else where when working with mutable data. By removing this worry, we can reduce the scope of things that we need to keep in our heads when trying to understand what a particular piece of code does.

The benefits stack up as your project grows, as it becomes infeasible to keep the totality of the code in ones head. And it's a huge benefit when working in a threaded environment and shared data can easily be corrupted.

What might seem like an inconvenience at first turns out to be a net benefit. Ensuring that the data is not modified outside the intended context has been offloaded to the language instead of being done by you manually. I would liken this to use of garbage collection, where the language is responsible for most memory reclamation. In both cases it's better to let the machine do tasks that can be automated leaving you to solve the real problems.

we'll do it live!

Tue, 10 Jul 2012 00:00:00 +0000

One thing I love about working in Clojure is how interactive the development environment is. Clojure being a Lisp provides a REPL (read, evaluate, print, loop), which works exactly like it sounds. You send an expression to the reader, which will then evaluate it, print the result, and wait for another expression to read.

Clojure IDEs provide tight integration with the REPL. It is possible to connect your application to it and have it load all the libraries and dependencies. At this point you can write your new code in the IDE and have the REPL evaluate it in the context of your running application.

In non-trivial applications it's often necessary to build up a particular state before you can add more functionality. For example a user has to login then view some data from a backend, then you need to write functions to format and display this data. With a REPL you can get the application to the state where the data is loaded and then write the display logic interactively without having to reload the application every time you make a change.

I find this method of development a lot more satisfying, as you get immediate feedback from your application when you add or modify code, and you can easily try things and see how they work. It encourages extermination and refactoring code as you go, which I think helps write better and cleaner code.

This technique is common in Lisp and Smalltalk development, but for reasons unknown has not penetrated into mainstream languages.

lost in patterns

Mon, 09 Jul 2012 00:00:00 +0000

Design patterns are heavily used in the OO world, and there are many lengthy books written about them. I'd like to examine why this is and what these patterns stem from exactly.

As the name implies, design patterns are templates for structuring code to solve common problems. This is a fine idea in and of itself, but the following question needs to be asked. If programming is ultimately about automation, and patterns are repetitive tasks by their very nature, then why are we forced to write them out by hand each time?

The reason for this appears to be due to lack of abstraction in the language. Many design patterns are simply specific cases of an underlying abstraction which unifies them. Having a language which can express such abstractions means that you don't have to learn many different patterns for specific situations.

Bruce Lee once said "I fear not the man who has practiced ten thousand kicks once. But I fear the man who has practiced one kick ten thousand times". I think this applies here as well: it's better to learn a general solution for many problems, than to have a specific solution for each small problem you run into.

So, next time you're looking at a language, don't simply look at the one that has the bigger list of features, instead look for one with a few features that work well together.

living in a structured world

Sun, 08 Jul 2012 00:00:00 +0000

If you've seen any Lisp code before, you've probably noticed that it looks different from other languages in that the parens come before the function name, prefix notation is prevalent, and that functions are often nested inside one another. The technical term for this is that Lisp uses s-expressions.

These might look awkward at first, and many newcomers immediately think that they can and should be improved upon. Surely it would be easy to write a preprocessor that would let you write code as you write it in other languages and then convert it to s-expressions. This is absolutely true and in fact there is one prominent attempt called sweet-expressions. Despite all that, the idea just doesn't catch on and I'd like to explore what the advantages of working with raw s-expressions are.

One immediate benefit is that Lisp syntax follows the principle of least astnoishment very well. Any time you read code, it always follows the pattern of (function-name arguments), which makes for very consistent looking code. This helps reduce the mental overhead when reading and writing code, instead of worrying about language quirks you can focus on the actual problem you're solving.

Another benefit is that the code provides extra information about itself, which is not available in other languages. With s-expressions you can visually see how functions relate to one another. In essence the code is rendered as a tree representing the execution logic.

Finally, the s-expressions make editing code a completely different experience from other languages. Instead of working in terms of lines, you work in terms of functions. With a ParEdit style editor you can select code not by line but by function! Now you can easily select, move, and reparent pieces of logic. Editing code becomes like playing with Lego pieces and arranging them in different ways.

In my experience these things make the language more enjoyable to work with and the benefits far outweigh any perceived ugliness. After a while you don't even see the parens.

Temporally oblivious

Wed, 04 Jul 2012 00:00:00 +0000

Objects are state machines, yet no mainstream OO language ensures the consistency of the internal state of the object over time. This means that in a multi-threaded environment it's possible to see the internal state of the object while it's being updated. What's even worse is that even if you don't see a partial state, you might be seeing an unexpected state, since someone else with a reference to the object might have updated it for their use, which conflicts with the way you're using it.

The whole situation is fairly messy, but what is the alternative you might ask. My answer would be not to use in place mutation unless absolutely necessary. Instead it's much better to use persistent data structures, which are temporally aware. A persistent data structure works in a fashion akin to version control. Any time a change to the data is made, a delta is created between the existing data and the new data. From user perspective you're simply copying the data, but you're only paying the price of the change.

This concept turns out to be very powerful as it inherently contextualizes any changes. It also allows doing things like rollbacks trivially as you just have to unwind your operations to see a previous state.

why all the parens

Thu, 28 Jun 2012 00:00:00 +0000

A common complaint you hear from people about Lisp is that there are too many parens. Let's compare what's involved in writing a Java method to writing a Clojure function:

    public static void foo(String bar, Integer baz) {
        System.out.println(bar + ", " + baz);
    }

    (defn foo [bar baz] 
      (println bar ", " baz))

The number of parens is exactly the same, but there's clearly more noise in the Java version. In my opinion the noise adds up and it distracts from the intent of the code. The more code you have the harder it is to tell what it's doing and conversely the harder it is to spot bugs in it. I'll illustrate this with a concrete example.

The problem is to display a formatted address given the fields representing it. Commonly an address has a street, a city, a postal code, and a country. We'll have to examine each of these pieces, remove the null and empty ones and insert some separator between them.

So given something like

street: 1 Main street
city: Toronto
posal: A1B 2C3
country: Canada

we'd like to output

1 Main street, Toronto, A1B 2C3, Canada

we should obviously handle empty fields and not have ,, if the field isn't there, and we should make sure we handle nulls in case the whole address is null or some fields in the address are null.

Let's first examine how we would write this in Java:

    public static String concat(String... strings) {
        if (null == strings) return null;
        StringBuffer sb = new StringBuffer();
        for (String s : strings) {
            if (null == s || s.equals("")) continue;
            sb.append(s);
            sb.append(',');
        }
        String s =  sb.toString();
        return s.substring(0, s.lastIndexOf(','));
    }

lines of code : 11
parens: 26
curly braces: 4
semicolons: 7
colons: 1
dots: 6

Now let's compare this to Clojure:

    (defn concat-fields [& fields]
      (apply str (interpose "," (remove empty? fields))))

lines of code : 2
parens: 8
brackets: 2

The Clojure version has significantly less code, and a lot less noise. In addition, we didn't have to do any explicit null checks in our code, and we were able to write the complete solution simply by composing together functions from the standard library!

One very important difference between the Java version and the Clojure version is that the Java version talks about how something is being done, while the Clojure version talks about what is being done. In other words, we have to step through the Java version in our heads to understand what the code is doing.

In the Clojure version this step is not present because the code says what it's doing, and all the implementation details have been abstracted from us. This is code reuse at work, where we can write simple functions that do one thing well and chain them together to achieve complex functionality.

This bears a lot of resemblance with the Unix philosophy: "Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface." Except in our case we're dealing with functions instead of programs and common data structures as a universal interface in the language.

popularity contests

Sat, 23 Jun 2012 00:00:00 +0000

The argument that Lisp is not popular because it's somehow a bad language is not really sound. A lot of great technologies have lost out to inferior ones because of poor marketing. The Lisp community has not in general been great at marketing the language, and it is viewed as downright scary by majority of people.

It also doesn't help that there is no definitive standard distribution of Lisp, or a comprehensive standard library. Most people aren't going to jump through hoops to learn an esoteric language. So, it is no surprise that there aren't a lot of big commercial Lisp projects. It becomes a catch 22, where due to lack of Lisp developers companies develop apps in more popular languages, and people don't bother learning Lisp because there are no jobs for it.

Clojure avoids a lot of the pitfalls by running on the JVM and interfacing with Java. Java is rather dominant in the industry, a lot of companies already use it, and using alternative languages on the JVM is also becoming a fairly common practice. Strong Java integration also means that you have access to a great wealth of existing libraries.

Having the ability to introduce Clojure in an existing project without having to change your environment is a huge plus. You can continue to use the same build tools, the same IDE, and same application servers for deployment. The only thing that changes is the actual language.

From the language design perspective I think it is also an improvement over the traditional Lisp syntax. For example let's compare let in CL to let in Clojure:

    (let 
      ((a1 b1) 
       (a2 b2) 
       (an bn))
      (some-code a1 a2 an))

    (let [a1 b1
          a2 b2
          an bn]
      (some-code a1 a2 an))

To me Clojure version is easier to read because there's less noise, and I find the literal vector notation helps break up the code visually. Which brings me to the second thing I like, having literal vector, set, and map notation. I find it makes code more legible and helps see what's going on in a function.

The next thing I really like, that Clojure introduces, is destructuring. You can take any arbitrary data structure and read it backwards. Here are a few examples of what I'm talking about:

    (def {:a [1 2 3] :b {:c 4} :d 5})

    (defn foo [{a :a b :b}]
      (println a b))

    (defn bar [{:keys [a b d]]
      (println a b d))

    (defn baz [{[a b c] :a {d :c} :b e :d}]
      (println a b c))

this also works in let statements, and again I find that it improves readability, especially in larger programs. While a minor nitpick I also like the naming conventions in Clojure standard library better. Names such as car and cdr are archaic in my opinion.

why you shouldn't jump through loops

Sun, 17 Jun 2012 00:00:00 +0000

In Java passing logic as a parameter requires an inordinate amount of work and it's never the first choice to do so. So in most cases you're better off just writing a loop and doing the null check in it. Let's look at a concrete example of what I'm talking about here. Let's say we want to filter collections based on a predicate. The standard way you would do that in Java is to write a loop:

public static List<Integer> filterEven(Collection<Integer> col) {
    if (null == col) return null;
    List<Integer> result = new LinkedList<Integer>();
    for (Integer i  : col) {			
        if (i % 2 == 0) result.add(i);			
     }		
     return result;
}

then if later I need to filter odd numbers I'll probably write another loop that looks almost identical except for the actual test. Obviously, the looping logic should be abstracted here, but let's look at what's involved in doing that in Java:

public interface Predicate<T> {
    public boolean matches(T t);
}

public class EvenPredicate implements Predicate<Integer> {
	
    public boolean matches(Integer i) {
 	return i % 2 == 0; 
    }			
}

import java.util.Collection;
import java.util.LinkedList;
import java.util.List;

public class Filter {

    public static <T> List<T> filterCollection(Collection<T> col, 
                                          Predicate<T> predicate) {
        List<T> result = new LinkedList<T>();
	    for (T t : col) {			
                    if (predicate.matches(t)) {
                        result.add(t);
                    }
            }		
            return result;
    }
}

That's a lot more work than just writing a loop, and unless you saw this pattern many times you probably wouldn't consider doing it. Now let's compare this to a language like Clojure, where I would use a higher order function and pass in the matcher without having to do any preliminary setup:

(filter even? (range 10))

what if I wanted to write a loop to do that

(loop [nums (range 10)
       even-nums []]
    (if (empty? nums)
        even-nums
        (recur (rest nums) 
                  (if (even? (first nums)) 
                     (conj even-nums (first nums)) even-nums))))

all of a sudden the situation is reversed, it's a lot more code to do explicit looping, and it's trivial to use a higher order function to do this task. So the language encourages you to write code through function composition by design. Being able to easily separate iteration from the logic applied inside it means that we can write code that's shorter, cleaner, and less error prone.

I don't need a method to function

Tue, 05 Jun 2012 00:00:00 +0000

Instance methods are always associated with a particular object that may or may not exist. This means that before we can call a method we must first check if an object is null. This becomes especially tedious if you have nested objects. For example if we have a following situation:

    users.getUser("foo").getAddress().getStreet();

the above code would be unsafe, since every single method call could potentially lead to a null pointer. This means that we have to instantiate and check each object individually:

    String street = null;
    User user = users.getUser("foo");
    if (null != user)
       Address address = user.getAddress();
       if (null != address)
           street = address.getStreet();

Not only is this tedious and error prone, but it's also one more thing that you actively have to think about.

Let's compare this situation to the functional approach. In a functional language functions exist independent of data, much like static methods in OO. This means that we can't get a null pointer while calling a function. The author of the function can do all the error checking in the function once, and the user does not need to worry about it. When you chain such functions together, the null values can bubble up as the result:

    (:street (:address (:foo users)))

This code will not throw any null pointer exceptions, and instead a null value will be returned. It has less noise, it's less error prone, and it's easier to read.

subject to change

Thu, 03 May 2012 00:00:00 +0000

The OO world view requires us to classify data in order to work with it. For example if we're talking about a person, we might create a Person class and add some fields to it, such as age, name, and etc. Then we'll create instances of this class and use them in our program.

The problem with this approach is that the classification is only meaningful within a particular context. The classification is not inherent to the data itself, but rather it's a transient view of the data at a specific time in a particular domain.

When we create a class we make assumptions about the context in which the data will be used. These assumptions are often incomplete, and even when they are, the nature of the problem can change over time. The requirements may change, new requirements might come up, or we might have simply misunderstood the problem when we designed our classes.

The way OO deals with this is by remapping the classes to a new domain. We might extend the class, write a wrapper, or use an adapter pattern to bridge the contexts. But this is solving a problem we ourselves have introduced by assigning a permanent classification to our data.