Plingdollar

Scrubbing Up

Wed, 11 Mar 2026 00:00:00 +0000

One of the key features that makes Scheme such a powerful language is the support for powerful macro transformations. A Scheme program goes through a stage of syntax expansion before the program is bound. Many syntax constructs in Scheme programs are in fact macros. So far in Feersum our macro support has been lacking a core concept however: hygiene. In this post I aim to outline the shape of the new macro system and how it will provide support for hygienic macro expansion.

In their 1988 paper “Syntactic Closures” Allan Bawden and Jonathan Rees wrote:

One way to avoid capture problems (like or’s problem with temp) is to use names so obscure that the macro’s client is unlikely to discover them accidentally.

If we inspect the current source code of Feersum we see that we are in fact being called out by a pair of computer scientists from the 80s:

(syntax-rules ()
  ((or) #f)
  ((or test) test)
  ((or test1 test2 ...)
   (let ((|90a3b246-0d7b-4f47-8e1e-0a9f0e7e3288| test1))
     (if |90a3b246-0d7b-4f47-8e1e-0a9f0e7e3288| |90a3b246-0d7b-4f47-8e1e-0a9f0e7e3288| (or test2 ...)))))

The only real response to such a cutting criticism is to actually do it properly. In the rest of this post I’ll try and summarise the general approach discussed by Bawden and Rees and sketch an implementation of Syntactic Closures for a toy language.

Environmental Issues

When we evaluate a program an interesting interaction takes place between the names in our source text and the variables that they refer to. It’s so natural that it often goes unnoticed. Let’s refresh ourselves just the same. Say we have a simple lambda calculus program expressed in s-expression form:

((lambda x x) 10)

In such a program we can trivially see the result should evaluate to 10. The way this works can be modelled out through maths as a series of reduction rules. Seen as we are talking about programming here however the other option is to “evaluate” this expression. To do this we recurse into each nested form carrying an execution environment with us. This environment is a mapping from the names in our program to the values we store for them.

In our trivial program there is only one name, x, and only one location that this refers to; the argument to our lambda. In a more complex program however the name x may refer to different locations depending on the execution scope within the program. Take for example this updated program which triples a value:

((lambda x (+ x
            ((lambda x (+ x x)) x))) 10)

Here in the outer lambda x is bound to the outer lambda’s argument. In the inner lambda it is instead bound to the inner lambda’s argument. This is a concept commonly known as “shadowing”.

A subtle variation introduces the key third concept we need to understand: closures. If instead of using the same name x to refer to both parameters we instead named the inner one y the outer x is no longer shadowed. This allows us to capture, or close over the outer x in the body of the inner lambda.

(((lambda x (lambda y (+ x y))) 100) 50)

In this program the inner lambda “remembers” information about the execution environment in which it was created. In the Syntactic Closures paper an orthogonal approach is laid out that allows syntax to remember the syntactic environment in which it was created to prevent names accidentally shadowing values in their new locations within the expanded program once syntax transformation is performed.

The Syntax Domain

Just as programs being executed have an execution environment and runtime closures which are used to bind meanings and context to names the same can be done in the syntax domain. This is the core abstraction proposed in the Syntactic Closures paper.

Let’s imagine our language has another special form as well as lambda called def-syntax. This form is responsible for defining a simple syntax transformer, or “macro”. Our macros will take a list of arguments and expand into some arbitrary syntax. The two sides of each rule within our syntax transformer are the pattern arguments on the left, and the template on the right:

(def-syntax or
	((_ a) a)
	((_ a b) ((lambda x (if x x b)) a)))

Here our definition of or has two cases. For a single value we simply expand to that value. For two values we expand to a lambda which binds the first value to the argument x. This then allows us to check and return x without evaluating it twice.

When our compiler encounters this special form at syntax expansion time, which takes place before the evaluation of the program, we perform some actions which feel familiar. First we convert the body of the def-syntax into a transformer and then we record in our syntax environment that the name or now refers to that macro. Within the remaining scope of the program future references to or will be treated as applying this transformer.

When we see a reference to the macro, such as (or 100 200) we first check each rule in turn to find a match based on the pattern. Once we have found a match we bind the values a -> 100 and b -> 200 in the syntax environment and move on to expand the template. From now on references in the template to the names a or b will be replaced, or substituted, with their respective values from the syntax environment.

This substitution on its own however is not sufficient to provide hygienic expansion. To do this we need to introduce two new concepts. The first is the idea of an identifier. An identifier is a pairing of a name and an opaque “stamp”. In our examples we’ll use an incrementing integer. These identifiers allow us to tell the various different xs apart in our syntax. Thinking back to our nested example from earlier if we annotate it with the identifiers in place of the names it might become:

((lambda x.1 (+ x.1
            ((lambda x.2 (+ x.2 x.2)) x.1))) 10)

Here we can see that the “outer x” and “inner x” are two logically different identifiers. This transformation is analogous to alpha renaming.

In addition to binding names to identifiers we can also introduce the idea of a Syntactic Closure. The idea here is that when a piece of syntax “escapes” from its place within the static structure of a program and identified scoping we need to capture the state of the syntax environment at that point so that it can be restored later; just as calling a lambda allows it to access the “remembered” context that it captured in the execution environment.

A syntactic closure contains two things:

A syntax environment. This is a mapping from names to their “syntactic item”. This could be the fact that x refers to the identifier x.1 or it could be that or refers to the transformer for our or macro.
A piece of syntax to apply it to.

With these two pieces in place we can now sketch out how a language can perform hygienic expansion of the or macro using Syntactic Closures.

The `Cst` Tree

We begin from a simple tree node type which directly encodes the concrete syntax of a simple language:

type Cst =
  | Symbol of string
  | Literal of int
  | Form of Cst list

This encodes the raw syntactic structure of our language. This will not be sufficient to model our understanding of the syntax tree outlined above however. To solve this we introduce a second parallel stx tree. Once our parser has produced the Cst we use the illum function to walk the concrete syntax and produce a new tree where each raw symbol has been given its initial binding to an identifier. First our tree structure.

type Ident = { Name: string; Stamp: int }

type Stx = 
  | Literal of int
  | Ident of Ident
  | Form of Stx list
  | Closure of Stx * StxEnv

and StxEnv = Map<string, StxBinding>

and StxBinding =
  | Lam
  | Var of Ident
  | Macro of Transformer

and Transformer = Stx -> StxEnv -> Stx

The illum function then takes our Cst and maps all names to identifiers with the stamp 0:

let public illum = function
  | Cst.Symbol s ->
    Stx.Ident({ Name: s; Stamp: 0})
  | Cst.Literal l -> Stx.Literal l
  | Cst.Form f ->
    f
    |> List.map (illum)
    |> Stx.Form

We then define a pair of functions to help us deal with syntax environments. The first takes a piece of syntax and wraps it into a Stx.Closure. The second resolves a given name in the StxEnv and returns the StxBinding. If no entry exists in our syntax environment for a name we return the ‘default’ meaning for that name. For our special form lambda this is our token Lam binding, for all other names we represent them as “free variables”.

let close (stx: Stx) (defEnv: StxEnv) : Stx =
  StxClosure(stx, defEnv)

let resolve (id: Ident) (stxEnv: StxEnv) : StxBinding =
  match stxEnv.TryFind id.Name with
  | Some binding -> binding
  | None ->
    match id.Name with
    | "lam" -> Lam
    | _ -> Var id

With these two functions in place we can begin to implement our expand method. It takes a piece of syntax and a syntax environment and produces an optional replacement piece of syntax and a new environment. The first output is option as some syntax items, such as def-syntax will have no representation in the final tree. In this case we would return None and elide it from the final expanded form of the program. The second output represents the new, potentially expanded, environment in which further syntax should be expanded. Using these two items we can then perform a mapFold over a sequence of Stx items in order to expand a program.

let rec public expand (stx: Stx) (stxEnv: StxEnv) : Stx option * StxEnv =

For the first few cases our syntax expansion is relatively trivial. Empty forms expand to themselves, and so do literals:

  match stx with
  | Stx.Literal _
  | Stx.Form [] -> (Some stx, stxEnv)

For identifiers we need to resolve them to their binding in the syntax environment. If they are not bound to a variable this is an error. References to syntax items should only be at the head of a form in order to apply them.

  | Stx.Ident id ->
    match (resolve id stxEnv) with
    | Var id -> (Some Stx.Ident(id), stxEnv)
    | _ -> failwith "Unexpected syntax item in value position"

The interesting cases arise when we encounter a non-empty form. The head of the form determines what kind of expansion needs to take place. If the head is an identifier we resolve it in the syntax environment, just as before, and then dispatch on the result:

  | StxForm (head :: args, f) ->
    match head with
    | StxIdent(id, sym) ->
      let resolved = resolve id stxEnv
      match resolved with
      | Macro transformer ->
        (Some (transformer stx stxEnv), stxEnv)
      | Lam -> (Some (expandLam head args f stxEnv), stxEnv)
      | DefSyn -> expandDefSyn args stxEnv
      | Var _ ->
        let expanded =
          StxForm(List.map (expandOne stxEnv) (head :: args), f)
        (Some expanded, stxEnv)
    | _ ->
      let expanded =
        StxForm(List.map (expandOne stxEnv) (head :: args), f)
      (Some expanded, stxEnv)

There are four cases for a resolved head identifier. If it’s a Macro we invoke the transformer directly — we’ll see how transformers work shortly. If it’s the special form Lam we delegate to expandLam. If it’s a DefSyn we delegate to expandDefSyn, which returns None and an updated environment rather than a piece of expanded syntax. For anything else — an ordinary variable reference in head position, or a non-identifier head — we simply recurse into the sub-forms.

The expandOne helper is a thin wrapper that enforces that we only call expand in positions where a result is required:

and private expandOne (stxEnv: StxEnv) (stx: Stx) : Stx =
  match expand stx stxEnv with
  | Some result, _ -> result
  | None, _ -> failwith "def-syn not valid in expression position"

Renaming Binders

When expanding a lam form we encounter the core of the hygiene mechanism. The parameter identifier needs to be given a fresh stamp so that it cannot accidentally shadow or be shadowed by identifiers with the same name introduced elsewhere — either by the user or by a macro.

and expandLam head args low stxEnv =
  match args with
  | StxIdent(id, s) :: body ->
    let (innerEnv, renamedId) = rename stxEnv id
    let expandedBody = List.map (expandOne innerEnv) body
    StxForm(head :: StxIdent(renamedId, s) :: expandedBody, low)
  | _ -> failwith "Invalid syntax for lam: expected (lam <id> <body>)"

The rename function produces a fresh identifier with a new stamp for the binding site and adds a Var mapping to the syntax environment so that any reference to the same name within the body will resolve to that fresh identifier. This is the key to preventing accidental capture: once a binder is renamed, all references within its scope pick up the renamed identifier through resolve. References originating from outside the lambda body — for example, free variables in a macro template — will carry a different stamp from when they were illuminated or closed over, and so will resolve differently.

Thinking back to our nested lambda example from earlier, expandLam is what ensures the two x binders end up as x.1 and x.2 rather than competing for the same name.

Defining Syntax

The expandDefSyn function handles the def-syntax special form. It parses each rule, builds a transformer, and returns an updated syntax environment:

and private expandDefSyn
    (args: Stx list) (stxEnv: StxEnv) : Stx option * StxEnv =
  match args with
  | StxIdent(macroName, _) :: ruleStxs ->
    let rules = List.map parseRule ruleStxs
    let transformer = makeSynTransformer rules stxEnv macroName.Name
    let newEnv = Map.add macroName.Name (Macro transformer) stxEnv
    (None, newEnv)
  | _ -> failwith "Invalid def-syn form: expected (def-syn <name> <rule>...)"

Note that expandDefSyn returns None for the syntax item itself — a macro definition produces no output in the expanded tree, only a change to the environment.

Parsing a rule simply extracts the pattern argument identifiers and the template from the rule’s structure:

and private parseRule (ruleStx: Stx) : MacroRule =
  match ruleStx with
  | StxForm([StxForm(StxIdent _ :: patArgs, _); template], _) ->
    let idents = patArgs |> List.mapi (fun i pat ->
      match pat with
      | StxIdent(id, _) -> id
      | _ ->
        failwith (
          $"Invalid pattern in macro rule at position {i}: " +
          $"expected an identifier, got %A{pat}"))
    { PatArgs = idents; Template = template }
  | _ ->
    failwith (
      $"Invalid macro rule: expected ((name pat...) template), " +
      $"got %A{ruleStx}")

The Transformer and the Syntactic Closure

The real payoff arrives in makeSynTransformer. This is where the definition-time environment is captured — forming the syntactic closure — and where it is later used to wrap the expanded template:

and private makeSynTransformer
    (rules: MacroRule list) (defEnv: StxEnv)
    (macroName: string) : Transformer =
  fun (callStx: Stx) (useEnv: StxEnv) ->
    match callStx with
    | StxForm(_ :: callArgs, _) ->
      let tryRule (rule: MacroRule) =
        match matchPatternArgs rule callArgs useEnv with
        | Some bindings ->
          let substituted = applyTemplate rule.Template bindings
          Some (StxClosure(substituted, defEnv))
        | None -> None
      match List.tryPick tryRule rules with
      | Some expanded ->
        match expand expanded useEnv with
        | Some result, _ -> result
        | None, _ ->
          failwith $"macro '{macroName}': def-syn in template is not valid"
      | None -> failwith $"No matching rule for macro '{macroName}'"
    | _ -> failwith $"Invalid macro call form for '{macroName}'"

The transformer closes over defEnv — the syntax environment at the point the macro was defined. When the transformer is later invoked, it receives useEnv — the environment at the call site. Pattern matching happens against useEnv, so the arguments from the call site are bound correctly. The template however is wrapped in a StxClosure carrying defEnv. When expand later encounters this closure it will expand the template’s contents in the definition-time environment, not the call-site environment.

This is the final case in expand — the one we have been building towards:

  | StxClosure(stx, env) ->
    let (result, _) = expand stx env
    (result, stxEnv)

When we hit a syntactic closure we expand its inner syntax using the captured environment, then return the result into the outer environment unchanged. The two environments never mix. An identifier x introduced by the macro template can only ever resolve to what x meant where the macro was written — not to whatever x might mean at the call site. The Bawden and Rees paper would approve.

Returning to our motivating example, when (or 100 200) is expanded the template ((lambda x (if x x b)) a) is wrapped in a closure over the definition-time environment. When that closure is later expanded, the x in the template resolves to a freshly stamped identifier created during expandLam — one that is entirely distinct from any x the caller might have in scope. No GUIDs required.

Much Ado About Mutton

For the full code of this toy example, including handling for definitions as well as a bind pass which assigns storage locations to items you can check out the full iwillspeak/Mutton on GitHub. For the final implementation of hygiene in the Feersum compiler see the (currently work-in-progress) Hygenic Macro Expansion PR.

On the Same Page

Tue, 28 Jun 2022 00:00:00 +0000

At the end of 2021 GitHub pages deployments moved to using GitHub actions. When a site is deployed you can see the jobs running to generate your site, package it up ready to be deployed, and push it to the GitHub Pages service. Lets try using these actions in a custom GitHub Actions workflow to build and deploy a site without using Jekyll.

There are two main steps in the deployment of a site to GitHub pages:

An artifact is generated containing the contents of the site to be deployed. This is carried out by the actions/upload-pages-artifact
The artifact is published to the GitHub Pages API. This is the job of actions/deploy-pages.

First up lets start by creating a skeleton workflow to build our site:

name: Build Docs

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Install Docket
      run: |
        mkdir -p _site/
        echo "<h1>Hello Custom Pages World" > \
           _site/index.html

This creates a simple job which builds what can generously be called a website into the _site/ directory. With our custom site build out of the way the next step is to package it as an artifact. We can do that with the following step added to the build job:

    - name: Push artifact
      uses: actions/upload-pages-artifact@v0

This should produce an artifact github-pages as part of the build. Before we can deploy this to Pages there’s a little setup we need to do in the UI. Head to the Settings tab and enable pages for some placeholder branch such as gh-pages-stub. This makes your site available to deploy to. To allow deployments from other branches go to the Environments tab and enable branch protections as outlined in this GitHub issue. With those steps out of the way we can move on to adding our deployment job.

To deploy to Pages we need a few permissions. Add the following to the root of the Actions yaml:

permissions:
  contents: read
  pages: write
  id-token: write

This makes the required authentication tokens available to the pipeline so that the deployment action can run. All that now remains is to add a deployment job to our pipeline:

  deploy:
    name: Deploy
    runs-on: ubuntu-latest
    needs: build
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    steps:
      - name: Deploy GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v1

The key parts here are the needs: build to force it to wait for our build step so that the artifact is available, and the url: ${{ steps.deployment.outputs.page_url }} in the environment definition that ties the deployment to the correct Pages site.

With all that in place we now have a working Actions workflow building a simple site and deploying to Pages. Check out the full code for this post to see it in action, or gaze upon the output in wonder.

Into the Dark

Sun, 26 Jun 2022 00:00:00 +0000

I’ve finally dragged this blog into the 21st century with the addition of Dark Mode. In this post I’ll walk through some of the main tasks, and dig into the approach I took.

The first step was to get rid of colours hard-coded into the CSS. The old style-sheet was full of #FFFs and #222s along with more esoteric tints, tones, and shades. I worked colour by colour to pull them into a single block at the top of the sheet in the :root scope:

:root {
  /* COLOURS */
  --col-accent: #16b1c3;
  --col-accent-dimmed: #0d7581;
  --col-light: #FFF;
  --col-light-subtle: #FBFBFB;
  --col-light-dimmed: #999;
  --col-dark: #222;
  --col-dark-subtle: #333;
  --col-dark-dimmed: #666;
}

My theme was based around a single accent colour of #16b1c3, and a dimmed variant of that colour. The remaining parts of the site were either wites, or off whites, which I classed as ‘light’ colours; or near blacks which I classed as dark colours. With the colours extracted to the top of the file and their uses replaced with var(--col-light) and so on the next step was to introduce a seam of abstraction to separate the concrete light and dark colours from their use in the rest of the style-sheet. For this I introduced aliases for each colour prefixed with either --col-fg for foreground (dark) colours, or --col-bg for background (light) colours.

:root {
  /* BASE colour scheme, with no preferences or overrides */

  --col-bg: var(--col-light);
  --col-bg-dimmed: var(--col-light-dimmed);
  --col-bg-subtle: var(--col-light-subtle);
  --col-fg: var(--col-dark);
  --col-fg-dimmed: var(--col-dark-dimmed);
  --col-fg-subtle: var(--col-dark-subtle);
}

Instead of directly referencing light or dark colours most places in the style were changed to reference a --col-fg- or --col-bg variant instead:

body {
  background: var(--col-bg) none;
  color: var(--col-fg-subtle);
}

Now all that was left was to change the foreground and background colours depending on the user’s preference. The first step in this was to respect the preference exposed from the browser. We can query this using a CSS meia query, just as we would do for device width. Given our base colour scheme is light we only need to query to see if the user has a preference for dark colours. If the user has no preference, or prefers lighter colourschemes we’re fine to use our default scheme.

/* Apply a dark color scheme */
@media (prefers-color-scheme: dark) {
  :root {
    --col-bg: var(--col-dark);
    --col-bg-dimmed: var(--col-dark-dimmed);
    --col-bg-subtle: var(--col-dark-subtle);
    --col-fg: var(--col-light);
    --col-fg-dimmed: var(--col-light-dimmed);
    --col-fg-subtle: var(--col-light-subtle);
  }
}

With that set the site was now dark-mode compatible. It respected the user’s stated preference for light or dark colours. We can go one better though and allow the user to change the colour scheme on the fly with JavaScript. I begun by adding a stub HTML element to the site’s layout which can be latched onto by the JS to provide progressive enhancement:

<div id="dark-mode-placeholder"></div>

The core of the JS then revolves around querying the same prefers-color-scheme: dark property we did from CSS, and writing a data property at the root of the document if we want to override it:

const prefersDarkMode =
    window.matchMedia("(prefers-color-scheme: dark)");

// Toggle the ligth / dark mode override.
const toggleMode = () => {
    const dataSet = document.documentElement.dataset;
    const currentMode = dataSet.colorMode;

    if (currentMode === undefined) {
        if (prefersDarkMode.matches) {
            dataSet.colorMode = "light";
        } else {
            dataSet.colorMode = "dark";
        }
        localStorage.setItem(
            "stashed-theme", dataSet.colorMode);
    } else {
        delete dataSet.colorMode;
        localStorage.removeItem("stashed-theme");
    }

    return dataSet.colorMode;
}

// Restore the theme from local storage if stashed
const stashedTheme =
    localStorage.getItem("stashed-theme");
if (stashedTheme) {
    document.documentElement.dataset.colorMode =
        stashedTheme;
}

// Add a button to the DOM to allow toggling dark mode.
const darkModeButton = document.createElement("a");
darkModeButton.addEventListener("click", () => {
    toggleMode();
});
darkModePlaceholder.appendChild(darkModeButton);

The final step was to make the CSS respect these new colour preference properties set by the JavaScript. To do this we need to handle both the cases where the user prefers light and has toggled to dark, and the user preferring dark and switching to light mode.

/* Dark mode override */
html[data-color-mode="dark"] {
  --col-bg: var(--col-dark);
  --col-bg-dimmed: var(--col-dark-dimmed);
  --col-bg-subtle: var(--col-dark-subtle);
  --col-fg: var(--col-light);
  --col-fg-dimmed: var(--col-light-dimmed);
  --col-fg-subtle: var(--col-light-subtle);
}

/* Light mode  override */
html[data-color-mode="light"] {
  --col-bg: var(--col-light);
  --col-bg-dimmed: var(--col-light-dimmed);
  --col-bg-subtle: var(--col-light-subtle);
  --col-fg: var(--col-dark);
  --col-fg-dimmed: var(--col-dark-dimmed);
  --col-fg-subtle: var(--col-dark-subtle);
}

That’s it! With the colour variables switched either based on the user’s media preference, or via a data-color-mode property at the root node dark modification is complete. The last remaining step for this site was to update the styles for code blocks to ensure that colours were legible in both light and dark environments.

I am by no means a frontend expert but I enjoyed the process of tweaking CSS and refactoring it as if it were backend code. I’m happy with the resulting style sheet. I hope my travels prove useful to others.

Red Green Syntax Trees - an Overview

Wed, 24 Nov 2021 00:00:00 +0000

It’s no secret that I’m a big fan of a good parser. One of the key parsing techniques that I have tried to focus on in the past is “infallible parsing”. The idea that for any input the parser should always generate some output. This is important to try and give as accurate a set of errors as possible when malformed or partial code is being analysed. An idea that builds on this is lossless syntax trees. In this post I’ll discuss a high level overview of a kind of lossless tree, discuss where it came from, and how it can help when parsing and compiling.

The idea of red / green syntax trees initially came from the Roslyn C# compiler. It was adopted by Swift in their libsyntax implementation. The Rust Analyzer project built further upon this with the Rowan crate. Rowan extends the model of red-green syntax trees with the idea of dynamically typed nodes.

The core of the idea is to represent parsed syntax as two distinct trees. The base tree is the green tree. This represents pieces of concrete syntax, such as the character 1, the operator +, or the expression 1 + 1. The green tree contains no position information. Each node in the tree is immutable meaning that portions of the tree with the same shape can share nodes.

For example given the 1 + 1 expression we could represent it as the following green tree:

           Node BIN_EXPR
                 |
     +-----------+-----------+
     |           |           |
 Token '1'   Token '+'   Token '1'

But we could equally share the token’s for 1 and use the following tree:

           Node BIN_EXPR
                 |
     +-----------+-----------+
     |           |           |
     |       Token '+'       |
     |                       |
     +------ Token '1' ------+

To the outside observer there should be no difference. This sharing of sub-trees is more useful where larger parts of the tree can be shared, such as repeated uses of the same path or calls to the same method.

An simplified definition of a tree might look like the following in F#:

type GreenToken =
	{ Kind: SyntaxKind
	  Text: string }
and GreenNode =
	{ Kind: SyntaxKind
	  Width: int
	  Children: Choice<GreenNode, GreenToken> list }

In the green tree each node has no specific location, only a width. The width of a token is the width of the text it contains. The with of nodes is cached on the node. It is the sum of the widths of its children.

On its own the green tree represents abstract syntactic elements. To provide a richer view over the top the red tree is the a layer over the top similar to the following:

type SyntaxToken =
	{ Offset: int
	  Parent: SyntaxNode option
	  Green: GreenToken }
and SyntaxNode =
	{ Offset: int
	  Parent: SyntaxNode option
	  Green: GreenNode }

Each node is extended with two things: a reference to the parent node, and an absolute offset of the start of this node form the beginning of the source text. The offset paired with the widths from the green nodes provides a source range for each element.

Nodes in the red tree all have a reference to a parent node which can be used to traverse the tree and find siblings. A red node fabricates it’s children lazily when a traversal descends into the node. This means we only pay the cost for the portion of the red tree we actually use. Once we no longer reference a portion of the red tree it is free to be garbage collected.

I’m currently experimenting with these ideas for a re-write of the Feersum scheme Compiler’s parser. See the Firethorn project for the red / green tree implementation.

For more details on this approach see:

Another notable mention, although logically distinct, is the Oil shell’s lossless syntax tree.

Granola Slices

Fri, 02 Jul 2021 00:00:00 +0000

A quick and easy recipe for the flapjack I miss from the office.

Ingredients

Porridge oats: 250 grams
Strawberry Granola: 10 grams
Cake Fruit: 30 grams
Pumpkin Seeds: 10 grams
Butter (melted): 150 grams
Sugar: 150 grams
Golden Syrup: 2 tbsp

Method

Pre-heat the oven to 180C (fan). Line a baking tray with greaseproof paper.

Combine the dry ingredients in a large bowl and set aside.

Place the butter and sugar into a microwaveable jug. Microwave the mixture for bursts of 30 seconds and stir until the butter has melted and combined with the sugar.

Pour the hot sugar/butter mixture into the dry ingredients. Add the golden syrup and combine well.

Tip out onto the baking tray and firm down with the back of a spoon or spatula until evenly distributed.

Bake for 8 to 12 minutes, or until golden brown on top.

Leave to stand for 10 minutes before dividing into portions.

Error Recovery in FParsec

Thu, 10 Sep 2020 00:00:00 +0000

An interesting discussion on error recovery with the nom parser combinator in Rust has been sitting on my “To Read” list for a few months. After re-reading it a few times I have been inspired to try out the approach in that article using the FParsec library.

Just as in that example we will attempt to parse the language defined by the following Parser Expression Grammar (PEG):

expr    ← sum
sum     ← product (('+' / '-') product)*
product ← value (('*' / '/') value)*
value   ← [0-9]+ / '(' expr ')'

The grammar is simple enough, and translating it into a parser combinator is fairly straightforward. Each non-terminal, on the left of the ←, becomes a parser function. Each production on the right is mapped into the appropriate FParsec primitives:

let expr, exprRef = createParserForwardedToRef()

let value =
    (pint64  |>> Ast.Value ) <|>
    between (pchar '(') (pchar ')') expr

let product =
    let op = pchar '*' <|> pchar '/'
    (value .>>. (many (op >>. value)))
    |>> binNode Product

let sum =
    let op = pchar '+' <|> pchar '-'
    (product .>>. (many (op >>. product)))
    |>> binNode Sum

exprRef := sum

So far so good. Parsing valid inputs produces a ParseResult.Ok containing our syntax tree and all is well.

'123' ~> Value 123L
'1*2' ~> Product (Value 1L,Value 2L)
'(1+2)*3' ~> Product (Sum (Value 1L,Value 2L),Value 3L)

The problem arises when the parser fails.

Error in Ln: 1 Col: 2
(*1)+2)
 ^
Expecting: integer number (64-bit, signed) or '('

FParsec has done an excellent job of making an error message for us. This is a good example of how a parser combinator library can make knocking together a quick parser a breeze. There are a few issues however:

It would be super nice if we always got some form of parse tree from our parser.
Our example (*1)+2) has more than one syntax error in it.

Parsers are generally used in two places. Either as part of a batch program performing a one-off compilation, or by a language server or IDE performing interactive syntax analysis. In either case, we want to provide as much information back to the programmer as possible. If we can get our hands on some form of syntax tree we can still provide a gracefully degraded IDE experience. Recognising all the syntax errors we can at once is helpful to the programmer who is in a tight edit -> compile loop.

The plan to solve this is simple.

The parser should always succeed.

Instead of the current output of the parser, which is just a SyntaxNode, we instead would like it to return a pair of SyntaxNode * Diagnostic list where Diagnostic is some structure containing a Position and an error message.

To achieve this we will take advantage of FParsec’s UserState to build our diagnostic list, and we will create some new parser combinators to backtrack on failure and emit diagnostics where needed.

First lets create a State type to be used by our parser:

type Diagnostic = Diagnostic of Position * string

type State =
    { mutable Diagnostics: Diagnostic list }

Simple enough, we have a mutable list of diagnostics. Let’s add a member function to encapsulate emitting a diagnostic:

    member s.EmitDiagnostic pos err =
        let diag = Diagnostic(pos, err)
        s.Diagnostics <- diag::s.Diagnostics

Nice. Now let’s create some parser combinators that use them. First we introduce the expect combinator. It takes a parser, and an error message. If the parser succeeds we wrap its return in a Some. If it fails we emit a diagnostic and return None.

let expect (p: Parser<'a, State>) e =
    let raiseErr (stream: CharStream<State>) =
        stream.UserState.EmitDiagnostic stream.Position e
        Reply(None)
    attempt p |>> Some <|> raiseErr

Our next is expectSyn. This is a thin wrapper over expect which maps failure to a new SyntaxNode.Error kind.

let expectSyn (p: Parser<SyntaxNode, State>) err =
    expect p err |>> Option.defaultValue SyntaxNode.Error

With this in place we can now start to re-write our parser to emit diagnostics and continue in the case of some errors. The general approach here is to wrap an expect over any right hand parts of a production that should not fail.

let sum =
    let op = pchar '+' <|> pchar '-'
    let err = "expected expression after operator"
    (product .>>.
        (many (op >>. expectSyn product err)))
    |>> (binNode Sum)

In our sum parser we don’t wrap the op, because we expect that to fail when there isn’t a + or -. If there is an operator however we don’t expect the right hand side to fail, so we use expectSyn to ensure it doesn’t. A similar transform is needed for the product parser. With this in place our parser succeeds even on invalid inputs:

'1+' ~>
  ([Sum (Value 1L,Error)],
   [Diagnostic (
     ("test", Ln: 1, Col: 3),
     "expected expression after operator")])

Here the right-hand side of our Sum expression has been stubbed out with an Error. We have a diagnostic too, with a location pointing to where the problem lies.

Buoyed on by this success, let’s add some expects to the value parser:

let value =
    let opn = pchar '('
    let close = expect (pchar ')') "missing closing ')'"
    let body =
      expectSyn expr "expected expression after ("

    (pint64  |>> Value ) <|> between opn close body

We use a plain expect for the closing ), and expectSyn for the body of the parenthesised expression. Once we have seen an opening ( neither of these are optional.

Now we can recognise errors in even more places:

'(1' ~>
  ([Value 1L],
   [Diagnostic (
     ("test", Ln: 1, Col: 3),
     "missing closing ')'")])
'()' ~>
  ([Error],
   [Diagnostic (
     ("test", Ln: 1, Col: 2),
     "expected expression after (")])
'(' ~>
  ([Error],
   [Diagnostic (
     ("test", Ln: 1, Col: 2),
      "missing closing ')'");
    Diagnostic (
     ("test", Ln: 1, Col: 2),
     "expected expression after (")])

Nice! We get two diagnostics for the last one: one for the missing expression body, and one for the closing ).

The last place we need to handle failure is at the root level of the parser. If the parser encounters something that doesn’t look much like an expression at all we need a way to record that and skip over it. To do this we introduce a new error parser. Its job is to skip over a single character, and record a diagnostic:

let error (s: CharStream<State>) =
    match s.ReadCharOrNewline() with
    | EOS -> Reply(ReplyStatus.Error,
                   expected "valid expression character")
    | ch ->
        sprintf "Unexpected character %c" ch
        |> s.UserState.EmitDiagnostic s.Position
        Reply(Error)

This is a little more complex because we need to handle the case that there are no characters left to skip over. In that case the parser should return ReplyStatus.Error so that FParsec can move on and match the end of file. In other cases we create a diagnostic message and return our SyntaxNode.Error value.

We add this in to our root parser as the last value in the choice. Because FParsec considers each choice in turn until one succeeds our error parser will only be called upon if the real parsers weren’t able to make any progress.

let parser = many (expr <|> error) .>> eof

With all that in place let’s try parsing our original malformed expression again:

'(*1)+2)' ~>
  ([Product (Error,Value 1L); Error; Value 2L; Error],
   [Diagnostic (("test", Ln: 1, Col: 8),
                "Unexpected character )");
    Diagnostic (("test", Ln: 1, Col: 5),
                "Unexpected character )");
    Diagnostic (("test", Ln: 1, Col: 2),
                "missing closing ')'");
    Diagnostic (("test", Ln: 1, Col: 2),
                "expected expression after (")])

Much better. We get some kind of parse tree at least, and the diagnostics point to the problems with the missing expression at 1:2. Where we fall down however is with the diagnostics for the missing opening (. This is as far as we come with this example though. Introducing these recovery expressions is as much an art as it is a science.

The full source code for these parsers is available on GitHub.

A Captivating Resolution

Sun, 06 Sep 2020 00:00:00 +0000

Compilers are generally split into three stages: syntax, semantic, and emit. The translation from syntax tree, produced by the first stage, to semantic tree, consumed by the latter stages, is called binding. Binding involves walking the syntactic representation, resolving variable references, and producing a tree based on the semantic structure of the program. For a single scope variable lookup is fairly simple. When dealing with lambdas that can capture values from an outer scope things get more complex. In this post I’ll walk through developing a simple binder that can handle lambda captures.

We will begin by assuming our input has already been parsed and is provided as a tree of SynNodes.

/// Syntactic node, as if from source code.
type SynNode =
    | Number of int
    | Define of string * SynNode
    | Load of string
    | Store of string * SynNode
    | Lambda of string * SynNode
    | Seq of SynNode list

Our bound tree will be made up of another union. We’ll begin by just supporting numbers:

/// Bound node, with captured variables resolved
type Bound =
    | Number of int

/// Bind the syntactic tree and produce a bound tree
let rec private bind = function
    | SynNode.Number n -> Bound.Number n
    | e -> failwithf "Error binding %A" e

Numbers are simple to bind as no resolution needs to happen.

Syntax	Bound
Number 123	Number 123

Next up is variable definitions. In this language we assume a variable needs to be defined before being used. To support this we need some place to store the variables in the current scope. To keep track of the defined variables along with other state as we perform the bind we introduce a BindCtx type:

/// Locals are indices into some arbitrary array of
/// storage. Locals begin their indexing again at 0
/// when a new lambda is defined.
type Local = Local of int

/// Binder context, flowed through the bind to keep
/// track of the current state.
type BindCtx =
    { mutable Locals: string list }
    
    /// Introduce a new local definition.
    member ctx.Define id =
        let idx = ctx.Locals.Length
        ctx.Locals <- id::ctx.Locals
        Local idx

With that in place we can now bind definitions:

type Bound =
    | Store of Local * Bound
    // ...

let rec private bind (ctx: BindCtx) = function
    | SynNode.Define(id, init) ->
        let local = ctx.Define id
        Bound.Store(local, bind ctx init)
    // ...

Defining variables introduces a new local to the binder context, and then represents the initialisation in the bound tree as a plain Store operation.

Syntax	Bound
Define ("foo",Number 123)	Store (Local 0,Number 123)
Define ("foo",Define ("bar",Number 456))	Store (Local 0,Store (Local 1,Number 456))

To bind plain loads and stores we need to look up the ID in the current context, and emit a bound operation that reads or writes to that location. Looking up the ID is performed by searching for a local variable in the list of locals. Once we have the index we need to “flip” it to return the offset from the base of our stack of local variables:

type BindCtx =
    // ..

    /// Lookup the given `id` in the current binder
    /// context.
    member ctx.Lookup id =
        List.tryFindIndex (fun l -> l = id) ctx.Locals
        |> Option.map (fun idx ->
	        Local (ctx.Locals.Length - 1 - idx))

Now we can look locals up in the environment adding loads and stores to the bind method is fairly simple. We search for the id and raise an error if no definition is found for it:

type Bound =
    | Load of Local
    // ...

let rec private bind (ctx: BindCtx) = function
    | SynNode.Load id ->
        match ctx.Lookup id with
        | Some local -> Bound.Load local
        | None ->
          failwithf "Reference to undefined %s" id
    | SynNode.Store(id, expr) ->
        match ctx.Lookup id with
        | Some local -> Bound.Store(local, bind ctx expr)
        | None ->
          failwithf "Assignment to undefined local %s" id
    // ...

Syntax	Bound
Seq [ Define ( "foo", Number 234); Load "foo" ]	Seq [ Store ( Local 0, Number 234); Load (Local 0) ]

Next up it’s time to start work on the lambdas. In this language lambdas have a single argument and a body. The first change is to expand our idea of a variable to include arguments as well as lambdas:

/// Storage locations for variables.
type Storage =
    /// Locals are indices into some arbitrary array
    /// of storage. Locals begin their indexing again
    /// at 0 when a new lambda is defined
    | Local of int
    /// Load from the function's argument
    | Arg
    
type Variable =
    { Name: string
    ; Storage: Storage }

Lambdas will introduce a new scope on the stack. We will model this as a brand new BindCtx. Each nested context contains a reference back to the parent context. Instead of just a list of strings we now use our new Variable type.

type BindCtx =
    { Parent: BindCtx option
    ; mutable NextLocal: int
    ; mutable Locals: Variable list }
    
    // ..

    /// Create a derived contex for binding lambdas
    static member WithParent parent =
        { Parent = Some(parent)
        ; NextLocal = 0
        ; Locals = [] }

As well as our existing Define that needs updating to add a Variable to our Locals we add a DefineArg to introduce a definition for the lambda’s argument.

    /// Introduce a new local definition.
    member ctx.Define id =
        let storage = Storage.Local(ctx.NextLocal)
        ctx.Locals <- { Name = id
                      ; Storage = storage }::ctx.Locals
        ctx.NextLocal <- ctx.NextLocal + 1
        storage

    /// Define a lambda argument
    member ctx.DefineArg id =
        let storage = Storage.Arg
        ctx.Locals <- { Name = id
                      ; Storage = storage }::ctx.Locals

With all this in place binding non-capturing lambdas is fairly simple. A new context is created with the current context as its parent. We define its argument and then bind the body with that new context.

let rec private bind (ctx: BindCtx) = function
    // ...
    | SynNode.Lambda(formal, body) ->
        let lambdaCtx = BindCtx.WithParent(ctx)
        lambdaCtx.DefineArg formal
        bind lambdaCtx body |> Bound.Lambda

So long as the lambda doesn’t reference a variable outside its scope all should be good.

Syntax	Bound
Lambda ("x",Load "x")	Lambda (Load Arg)

To resolve variable references from parent scopes things get a little more complex. If a lambda references a variable from one of the enclosing scopes it is said to ‘capture’ or ‘close over’ that value.

To allow the nested lambda to share access of this captured value with other scopes we move the storage location from the function’s locals into some ‘environment’ structure. This extends the lifetime of the variable beyond the scope of the lambda it was defined in so references to it in child lambdas can outlive the initial call. Now if anyone wants to modify the value there’s a centralised location to use. In the original scope access to the variable goes directly to the environment. In nested scopes we keep track of how far up the call chain we need to walk to find the variable’s environment using a chain of Captures.

The first change to achieve this is to have variable lookup in a given context fall back to searching in the parent. We use a special ParentLookup method here to encompass both the lookup and capture of the variable.

    /// Lookup the given `id` in the current binder.
    member ctx.Lookup id =
        match ctx.LookupVar id with
        // If this was in our environment, then return
        // the variable's current storage location
        | Some(v) -> Some(v.Storage)
        | None -> ctx.ParentLookup id

The ParentLookup itself first checks to see if the current scope even has a parent. If a parent is found we delegate the actual lookup to a third method LookupAndCapture. If we find a variable then we return a capture reference to it.

    /// Look up the id in the parent context, and
    /// return an `Capture` to it.
    member ctx.ParentLookup id =
        match ctx.Parent with
        | Some(parent) -> 
            match parent.LookupAndCapture id with
            | Some(storage) ->
                Some(Storage.Capture(storage))
            | None -> None
        | None -> None

The LookupAndCapture method is quite similar to the plain Lookup, but moves values to the Environment if they don’t already live there.

    /// Lookup the given `id` in the current context, and
    /// move it to captured storage if it hasn't already
    /// been moved there.
    member ctx.LookupAndCapture id =
        match ctx.LookupVar id with
        | Some(v) -> 
            match v.Storage with
            | Environment(_) -> Some(v.Storage)
            | s -> 
                v.Storage <- Storage.Environment(s)
                Some(v.Storage)
        | None -> ctx.ParentLookup id

With all this in place it should now be possible to bind lambdas that reference variables in an outer scope. There’s one last wrinkle to iron out however. When we come to lower this code we need to know when to move values into the environment. To do this we need to know what values have been captured by a given lambda. We’ll add a list of the known captures to the BindCtx, and add to it each time we perform a ParentLookup:

/// Binder context, flowed through the bind to keep track of the current state
type BindCtx =
    { // ..
    ; mutable Captures: Storage list }
    
    member ctx.ParentLookup id =
        match ctx.Parent with
        | Some(parent) -> 
            match parent.LookupAndCapture id with
            | Some(storage) ->
                ctx.Captures <- storage::ctx.Captures
                Some(Storage.Capture(storage))
            | None -> None
        | None -> None

Now all that remains is to add this to our Lambda type, and set it when we bind:

type Bound =
    // ...
    | Lambda of Storage list * Bound
    
    // .. 
        let body = bind lambdaCtx body 
        Bound.Lambda(lambdaCtx.Captures, body)

Syntax	Bound
Seq [ Define ( "a", Number 1337 ); Lambda ( "x",Load "a" ) ]	Seq [ Store ( Local 0,Number 1337); Lambda ( [Environment (Local 0)], Load ( Capture ( Environment (Local 0))))]

And that’s it. We can now bind syntax trees into a semantic model that tracks when captures need to take place and provides enough information to lower to some form of executable code. To view the solution in full check out the Source Code for this Article. For a more in-depth and slightly more complex solution that inspired this check out the Feersum compiler’s bind method.

LISP in Two Days with Rust

Wed, 10 Jul 2019 00:00:00 +0000

As a sidetrack from the development of my programming language I’ve spent some time developing a LISP. The plan is to use the language as a testing ground for experimentation with transforming an AST in Rust. The syntax of LISP is simple and was developed to be easy to parse. I figured it would make a good starting point for an experimental compiler.

The language I’ll be developing here is heavily inspired by the lispy Scheme derivative. Some elements of behaviour are cribbed directly from Emacs LISP as it was the LISP implementation I had to hand. By the end we should have a language capable of evaluating expressions like the following:

(begin
	(define foo 1007)
	(define bar 330)
	(print (+ foo bar))) ; ~> prints 1337

Step One - Read

The first step on the road to any programming language is parsing. We will begin by splitting the job of recognising source code like (+ 1 foo) into two parts. Divide and conquer! The first step is to split the string into a list of tokens. This step is called lexical analysis in the fancy pants programming language world, where each of the tokens are called lexemes. Tokens is good enough for us though and easier to type. We’ll call it a tokeniser.

Our tokeniser works logically by running a set of regular expressions over the source text and choosing the token kind based on which expression matched. For our use case though we don’t need a full regex engine and can make do with a state machine built from a few match statements. For a more in-depth discussion check out my previous blogpost on this.

The tokens we need to recognise for LISP are pretty simple:

( and ) for grouping S-expressions
[0-9]+ for numbers
\n, \t, ` ` etc. for whitepsace
Everything else is an identifier

We begin by defining the states that our tokeniser can be in. In Rust this is best modelled as an enum:

enum TokeniseState {
    Start,
    Lparen,
    Rparen,
    Number,
    Symbol,
    Whitespace,
}

The signature of the tokenise function is pretty simple. It accepts a borrowed string and returns a vector of tokens.

fn tokenise(source: &str) -> Vec<ast::Token>

It’s probably time to define what a token is. It might be tempting at first to model a token as an enum too. You can do this but it sometimes becomes cumbersome. There’s often state that’s shared between all tokens. To solve this we split the token into two parts. First a kind which is an enum containing the token’s type along with any token-type-specific data such as the value of a Number token. The rest of the common fields are collected along with the kind into a struct:

#[derive(Debug, PartialEq)]
pub enum TokenKind {
    LeftBracket,
    RightBracket,
    Number(i64),
    Symbol(String),
}

#[derive(Debug, PartialEq)]
pub struct Token {
    pub kind: TokenKind,
    span: Span<ByteIndex>,
}

Note that the kind is public to allow for ergonomic matching over tokens later. We’re using the codespan crate for source position and span information.

With all that in place it’s time to tackle the body of the tokenise method. We loop collecting tokens and do a two-level dispatch, first on our current state then on the next character:

loop {
    let mut state = Start;
    let mut end = start;

    for c in source[start..].chars() {
        let next = match state {
            Start => match c {
                '(' => Some(Lparen),
                ')' => Some(Rparen),
                '0'...'9' => Some(Number),
                'a'...'z' => Some(Symbol),
                c if c.is_whitespace() => Some(Whitespace),
                _ => None,
            },
            Lparen | Rparen => None,
            Number => match c {
                '0'...'9' => Some(Number),
                _ => None,
            },
            Symbol => match c {
                'A'...'Z' | '0'...'9' => Some(Symbol),
                _ => None,
            },
            Whitespace => {
                if c.is_whitespace() {
                    Some(Whitespace)
                } else {
                    None
                }
            }
        };

        // If we transitioned then accept the character
        // by moving on our `end` index.
        if let Some(next_state) = next {
            state = next_state;
            end += c.len_utf8();
        } else {
            break;
        }
    }

    let token_str = &source[start..end];
    let span = Span::new(start, end);

    start = end;

    let kind = match state {
        Start => break,
        Lparen => ast::TokenKind::LeftBracket,
        Rparen => ast::TokenKind::RightBracket,
        Number => ast::TokenKind::Number(token_str.parse().unwrap()),
        Symbol => ast::TokenKind::Symbol(token_str.into()),
        // Skip whitespace for now
        Whitespace => continue,
    };

    result.push(ast::Token::with_span(
        kind,
        span.map(|s| ByteIndex(s as u32 + 1)),
    ));
}

Here I’ve elided some of the characters that can make up symbols for readability.

This should transform an input text such as (+ 1 foo) into a list of tokens:

vec![
	Token { kind: TokenKind::LeftBracket, ... },
	Token { kind: TokenKind::Symbol("+"), ... },
	Token { kind: TokenKind::Number(1), ... },
	Token { kind: TokenKind::Symbol("foo"), ... },
	Token { kind: TokenKind::RightBracket, ... },
]

The next step is to take the list of tokens and construct a syntax tree. Each node in the tree is represented by, you guessed it, an enum. In LISP there are two kinds of nodes in a syntax tree: atoms, and forms. The atoms in our LISP are Numbers and Symbols. Forms are parameterised S-expressions containing a list of forms or atoms. We could make do with a syntax tree node with just these two members. Our language has two special forms however: if for conditional execution, and define to introduce variable bindings. We will model these strongly in our tree too.

#[derive(Debug, PartialEq)]
pub enum Expr {
    /// A direct reference to a variable symbol
    Symbol(Token, String),
    /// A numeric literal
    Number(Token, i64),
    /// A conditional expression
    If(Token, Token, Box<Expr>, Box<Expr>, Box<Expr>, Token),
    /// A variable declaration
    Define(Token, Token, Token, Box<Expr>, Token),
    /// A funciton call expression
    Call(Token, Token, Vec<Expr>, Token),
}

Note that we’re including all the tokens in the tree, not just the ones which have important values. This is so we have more position information available later when running linters or syntax formatters. This technique, called full-fidelity trees, is used by many compilers such as C#’s Roslyn compiler and the Swift compiler. If we were to produce an IDE integration at a later date this extra metadata would prove invaluable.

We can recognise LISP with just a single token of lookahead. We will use the Rust Peekable iterator for this. To begin with the peekable list of tokens is all that our parser needs:

struct ParseState<I: Iterator<Item = ast::Token>>(std::iter::Peekable<I>);

Our top level parse function examines the next token. If it is an atom then we parse it directly, otherwise we delegate to parse_form to read the body of a form.

fn parse_expr(&mut self) -> ast::Expr {
    if let Some(token) = self.0.next() {
        use ast::TokenKind::*;
        match token.kind {
            LeftBracket => self.parse_form(token),
            RightBracket => panic!("unexpected token!"),
            Number(n) => ast::Expr::Number(token, n),
            Symbol(ref s) => {
                let sym = s.clone();
                ast::Expr::Symbol(token, sym)
            }
        }
    } else {
        panic!("invalid expression.")
    }
}

It’s a bit panic!y. A production grade parser would recover more gracefully from those errors either by returning a Result, or by buffering a diagnostic and stubbing out the missing tokens. The latter approach allows the parser to extract some syntactic information from any source text. This is becoming a far more common approach with the continued rise of the IDE.

fn parse_form(&mut self, open: ast::Token) -> ast::Expr {
    use ast::TokenKind::*;
    match self.0.peek() {
        Some(&ast::Token {
            kind: Symbol(ref sym),
            ..
        }) => match &sym[..] {
            "if" => {
                let if_tok = self.0.next().unwrap();
                let cond = self.parse_expr();
                let if_true = self.parse_expr();
                let if_false = self.parse_expr();
                let close = self.0.next().unwrap();
                ast::Expr::If(
                    open,
                    if_tok,
                    Box::new(cond),
                    Box::new(if_true),
                    Box::new(if_false),
                    close,
                )
            }
            "define" => {
                let define_tok = self.0.next().unwrap();
                let sym_tok = self.0.next().unwrap();
                let value = self.parse_expr();
                let close = self.0.next().unwrap();
                ast::Expr::Define(open, define_tok, sym_tok, Box::new(value), close)
            }
            _ => {
                let sym_tok = self.0.next().unwrap();
                let mut args = Vec::new();
                while let Some(token) = self.0.peek() {
                    if token.kind == RightBracket {
                        break;
                    }
                    args.push(self.parse_expr());
                }
                let close = self.0.next().unwrap();
                ast::Expr::Call(open, sym_tok, args, close)
            }
        },
        _ => panic!("invalid expression"),
    }
}

The parse_form method takes a similar approach. It attempts to mach one of our known special forms falling back to recognising a generic Call expression. With all this in place we can write a high-level parse method that our REPL will use to read source text into ast::Exprs:

pub fn parse(source: &str) -> ast::Expr {
    let tokens = tokenise(source);
    ParseState(tokens.into_iter().peekable()).parse_expr()
}

Step Two - Eval

Now we have a syntax tree parsed it’s time to move on to evaluating it. Our eval method will take an Expr and produce a Value. Because evaluation can encounter errors we return a Result type:

#[derive(Debug, PartialEq, Copy, Clone)]
pub enum Value {
    Number(i64),
    Callable(Callable),
    Nil,
}

type Callable = fn(Vec<Value>) -> EvalResult;

pub struct EvalError(String);

pub type EvalResult = Result<Value, EvalError>;

pub fn eval(expr: ast::Expr) -> EvalResult {
    eval_with_env(expr, &mut make_global_env())
}

Values can contain either a number or a callable function. We will not only return these as the result of evaluation but also store them in our environment.

There’s not much code in eval, but two interesting things are going on. First we call make_global_env to create a new global environment. The main evaluation takes place in eval_with_env.

The evaluation environment for our LISP is just a HashMap of Symbol to Value. The make_global_env function just creates a new map and inserts the functions we want to be globally visible:

pub fn make_global_env() -> HashMap<String, Value> {
    let mut env = HashMap::new();

    env.insert(
        "begin".into(),
        Value::Callable(|values| Ok(last_or_nil(values))),
    );
    env.insert(
        "+".into(),
        Value::Callable(|values| {
            let mut sum = 0;
            for value in values.iter() {
                sum += value.into_num();
            }
            Ok(Value::Number(sum))
        }),
    );
    env.insert(
        "-".into(),
        Value::Callable(|values| {
            Ok(if let Some((first, rest)) = values.split_first() {
                let mut sum = first.into_num();
                if rest.len() == 0 {
                    Value::Number(-sum)
                } else {
                    for value in rest {
                        sum -= value.into_num();
                    }
                    Value::Number(sum)
                }
            } else {
                // (-) ~> 0 ; apparently
                Value::Number(0)
            })
        }),
    );
    env.insert(
        "/".into(),
        Value::Callable(|values| {
            if let Some((first, rest)) = values.split_first() {
                let mut res = first.into_num();
                Ok(if rest.len() == 0 {
                    Value::Number(1 / res)
                } else {
                    for value in rest {
                        res /= value.into_num();
                    }
                    Value::Number(res)
                })
            } else {
                Err(EvalError("Wrong number of arguments: /, 0".into()))
            }
        }),
    );
    env.insert(
        "*".into(),
        Value::Callable(|values| {
            let res = values.iter().fold(1, |acc, v| acc * v.into_num());
            Ok(Value::Number(res))
        }),
    );

    env
}

With the environment taken care of evaluation is fairly simple. Number atoms evaluate to themselves. Symbols are looked up in the environment:

use ast::Expr::*;
match expr {
    Symbol(_, s) => env
        .get(&s)
        .cloned()
        .ok_or_else(|| EvalError(format!("eval: Undefined symbol {}", s))),
    Number(_, n) => Ok(Value::Number(n)),

If evaluates the condition expression and then chooses which branch to evaluate based on the result:

    If(_, _, cond, then, elz, _) => Ok(if eval_with_env(*cond, env)?.is_truthy() {
        eval_with_env(*then, env)?
    } else {
        eval_with_env(*elz, env)?
    }),

Define evaluates its expression and inserts the result into the environment hashmap:

    Define(_, _, sym, value, _) => {
        let value = eval_with_env(*value, env)?;
        let sym = to_sym(sym)?;
        env.insert(sym, value.clone());
        Ok(value)
    }

For function calls we search for a Callable in the environment, evaluate all of the arguments, and make the call:

    Call(_, sym, args, _) => {
        let sym = to_sym(sym)?;
        match env.get(&sym) {
            Some(Value::Callable(c)) => c(args
                                          .into_iter()
                                          .map(|a| eval_with_env(a, env))
                                          .collect::<Result<Vec<_>, _>>()?),
            _ => Err(EvalError(format!("eval: Invalid function {}", sym))),
        }
    }
}

Since our Expr type is simple that’s all the cases we need to deal with. This is the advantage of LISP. The syntax is small. The real power comes from the way it can be combined.

Step Three - Print

With parsing and evaluation in place we’re almost at a functioning read, execute, print loop. Printing the result of evaluation is pretty simple:

fn print(result: eval::EvalResult) {
    match result {
        Ok(value) => println!(" ~> {}", value),
        Err(error) => println!(" !! {}", error),
    }
}

To read in our source line we first need a prompt:

fn read() -> ast::Expr {
    let mut buff = String::new();
    print!(" > ");
    std::io::stdout().flush().unwrap();
    std::io::stdin().read_line(&mut buff).unwrap();
    parse::parse(&buff)
}

The final REPL is then pretty simple to construct:

let mut env = eval::make_global_env();
loop {
    print(eval::eval_with_env(read(), &mut env));
}

Note that the global environment is initialised before the loop so state from each REPL entry is available to following entries.

 > (define foo 100)
 ~> 100
 > (define bar 99)
 ~> 99
 > (+ foo (- foo bar))
 ~> 101

Step Four - Experiment

With the basics in place it’s time to experiment. It’s easy to add new callable functions to the “standard library” by including them in the global scope. The language outlined in this post doesn’t have support for user-defined functions or any form of loops. Another big LISP feature missing is the ability to quote and eval forms to turn them from source code into data and back.

The source for the programming language experiment behind this post is available on GitHub.

Optional Extras

Sat, 02 Mar 2019 00:00:00 +0000

A neat feature of the Option<T> type in Rust is that it implements the From<T> trait. This allows for ergonomic optional parameters:

fn print_int<T: Into<Option<u32>>>(stuff: T) {
    match stuff.into() {
        Some(value) => println!("{}", value),
        None => println!("no value!"),
    }
}

This function can be called with a standard Option value like print_int(Some(100)) or print_int(None). The key is that the Into conversion also allows you to call without wrapping values in Some. See it in action in the playground.

In Rust a common pattern when creating complex objects is to use the Builder Pattern. This is useful where you want to begin with some default state and apply partial customisations to it. In many cases however it is a bit heavyweight. Take for example constructing a node for an expression tree. In this case we might already have the components of the tree up front. We want to support parts of the tree being missing. In this case we can create builder functions with Into<Option<T>> arguments make this kind of construction a little more ergonomic:

#[derive(Debug)]
pub enum Expr {
    Literal(Option<i64>),
    Unary(Option<Op>, Option<Box<Expr>>),
    Binary(Option<Box<Expr>>, Option<Op>, Option<Box<Expr>>),
}

pub fn binary<L, O, R>(lhs: L, op: O, rhs: R) -> Expr
    where L: Into<Option<Expr>>,
          O: Into<Option<Op>>,
          R: Into<Option<Expr>>,
{
    Expr::Binary(lhs.into().map(Box::new),
                 op.into(),
                 rhs.into().map(Box::new))
}

When we come to create a tree we don’t have to mess around with Some values and manually Box things:

let tree = syntax_builder::literal(100);
let tree = syntax_builder::unary(None, tree);
let tree = syntax_builder::binary(tree, Op::Plus, None);

Check this out on the playground.

Racing Onwards

Sat, 08 Dec 2018 00:00:00 +0000

I’m a long term Emacs user. When I first started writing rust I concocted an Emacs configuration using racer-mode and rust-mode directly which got the job done. Since then the state of Rust editor support has come a long way. I’ve finally taken the plunge and moved to RLS for a richer language completion experience.

RLS is the Rust Language Server. It implements the standardised “language server protocol”; which provides a common interface between editors and language completions. For Rust it uses a mixture of racer the Rust autocompletion tool and more heavyweight analysis.

Before setting things up you need to make sure you have a few toolchain components installed:

$ rustup component add rust-src rls-preview rust-analysis rustfmt-preview

The good news is that these should all be available on the stable toolchain right now.

Configuration wise I prefer to manage things with use-package. For a simple setup you should just need rust-mode and lsp-mode:

(use-package rust-mode
  :ensure t
  :mode "\\.rs"
  :init (add-hook 'rust-mode-hook 'lsp))
(use-package lsp-mode
  :ensure t
  :defer t
  :commands lsp
  :config
  (use-package lsp-clients))

Personally I prefer to use Company for completions. To switch to that all you ned to do is require the company-lsp mode.

(use-package company-lsp
  :ensure t
  :after company)

That should be it. I like to keep my mode line neat and tidy so my full configuration contains a few :diminishes and re-maps some keys. Check out the final result. My old racer-mode config is there too, commented out for posterity.

Once More with Handlers

Sat, 24 Feb 2018 00:00:00 +0000

I previously wrote about creating a custom authentication middleware in ASP .NET Core. With the move to ASP .NET Core 2.0 this approach at creating custom authentication providers no longer works. It’s time drag custom authentication into the Year of the Fruit Bat with an ASP .NET Core 2.0 AuthenticationHandler.

Where previously we registered a custom middleware when configuring our app we instead just need to register the builtin authentication with app.UseAuthentication(). The configuration of our custom auth provider takes place when configuring services instead:

public void ConfigureServices(
    IServiceCollection services)
{
    services
        .AddAuthentication(o =>
        {
            o.DefaultAuthenticateScheme =
                TokenAuthOptions.DefaultScheme;
            o.DefaultChallengeScheme =
                TokenAuthOptions.DefaultScheme;
        })
        .AddSimpleTokenAuth(o => {
            // configure our auth options here
        });
}

After calling AddAuthentication on the service collection you receive an AuthenticationBuilder object. For our custom auth provider we need to create an extension method for it to register our handler and options:

public static AuthenticationBuilder AddSimpleTokenAuth(this AuthenticationBuilder builder, Action<TokenAuthOptions> options)
{
    return builder
        .AddScheme<TokenAuthOptions,
                   SimpleTokenAuthHandler>(
             TokenAuthOptions.DefaultScheme, options);
}

For this to work we need two new classes. The first replaces our old middleware class, this time deriving from AuthenticationHandler. It contains the logic for authenticating users based on the current request. I’ve chosen to call it SimpleTokenAuthHandler. The second is the TokenAuthOptions class. It’s job is to provide additional configuration for custom authentication schemes. The AddScheme call ties these two together under the named scheme TokenAuthOptions.DefaultScheme.

First up let’s set up the TOkenAuthOptions class:

public class TokenAuthOptions : AuthenticationSchemeOptions
{
    public const string DefaultScheme =
        nameof(SimpleTokenAuthHandler);

    public string Scheme => DefaultScheme;
}

If your authentication scheme request additional data to be configured then this is the place to put it. Users can then update the settings in this options instance from the AddSimpleTokenAuth call in ConfigureServices. Following on from the previous middleware example however we won’t be taking any extra options either.

The last part of the puzzle is the SimpleTokenAuthHandler class. This inherits from the base AuthenticationHandler provided by ASP .NET. We override the HandleAuthenticationAsync method and perform our custom authentication check. There are three main results that you might want to return from this method: AuthenticateResult.Success when you have been able validate the user correctly, AuthenticateResult.Fail when there was a problem with the authentication details provided, and AuthenticateResult.NoResult when there wasn’t any authentication attempt made for your custom scheme.

protected override Task<AuthenticateResult> HandleAuthenticateAsync()
{
    if (Request.Query.ContainsKey("auth"))
    {
        var identity = new GenericIdentity(
            "test_user", "queryString");
        var ticket = new AuthenticationTicket(
            new ClaimsPrincipal(identity),
            Options.Scheme);

        return Task.FromResult(
            AuthenticateResult.Success(ticket));
    }

    return Task.FromResult(
        AuthenticateResult.NoResult());
}

With this all in pace we can make and authenticated and unauthenticated request:

GET /?auth HTTP/1.1
Host: localhost:5000
Accept: */*

HTTP/1.1 200 OK
Server: Kestrel

Identity
  Name: test_user
  Label: 
  AuthType: queryString
  Authenticated?: True
  Claims: test_user

GET / HTTP/1.1
Host: localhost:5000
Accept: */*

HTTP/1.1 200 OK
Server: Kestrel

Identity
  Name: 
  Label: 
  AuthType: 
  Authenticated?: False
  Claims: 

This is a little bit different from the identities we ended up with in the old ASP .NET Core 1 version. Where previously we added a second identity which was authenticated the use of the standard AuthenticationHandler has meant we end up with just a single identity, which is either authenticated or unauthenticated.

The full code for this example is available as an update to the Gist. Happy authenticating!

Upon Reflection

Sat, 10 Feb 2018 00:00:00 +0000

I’ve been spending most of my time recently performance profiling and optimising one of our main services. Part of its job involves exposing data objects through a generic interface. The abstraction allows public properties on each object to be read using an untyped indexer. To do this we look up all the properties of each data object via reflection and cache them so later reads and writes are quicker. Turns out this caching is not enough. With many short-lived objects we ended up almost half our time in some performance critical paths just populating this reflection cache. The initial implementation is naïve, but what can be done about it?

To get started imagine that we have the following two C# classes:

public class Base
{
    private IDictionary<string, PropertyInfo> _props;
    
    public Base()
    {
        _props = GetProps(GetType());
    }
    
    IDictionary<string, PropertyInfo> GetProps(Type t) =>
       t.GetMembers()
       .Where(mi =>
           mi.MemberType == MemberTypes.Property)
       .Cast<PropertyInfo>()
       .ToDictionary(prop => prop.Name);
}

public class User : Base
{
    public string Name { get; }
    public int Age { get; }
}

Each time a new User is allocated we will populate the cache of properties we can read and write (Name, Age, etc.). Obviously this is a tad less than efficient. The properties on a User aren’t going to change between instantiations. An obvious optimisation here is to cache these results. Lets update our Base class a little:

public class Base
{
    public Base()
    {
        _props = TypeProps.GetOrAdd(
            GetType(),
            GetProperties);
    }

    // ... As before ... 

    static ConcurrentDictionary<
        Type,
        IDictionary<string, PropertyInfo>> TypeProps =
        new ConcurrentDictionary<
            Type,
            IDictionary<string, PropertyInfo>>();
}

Here we’ve allocated a static dictionary of Type to IDictionary of properties. For each different type we encounter a new entry will be added to the dictionary containing the property information for that type. As you might expect this speeds things up greatly.

Benchmarking the before and after you’ll find that this per-type cache rather than per-instance cache improves the speed of allocating an object by a few orders of magnitude. Allocation went from 1,100ns to 40ns in my own benchmarking. It’s tempting to claim the win and leave things there, however we can do more. Enter the Curiously Recurring Template Pattern. Instead of allocating our own dictionary of type to properties we’ll use C#’s generics and static members to get the language to do this caching for us.

The idea is pretty simple. By making our base class take a type parameter we can lookup all the properties on that type at static initialisation time using typeof:

public class Base<T>
    where T : Base<T>
{
    static IDictionary<string, PropertyInfo> Props = GetProps(typeof(t));
}

We then update each of our derived classes to pass itself in as the template parameter:

sealed class User : Base<User>
{
    // ... As Before ...
}

With the properties now just stored in a static there’s no need to look up the correct dictionary at runtime using GetType. This update then gets you another order of magnitude in allocation speed; from 40ns to 4ns in my testing.

Another important thing to note in this is the reduction in memory churn for the garbage collector too. The allocation size for the final User object is just the object header, method table and the two properties. In the contrived example of the benchmark this results in a 3x reduction in the number of garbage collections.

It’s not all rainbows however. When using this pattern a lot of the benefits of object inheritance are lost. Deeper inheritance hierarchies become more of a faff requiring an extra type parameter to be passed up the chain. Adding this pattern into an existing codebase will require a lot of tedious changes.

In conclusion the Curiously Recurring Template Pattern can provide powerful opportunities for optimisation, but don’t use it unless profiling suggests you need that extra boost!

Make it work Switch

Tue, 19 Dec 2017 00:00:00 +0000

During some performance testing of an application a colleague and I had trouble processing more than 150 requests per second per instance. After much hair-pulling the bottleneck was identified as the ServicePointManager.RequestLimit. This global variable controls the number of parallel HTTP requests that a .NET app will make to a given domain. By default it is set to 2, throttling the load that an off-the-shelf .NET app can place on a remote server.

It isn’t the first time that I’ve encountered this variable limiting an application. In fact it isn’t even the first time it has caused problems in that specific application; however the previous mitigation of setting it to 2000 was accidentally lost during a partial re-write.

These kinds of gotchas with default configuration are referred to at work as “make it work switches”. The ServicePointManager is far from the only culprit. Software is littered wit them, such as Elastic Search’s discovery.zen.ping.multicast.enabled; Javascript’s "use strict";; and the new .csproj format’s <LangVersion>Latest</LangVersion>. These kind of papercut issues can really impact user experience. Not only are they annoying to keep track of, but are often stumbling blocks for beginners.

These kinds of stumbling blocks are often the most important to work on. They affect how software is perceived by new users, and how more experienced users feel about using your software. Feel is important. It can make super-fast software feel clunky and cumbersome to use. On the flip side a well designed interface which “just works” can make even the most complex things seem simple and intuitive.

Things that “just work” by default is often touted as the reason to use macOS or iOS. There’s a reason for this. Apple has historically had a dedication to simplicity. Maybe too much so for some people. One of the ways that this often shows is the lack of configuration available. The legend goes that if you wanted to add a config option Steve Jobs would ask you what the default should be. When you’d decided that then he would tell you to hard code it to that value. While configurable software allows you to quickly adapt it to do new and different things it does come at a cost. After all, there would be no make it work switches if there were no switches at all.

Authentication Middleware in ASP.NET Core

Thu, 17 Aug 2017 00:00:00 +0000

There are two sides to the auth story in ASP .NET Core: authentication and authorisation. Authorisation is usually controlled in an MVC app with the [Authorize] attribute. The simplest, no-argument [Authorize] checks that the user has a single identity which is authenticated. Authentication is responsible for inspecting the request and adding known identity information about the user. In many cases this will be validating OAuth or JWT.

EDIT: For an updated version of this authentication scheme which works with ASP .NET Core 2.0 check out Once More with Handlers.

Supposing you have something simpler though. An app which doesn’t need to issue tokens of its own; instead it just needs to validate tokens issued by another app. In that case it’s time to create your own custom authentication middleware.

First we’ll add an app.UseSimpleTokenAuth() to our application build chain. This needs to come before any parts of the pipeline which depend on authentication. Obviously that extension method doesn’t exist yet. The next step is to create it:

public static class MiddlewareExtensions
{
    public static IApplicationBuilder
    UseSimpleTokenAuth(
        this IApplicationBuilder app)
    {
        return app
          .UseMiddleware<SimpleTokenAuthMiddleware>();
    }
}

This method registers everything needed by our middleware. At the moment that’s just the middleware itself. If we want to pass configuration information to the middleware through dependency injection at a later date this would be the location to add the .Configure<>(..) calls.

Once we have our middleware registered it’s time to write it. A simple middleware just receives a RequestDelegate in its constructor, and calls that delegate when Invoke is called:

public class SimpleTokenAuthMiddleware
{
    private readonly RequestDelegate _next;

    public SimpleTokenAuthMiddleware(
        RequestDelegate next)
    {
        _next = next;
    }

    public Task Invoke(HttpContext context)
    {
        // This is where we will check auth later
        return _next(context);
    }
}

Now comes the difficult bit. Inspecting the request to see if the user is authenticated. In this simple case I’m just going to check for the presence of an auth query parameter. In reality though you’d want to look for a token in headers or cookies and validate it somehow.

if (context.Request.Query.ContainsKey("auth"))
{
    var identity = new GenericIdentity(
        "test_user", "queryString");

    context.User.AddIdentity(identity);
}

If the user is authenticated we create a new GenericIdentity for the user. The first parameter specifies the user’s name and the second the authentication method used to generate the identity. You must include some value for the authentication method otherwise the identity can’t be used with the [Authorize] attribute.

That’s it. If a request has auth in the query string an identity will be added and the user will be considered authenticated. To verify this we can create a simple Run method in our Startup.cs:

app
    .UseSimpleTokenAuth()
    .Run(async context =>
            {
                if (context.User?.Identities == null)
                {
                    await context.Response.WriteAsync(
                    "No user identities");
                }

                foreach (var id in 
                         context.User.Identities)
                {
                    var sb = new StringBuilder();

                    sb.AppendLine("Identity");
                    sb.AppendLine(
                      $"  Name: {id.Name}");
                    sb.AppendLine(
                      $"  Label: {id.Label}");
                    sb.AppendLine(
                      $"  AuthType: {id.AuthenticationType}");
                    sb.AppendLine(
                      $"  Authenticated?: {id.IsAuthenticated}");
                    var claims = string.Join(
                        ", ",
                        id.Claims.Select(
                            c => c.Value));
                    sb.AppendLine($"  Claims: {claims}");

                    await context.Response.WriteAsync(
                        sb.ToString());
                }
            });

Now if you hit your app you should see a blank anonymous identity:

GET / HTTP/1.1

HTTP/1.1 200 OK

Identity
  Name: 
  Label: 
  AuthType: 
  Authenticated?: False
  Claims:  

Hit it with auth in the query string and a second, authenticated, identity appears:

GET /?auth HTTP/1.1

HTTP/1.1 200 OK

Identity
  Name: 
  Label: 
  AuthType: 
  Authenticated?: False
  Claims: 
Identity
  Name: test_user
  Label: 
  AuthType: queryString
  Authenticated?: True
  Claims: test_user

You can find the full code for this post in a gist.

Rusting up Some Iron

Tue, 16 May 2017 00:00:00 +0000

Rust is one of my favourite languages. One of it’s great strengths is its ecosystem of crates, another is the FFI system which enables writing native extensions when performance or safety is required. Over the last week I’ve been pulling these two together in IronRure, a .NET Standard wrapper around the Rust regex crate.

I started this with zero knowledge of FFI in C#. I was pleasantly surprised with how easy P/Invoke was to use. There’s plenty of documentation on its use from both Microsoft and Mono. I feel I should pull together the lessons I’ve learned though. Don’t hesitate to comment if you think I’m Doing it Wrong.

Stage 1 - Quick and Dirty

The first step is to get the compiled Rust code next to your managed assembly so that P/Invoke can find it and load it. The simplest way to get this working is to compile the Rust ahead of time with cargo and add a direct Content reference to the output. Here’s what I started with:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard1.4</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Content Include="..\..\thirdparty\regex\regex-capi\target\release\rure.dll">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </Content>
  </ItemGroup>
</Project>

Since I was targeting .NET Standard I wanted this to work on Windows, Mac and Linux. To solve that we’ll need to wrap the native code in a NuGet package. Content files will get you a long way though when messing about with an idea.

Going Cross Platform

The first step in making sure that your bindings will work on different platforms is to make sure that DllImport statements aren’t platform specific. Luckily Rust libraries have a well known naming structure which matches up with what .NET expects.

[DllImport("foo")]
public static extern void hello();

By using DllImport wihout a lib prefix and .dll/.so/.dylib suffix the runtime will pick the correct file on Linux, Windows and Mac!

Packing it Up

Although .NET code is not targeted to a specific CPU your native Rust code is. In .NET the underlying target architecture is called the Runtime Identifier, or RID for short. RIDs are arranged into a tree. If you’re running on win10-x64 the runtime will try to load native code for the win7-x64, win-x64 and win RIDs when installing a NuGet package. We can exploit this to create a set of native-only NuGet packages containing just the compiled Rust code which are then referenced as NuGet dependencies in the managed FFI wrapper.

You could create a different NuGet package for each runtime identifier being targeted. I chose however to create a single package for each operating system and include all the runtime identifiers I was supporting for that OS. On Mac there is only a single RID to target really: osx. After compiling the crate (using the --release flag) the output needs to be copied so it has the following structure:

contents/
└── runtimes
    └── osx
        └── native
            ├── librure.a
            ├── librure.d
            └── librure.dylib

On Windows, where we are targeting more than one RID it is a bit more complex:

contents/
└── runtimes
    ├── win-x64
    │   └── native
    │       ├── rure.lib
    │       ├── rure.d
    │       └── rure.dll
    └── win-x86
        └── native
            ├── rure.lib
            ├── rure.d
            └── rure.dll

We can then pack this up into a NuGet package with the following .nuspec file:

<?xml version="1.0" encoding="utf-8"?>
<package>
  <metadata>
	  ...
  </metadata>
  <files>
    <file src="contents\**" />
  </files>
</package>

The FFI wrapper library just needs to add a PackageReference like any other library. When you compile the appropriate native library will be picked from its NuGet and placed next to the output program ready for P/Invoke to find.

The Pragmatic PasswordBox

Fri, 14 Apr 2017 00:00:00 +0000

In WPF the PasswordBox control does not expose the password it contains through data binding. It is a security feature designed to protect the sensitive data the box contains. It is however quite a faff and has sparked much debate.

Recently I’ve been experimenting with ReactiveUI. ReactiveUI brings together the elegance and power of Reactive Extensions (RX) and MVVM to provide a clean, simple, and reactive way to build interfaces. One of the key tenets of ReactiveUI is that it’s not actually bad to put logic in code-behind. With that in mind the power of Rx provides an elegant way to bind the contents of a PasswordBox to the view model.

Observable.FromEvent<RoutedEventArgs,SecureString>(a =>
{
     RoutedEventHandler proxy = (_, args) =>
     {
         a(Password.SecurePassword);
     };
     return proxy;
},
h => Password.PasswordChanged += h,
h => Password.PasswordChanged -= h)
    .BindTo(this, x => x.ViewModel.Password);

I think this is a pretty elegant solution to the whole problem. ReactiveUI calls these hack bindings, and they are, but sometimes as a developer you have to break out a little pragmatism.

I have also found this technique useful to bind the SelectedItems property of a ListBox or ListView control back to the view model. Use it sparingly though. It’s difficult to test and debug these bindings.

The Dark Side

Thu, 02 Mar 2017 00:00:00 +0000

At work I’m paid to write C#. At home I’m a dedicated Mac user. When I initially learnt C# for the job interview years back I had to use Mono to run it on my Mac. A lot has changed in the .NET world since then, most notably the arrival of .NET Core. So I finally decided to give it a try.

The installation on OS X was really simple. So far I’m really enjoying compiling stuff from the command line with the dotnet tool. Project.json files look to be a lot easier to branch and merge than .csproj files. Impressed with the whole thing I decided to knock together a quick project to give the whole thing a proper shakedown.

So far building and testing from emacs using dotnet build and dotnet test has been pretty good. There’s a few rough corners which show themselves, such as dotnet restore working from the project root, dotnet build requiring a glob to find project files to build and dotnet test not accepting globs at all. I guess the tooling is still a preview after all. That said I did manage to get something working pretty easily and quickly. The result was PollyTick; a library for collecting execution statistics for executions of Polly policies. Polly itself is a great little library, but that’s for another day.

Setting up a build on Appveyor for this library was pretty simple too. I’ve previously used Appveyor to build Rust on Windows. The whole .NET Core restore, build, test, package workflow fits quite nicely into Appveyor’s four stages of workflow. After a bit of fiddling I managed to get a build up and running. The resulting config ended up being pretty simple:

image: Visual Studio 2015
install:
  - ps: dotnet --version
  - ps: dotnet restore
build_script:
  - ps: dotnet build src/*/project.json
test_script:
  - ps: ForEach ($folder in (Get-ChildItem -Path test -Directory)) { dotnet test $folder.FullName }

The fiddly bit was being able to run all test projects. The dotnet test command doesn’t support the globs that dotnet build does, so a bit of PowerShell is required to get the job done. I did experiment with caching NuGet packages however that just seemed to make the build slower. Overall the process was pretty painless though. The Appveyor builds themselves seem super responsive and it builds this little project super quickly.

Here’s to the future of .NET and C#.

The Lies

Wed, 01 Mar 2017 00:00:00 +0000

At the weekend I was listening to Stereophonics ~ Performance and Cocktails. The song Half the Lies you Tell Ain’t True reminded me of a certain public figure. I have no idea why.

But when you rely on a lie that’s true,
And no one believes in the things you do
‘Cause half the lies you tell ain’t true

It’s been stuck in my head ever since. My brain can’t quite remember the verse, so it’s substitutes some other Stereophonics lyrics instead:

She longs to be another teenage dream,
The problem is she’s turning forty three,
She still takes her clothes off.

Somehow, it seems to go.

The Missing Link

Mon, 27 Feb 2017 00:00:00 +0000

LLVM is great for hacking about with languages. It takes a lot of the complexity out of code generation. It allows you to turn your SyntaxTree or whatever you’ve got into real live object code. Code you can directly execute. Well, almost.

The last crevasse that LLVM plonks us down next to is linking. Taking a collection of object files, adding the C runtime library and producing an executable isn’t quite as simple as it first seems. Luckily however the’s a cunning trick to sidestep the problem: let someone else do it for you. The wonderful developers who make Clang have already solved those issues. Suddenly compiling down to an independent executable seems a lot easier.

I’m going to dive straight in and assume there’s an LLVMModuleRef compiled and ready to go. The first thing we need to do is write the module out somewhere to disk where clang can read it from. In Rust I’m reaching for the tempdir crate to make sure things get cleaned up afterwards, but really the exact place these intermediate files get written doesn’t matter.

// Create a tempdir to write the LLVM IR to
let tmp_dir = TempDir::new("output")
    .expect("create temp dir");
let temp_path = tmp_dir.path().join("temp.ll");

Writing out an object file requires using the LLVM TargetMachine infrastructure. That sounds a bit fiddly though so it’s easier to just write out LLVM IR for now with LLVMPrintModuleToFile.

let path = CString::new(temp_path.to_str().unwrap()).unwrap();
unsafe {
    let mut message = ptr::null_mut();
    if core::LLVMPrintModuleToFile(self.raw, path.as_ptr(), &mut message) == 0 {
        Ok(())
    } else {
        let err_str = CStr::from_ptr(message)
            .to_owned()
            .into_string()
            .unwrap();
        core::LLVMDisposeMessage(message);
        Err(err_str)
    }
}

With that in place all we need to produce a fully functioning executable is to shell out to Clang! In rust this is super easy with the Command builder. We just need to tell Clang where our LLVM IR is, and where to put the result when it’s done.

// Shell out to Clang to link the final assembly
let status = try!(Command::new("clang")
    .arg(temp_path)
    .arg("-o")
    .arg("a.out") // It's traditional :-p
    .status());

And, if all goes well, Clang should do our dirty work and leave us with an executable a.out. All linked and ready to go!

I’m using this trick in anger at the moment as I continue hacking about on Ullage. I’ve since seen that Wilfred has done the same thing for bfc.

Low Down - Part 2 - Initialisation

Tue, 01 Nov 2016 00:00:00 +0000

In my previous post I left out something quite important: initialising the components of LLVM we wanted to use. Luckily there’s a quick and easy way to get that sorted, and we get to learn about a neat part of the Rust standard library.

LLVM itself is composed of many modular libraries. This is great for library consumers as you can pick just the bits you want to use in whatever application you’re writing. It does however mean there’s not a single initialise method to call. Some modules, such as the PassRegistry, expect you to initialise each object which is created. For compilation though we are expected to initialise the targets we want to use and, optionally, their ASM printers and parsers.

Some of the target initialisation routines can be called more than once, but not all are guaranteed to be reentrant or thread safe. To solve this we can use Rust’s Once to ensure that we only call the initialisation routine once:

fn ensure_initialised() {
    use std::sync::{Once, ONCE_INIT};

    static INIT: Once = ONCE_INIT;

    INIT.call_once(|| {
        // Initialise LLVM
        unsafe {
            execution_engine::LLVMLinkInMCJIT();
            if target::LLVM_InitializeNativeTarget() != 0 {
                panic!("Could not initialise target");
            }
            if target::LLVM_InitializeNativeAsmPrinter() != 0 {
                panic!("Could not initialise ASM Printer");
            }
        }
    });
}

Here we are linking in the MC JIT so we can create and execute machine code for the current target, and initialising the current and it’s ASM printer. This should be enough to be able to start working with JIT compilation using MCJIT. More on that later.

The Low Down - Using LLVM From Rust

Sun, 23 Oct 2016 00:00:00 +0000

LLVM is a collection of libraries and programs for creating compilers. It is built around a powerful intermediate representation, LLVM IR. It has tools for compiling LLVM IR to native instructions, performing optimisations and even performing JIT compilation. The libraries that power this have an extensive C API allowing you to harness this power yourself, so that’s what we’re going to do! I’ll show you how to create a simple Rust program which creates a simple function from scratch in LLVM.

First let’s get up and running. Start by creating a new Rust project adding a reference to the llvm-sys crate to the Cargo.toml file:

[dependencies]
llvm-sys = "*"

With this in place it’s probably worth making sure everything will compile first. The llvm-sys crate doesn’t contain a copy of LLVM itself, instead it relies on an existing installation. This should just be a matter of consulting your platform’s package manager. If you’re on Linux make sure to install the llvm-dev package. OS X users can find the llvm package in Homebrew. Because OS X comes with a version of LLVM itself Homebrew won’t add it to your path. To compile using the version Homebrew has installed you’ll need to modify your path first: $ export PATH=/usr/local/opt/llvm/bin:${PATH}. You can check you’ve got LLVM installed correctly by running $ llvm-config —version from the command line. You should see some output similar to the following:

$ llvm-config --version
3.8.1

If llvm-config can’t be found make sure that you’ve got a dev version of LLVM installed, along with the headers. On linux platforms these are usually suffixed with -dev. If that is the case make sure that your path includes the /bin folder from that distribution of LLVM.

If all is going well so far building your first Rust program which references LLVM should be just a cargo build away. If that all compiled cleanly then give yourself a pat on the back. Getting a rust program to build and link to LLVM is half of the battle.

Once the foundations are in place the next step is to start using LLVM for something. All code in LLVM lives in a Module. Modules are created with a call to LLVMModuleCreateWithName. Because this is C land you’ll also have to make sure to clean up modules once you’re done with them by calling LLVMDisposeModule:

extern crate llvm_sys;

use std::ffi::CStr;

use llvm_sys::prelude::*;
use llvm_sys::{core, target};

fn main() {
    // Create a module to compile the expression into
    let mod_name = CStr::from_bytes_with_nul(b"hello\0").unwrap();
    let module = unsafe { core::LLVMModuleCreateWithName(mod_name.as_ptr()) };
    
    // ...
    
    // Clean up the module after we're done with it.
    unsafe { core::LLVMDisposeModule(module) }
}

Although this doesn’t do anything incredibly interesting at the moment it’s worth noticing a few things. Firstly we are importing a few things from llvm_sys. The create is organised into a series of Rust modules named after the header file from the C api. The prelude contains most of the pointer & type definitions that are shared. Secondly seen as we’re using the llvm-sys crate here we’re dealing with raw bindings to C functions. That means that we need to pepper our code with unsafe to call any of them. It is general good practice in production Rust code to wrap these unsafe pointers and functions up in ‘safe’ Rust wrappers to keep the unsafe separate from the main logic of your program, and make sure that objects are free’d properly using the Drop trait. I’m not doing that here though as the aim of this post is to demonstrate the use of the API itself.

Next up it’s time for some Types. LLVM IR is similar in some ways to assembly code but it contains far more type information. Before we can create a function in our module we need to create a type for it. Basic types already exist for things like integers. For more complex types such as function types we need to call a function to ‘build’ it from its constituent parts:

 // Create a function type and add a function of that type to the module
let function_type = unsafe {
    let int64 = core::LLVMInt64Type();
    let mut param_types = [int64, int64];
    core::LLVMFunctionType(int64, param_types.as_mut_ptr(), param_types.len() as u32, 0)
};

Once we have the type for our function adding it to the module is simple. We just need to give it a name:

let function_name = CStr::from_bytes_with_nul(b"test\0").unwrap();
let function =
    unsafe { core::LLVMAddFunction(module, function_name.as_ptr(), function_type) };

You can’t go straight ahead and start adding code to this function. LLVM IR is built up from a ‘graph’ of basic blocks. Each block contains code which is executed in series and ends with either a return statement or a jump to another block. Our function will be super simple so we just need a single block. One thing to note here is that we give it a name. This acts as a label and allows control flow to jump to the start of the block.

// Create a basic block and add it to the function
let block_name = CStr::from_bytes_with_nul(b"entry\0").unwrap();
let entry_block = unsafe { core::LLVMAppendBasicBlock(function, block_name.as_ptr()) };

The easiest way to instructions into this block is using an IR Builder. This provides a cursor into a basic block where instruction will be inserted and a large number of methods for creating new instructions. You only need one of these objects. Once you’re done inserting instructions into one basic block you can point it at another with LLVMPositionBuilderAtEnd and use it again.

// Create an LLVM instruction builder and position it at our new
// basic block
let builder = unsafe { 
    let b = core::LLVMCreateBuilder();
    core::LLVMPositionBuilderAtEnd(b, entry_block);
    b
};

Now we have a builder it’s time to create the body of our function. First we get references to each of the function’s parameters using LLVMGetParam. We then insert an add instruction to sum up both of these values. Like most instructions in LLVM this creates a new value, and we give that value a name. Don’t worry about the number of variables this creates. When optimisations are run most of these ‘intermediate values’ will be removed.

// Fill in the body of the function.
unsafe {
    let a = core::LLVMGetParam(function, 0);
    let b = core::LLVMGetParam(function, 1);

    let temp_name = CStr::from_bytes_with_nul(b"temp.1\0").unwrap();
    let res = core::LLVMBuildAdd(builder, a, b, temp_name.as_ptr());

    core::LLVMBuildRet(builder, res);
}

And that’s it. You should now have a representation of a simple function in memory which adds together two 64 bit numbers. All that’s left is to clean up our builder object:

// Clean up the builder now that we are finished using it.
unsafe { core::LLVMDisposeBuilder(builder) }

From here you could compile this module down to native code or JIT compile it and run it in the same process. Maybe I’ll cover that in a later post. Right now let’s dump the whole module to see what LLVM has created for us:

// Dump the LLVM IR to stdout so we can see what we've created
unsafe { core::LLVMDumpModule(module) }

Compile and run and you should see a textual representation of the LLVM IR we built:

; ModuleID = 'hello'

define i64 @test(i64, i64) {
entry:
  %temp.1 = add i64 %0, %1
  ret i64 %temp.1
}

I hope you’ve enjoyed yourself. Who knows maybe this even proves useful to someone. If it does please do let me know in the comments. If i’ve made a mistake somewhere, or mis-represented the LLVM API then please let me know. I’m only just getting started with it myself.

You can find the full code for this article in this gist. You should be able to clone the gist directly and compile it with Cargo.

Top-down Operator Precedence Parsing in Rust

Sat, 03 Sep 2016 00:00:00 +0000

Following on from from creating a tokeniser the next step is writing a parser. The simplest way to create a hand-rolled parser is using recursive decent. In this parsing method each non-terminal in the grammar is converted into a function which encodes it’s parsing logic. This is great for parsing things like function definitions, variable declarations and other parts of a language which can be easily recognised from their prefix. Where it falls down a bit however is parsing expressions with infix operators. This is where Pratt operator precedence parsing comes in.

To parse multiple operator precedence lives in standard recursive decent you must left-factor your grammar. This results in one non-terminal per precedence level. To parse a literal in such a grammar the parser must descend through each precedence level again and again. This can result in quite a lot of overhead for large expressions and languages with many precedence levels in their grammar.

The solution that Pratt came up with was to parse all the consecutive operators at the current precedence level before descending to the next one. The algorithm itself is quite simple:

fn expression(tokens, rbp) {
    let left = tokens.next().nud(tokens);
    while (next_binds_tighter_than(tokens, rbp)) {
        left = tokens.next().led(tokens, rbp);
    }
    left
}

When parsing an expression we first ask the current token to parse its null denotation. This basically parses just the current literal value or variable reference. We then inspect the next token in the stream. If the next token has a binding power greater than ours then we ask it to parse it’s left denotation. For a simple binary operator this involves taking the operator and it’s following expression from the token stream and returning a binary operator expression. Once we no longer have any tokens in the stream that bind higher than our precedence level we return the expression we have parsed.

One of the key ideas here is that, as in Douglas Crockford’s implementation, each token is responsible for the logic required to parse itself and any following tokens which make up the expression that it is part of. Let’s start with a simple grammar with two operators at different precedence levels and numeric literals:

#[derive(Clone)]
pub enum Token {
    Add, Mull, Num(i64)
}

The first step is to define the left binding power for each of these tokens. The higher the precedence the tighter the operator binds:

impl Token {
    fn lbp(&self) -> u32 {
        match *self {
            Token::Add => 10,
            Token::Mull => 20,
            _ => 0
        }
    }
}

The next step is to implement the nud method. We only have one value type in this grammar so it’s pretty simple.

fn nud(&self) -> Result<Expression,String> {
    match *self {
        Token::Num(i) => Ok(Expression::Num(i)),
        _ => Err("expecting literal".to_string())
    }
}

Next up the led method. This is responsible for parsing our binary operators.

    fn led(&self, parser: &mut Parser, lhs: Expression) -> Result<Expression,String> {
        match *self {
            Token::Add | Token::Mull => {
                let rhs = try!(parser.expression(self.lbp()));
                Ok(Expression::Binary(Box::new(lhs), self.clone(), Box::new(rhs)))
            }

            _ => Err("expecing operator".to_string())
        }
    }

Now all that’s left is to create our Parser and fit all of these parts together. For simplicity our parser will take an iterator of pre-made tokens. We’ll define a few helper functions as well to aid calling the Token’s nud, led, and lbp methods.

pub struct Parser<'a> {
    tokens: Peekable<Iter<'a, Token>>
}

impl<'a> Parser<'a> {

    fn new(tokens: Iter<'a, Token>) -> Self {
        Parser{ tokens: tokens.peekable() }
    }
    
    fn next_binds_tighter_than(&mut self, rbp: u32) -> bool {
        self.tokens.peek().map_or(false, |t| { t.lbp() > rbp })
    }

    fn parse_nud(&mut self) -> Result<Expression,String> {
        self.tokens.next().map_or(Err("incomplete".to_string()), |t| {
            t.nud()
        })
    }

    fn parse_led(&mut self, expr: Expression) -> Result<Expression,String> {
        self.tokens.next().map_or(Err("incomplete".to_string()), |t| {
            t.led(self, expr)
        })
    }

Next up is the main expression function. This isn’t too far from the pseudo-code we started with:

fn expression(&mut self, rbp: u32) -> Result<Expression,String> {
    let mut left = try!(self.parse_nud());
    while self.next_binds_tighter_than(rbp) {
        left = try!(self.parse_led(left));
    }
    Ok(left)
}

And that’s it. We’re all done. For a full code listing check out the Gist.

Doing Nothing

Sat, 14 May 2016 00:00:00 +0000

When hacking on a problem it’s easy to get carried away with the how. How are you going to divide up the implementation. How are you going to ensure it’s testable. Usually these hows are quite narrow.

I used to head to and from work on the bus, however i’ve recently switched to cycling in. It’s quite nice to be able to take a break from the how and do nothing. Taking some time out often means you return to the fray with new ideas.

Cycling is often a good way to wind down. At the end of a day’s development I often leave the office with my head full of what I’m working on. How i’m going to solve the next problem. Where I’m going to go next with the implementation. What I’m going to try next when debugging.

Then I start to cycle home. Now I’ve got other things to worry about: traffic lights, other road users, the route I’m taking. In some ways I’m not really achieving much. I’m doing nothing. However after these breaks is when I find I come up with some great ideas. When I’ve had the opportunity to let my mind wander a bit. Here’s to doing nothing.

768 is the Magic Number

Thu, 26 Nov 2015 00:00:00 +0000

When reconstrucing a binary MSMQ message from a backup I was first tempted to just set the Body property on the message directly. This doesn’t quite work however as the Body property is serialised with whatever formatter you’re using when the message is sent.

var mq = new MessageQueue(@".\Private%\test");
var messageBody = File.ReadAllBytes("...");

var message = new Message();
message.Body = messageBody;

mq.Send(message); // double-encodes the body :-(

There is a way around this serialisation though. MSDN has this to say:

The contents of the Body property are serialized when the message is sent, using the Formatter property you specify. The serialized contents are found in the BodyStream property. You can also set the BodyStream property directly, for example, to send a file as the data content of a message.

Sounds just like what we want! Just setting the BodyStream property to a MemoryStream with our bytes in gets us started. The only problem is that tools peeking at the message don’t know what format it is in. This is where the magic number 768 makes it’s show. By setting the BodyType to this magic number tools will know the body is serlialised with a BinaryMessageFormatter:

var message = new Message();
message.BodyStream = new MemoryStream(messageBody);
message.BodyType = 768; // Binary formatted .NET object

Now you can send the message and recipients won’t know the differnece from the origional backup.

Hacktoberfest

Sun, 11 Oct 2015 00:00:00 +0000

One of my favourite new languages right now is Rust. It’s an interesting combination of strong typing, low level control and memory safety. Rust is a rapidly growing and changing language, and now is a great time to get involved as part of Hacktoberfest.

Hacktoberfest is GitHub & Digital Ocean’s yearly celebration of open source code. Submit 4 pull requests to open source projects on GitHub and get a snazzy tee shirt.

Getting Started

If you’re new to open source software GitHub has a great guide to contributing to open source. Find a project on GitHub that you’re interested in; fork the project and get hacking.

A good place to get started with contributing to many projects is to check out their GitHub issues and find a simple bug to fix. Hacking around to fix the bug will teach you about some of the structure of the project. You’ll also be forced to find out how to build it and run the tests. It’s also a good litmus test for contribution. If you can’t just clone the repo and build it then the project might not be ready for contributions.

If you want to get started working towards your free shirt Rust has issues tagged with E-easy which are simpler to get your teeth into. If you’re having trouble then there’s also the E-mentor tag, which contains issues where a member of the Rust team has offered to lend a hand.

A good issue to start you with contributing to Rust is #24407. This issue tracks adding --explain support for Rust error codes. Being able to call rustc --explain E0001 is a great rust feature, but not all error codes are supported yet.

Keeping Track

Once you’ve got a few pull-requests opened you can check how well you’re doing, and share your contribution with the community with the #hacktoberfest hashtag.

A Rusty Guide to Tokenising by Hand

Wed, 15 Apr 2015 00:00:00 +0000

I recently wrote about creating a tokeniser in C++ using Ragel. In that post I created a lexer which recognised a set of three tokens. In this post I’ll discuss creating a lexer by hand in Rust to recognise the same language.

The job of a tokeniser is to break down a piece of text into a set of tokens, or lexemes. A token consists of two parts: an identifier which defines which regular expression that the string matched, and an optional value. If the token represents a number then the value might be the integer representation of that number.

We defined the language with the following three definitions:

Var ::= [a-z][a-z0-9]*
Num ::= [0-9]+
Plus ::= '+'

There are four main stages to creating a tokeniser from this set of regular expressions:

Create a nondeterministic finite-state automata (NFA) which accepts each of the regular expressions.
Join these automata together to create a single NFA which accepts all of the regular expressions.
Convert the NFA to a deterministic finite-state automata (DFA).
Write a program which implements the DFA.

NFA Construction

We will start with the simplest regular expression '+'. The NFA that accepts this has two states. We move from the initial state to the final state when we see '+'.

The machine that accepts Num tokens also has two states. The first transition from the start state ensures that at least one digit is matched. We then allow any number of additional numbers with a looping transition.

For the final FSA which accepts Vars has two main parts. The first is the two-node machine which accepts [a-z]. This is then joined with a ε transition to a one-state machine which accepts the Kleene star of [a-z0-9]*.

NFA Composition

Once we have the NFAs for each of the individual tokens we can combine them into a single machine. We do this by adding a new start state and then adding a ε transition from this new state to the start state of each of the individual machines.

We have also added in a machine which accepts any number of whitespace characters. This will allow us to separate each token by optional whitespace.

NFA → DFA Conversion

To convert the NFA to a DFA we will create a DFA in which each state represents a set of NFA states.

To calculate these sets of states we compute the ε-closure of each transition in the NFA. We start with the ε-closure of the start state and repeatedly choose a transition we haven’t followed and then add in a new transition going to the ε-closure of that transition.

DFA State	NFA States	Transitions	Final
S	{ `s`, `v1`, `p1`, `w1` }	`\s` -> A	n
		`\+` -> B
		`0-9` -> C
		`a-z` -> D
A	{ `w1` }	`\s` -> A	y
B	{ `p2` }		y
C	{ `n2` }	`0-9` -> C	y
D	{ `v2`, `v3` }	`a-z` -> E	y
		`0-9` -> e
E	{ `v3` }	`a-z` -> E	y
		`0-9` -> e

Implementation!

Now all that’s left to do is to write a program that implements the state machine we have constructed. We’ll start by defining the three structures that we will use.

The first is the Tok structure. This represents each token we generate. A nice feature of Rust is that we can store the value directly in the enum along with the identifier of the token. This is really just syntactic sugar for a tagged union in c-land. There is one major advantage though. You can’t read the wrong type of data out due to the type safety enforced by the Rust language.

#[derive(Debug,PartialEq)]
pub enum Tok {
    Var(String),
    Num(i32),
    Plus
}

Next up is the state enumeration. It’s a pretty simple one. One thing to note is that we make sure it implements PartialEq so we can compare values of it later on.

#[derive(PartialEq)]
enum State {
    S, A, B, C, D
}

The final structure is the tokeniser itself. This stores the buffer of characters to be tokenised, represented as a String, and the start of the token currently being tokenised. Once we finish consuming a token we advance this index so that the next token in the stream can be recognised.

pub struct Tokeniser {
    ts: usize,
    chars: String
}

The final state machine then loops through the characters in the buffer, starting from where it left off. If the character refers to a valid transition in the state machine it moves to that state and keeps on looking for more characters. Once we can no longer move to another state we return a token based on the last accepting state that we entered.

Seen as in this state machine we can’t move to any non-accepting states after entering an accepting state we make the simplifying assumption that the last accepting state was just the last state. If the last state wasn’t an accepting state then we return None. If the last state was an accepting state we return a Some(Tok) with the right data read from the slice of the buffer which we just recognised.

fn next_match(&mut self) -> Option<Tok> {

    loop {
        
        let mut state = State::S;
        let mut te = self.ts;

        for c in self.chars[self.ts..].chars() {

            // find the next transition in the
            //state machine
            let next = match state {
                State::S => match c {
                    ' ' | '\t' => Some(State::A),
                    '+' => Some(State::B),
                    '0'...'9' => Some(State::C),
                    'a'...'z' => Some(State::D),
                    _ => None
                },
                State::A => match c {
                    ' ' | '\t' => Some(State::A),
                    _ => None
                },
                State::B => None,
                State::C => match c {
                    '0'...'9' => Some(State::C),
                    _ => None
                },
                State::D => match c {
                    'a'...'z' | '0'...'9' => Some(State::D),
                    _ => None
                }
            };

            // If we found a transition then consume
            // the character and move to that state
            if let Some(next_state) = next {
                state = next_state;
                te += 1;
            } else {
                break;
            }
        }

        // once we can no longer match any more
        // characters we decide what token to return
        let token_str = &self.chars[self.ts..te];

        self.ts = te;

        // If we recognised some whitespace, look for
        // the next token instead
        if state == State::A {
            continue;
        }

        // Depending on which state we're in we know
        // which token we have just accepted
        return match state {
            State::B =>
              Some(Tok::Plus),
            State::C => 
              Some(Tok::Num(token_str.parse().unwrap())),
            State::D =>
              Some(Tok::Var(token_str.to_string())),
            _ => None
        };
    }
}

For more code take a look at the full Microcrate.

Microcrates: Little Flecks of Rust

Fri, 27 Mar 2015 00:00:00 +0000

Gists are really just git repos with no directory structure. This is something that some Rubyists have taken advantage of in the past to create “Microgems”. Would it be possible to do the same thing in Rust?

I was in the process of messing around with some Rust code. I had created a new crate with cargo and hacked around a bit. Not wanting to lose my changes I’d created a git repository too. We aren’t talking about much code here, just a few dozen lines in a single .rs file. I didn’t really want to create a full GitHub repo just for this small amount of code.

“Wouldn’t it be really nice if I could just put it all in a Gist” I thought.

By convention a Rust crate has a few different directories, src/ for all of the source files, test/ for integration tests and so on; at the top level is the Cargo.toml file containing the package metadata. We can’t use these in a Gist. We need to just stick our source files right next to the Cargo.toml. It turns out that configuring Cargo to understand our wacky new folder structure is actually quite simple. We just need to se the path to our source file in the Cargo.toml

[package]
 
name = "..."
version = "0.0.1"
authors = ["..."]
 
[[bin]]
    name = "example"
    path = "main.rs"

If you’re creating a library rather than an executable you’ll ned to use the [lib] section instead of the [[bin]] array. There’s more information about what you can configure for each target in the cargo docs. You can see the results in action in a simple hand-written lexer I wrote.

Tokenising in C++ with Ragel

Tue, 24 Mar 2015 00:00:00 +0000

Ragel is an alternative to Lex/Flex for creating lectures from a set of regular expressions. It is far more powerful however. Where Lex allows you to chain together a set of regular expressions and run some actions written in C or C++ in response to a match Ragel allows you to generate an arbitrary state machine by combining regular expressions and then run actions at any point in the matching of those expressions. It also supports a wide variety of ‘host’ languages, including C#, Ruby and Go.

We will attempt to recognise a simple set of tokens consisting of variables, numbers and the plus sign.

Var ::= [a-z][a-z0-9]*
Num ::= [0-9]+
Plus ::= '+'

To recognise it we define the following Ragel machine. It uses the shorthand syntax for defining a tokeniser machine. This allows us to define a set of regular expressions actions that will be run when the expression is matched similar to the functionality of lex.

%%{
  machine ExampleLexer;

  main := |*

    digit+ => {
      CAPTURE_TOKEN(Num);
      fbreak;
    };

    alpha (alpha | digit) * => {
      CAPTURE_TOKEN(Var);
      fbreak;
    };
    
    '+' => {
        CAPTURE_TOKEN(Plus);
        fbreak;
    };

    space+;
  *|;
}%%

Notice that we explicitly have to match whitespace and ignore it. We will define the macro CAPTURE_TOKEN to expand to the C++ code we require to return a token of that type.

We will represent tokens as a simple C++ struct:

struct Token
{
    enum TokenType {
        Var,
        Num,
        Plus,
        End,
        None
    };

    TokenType type;
    std::string value;
};

We now just need to define a simple C++ class to encapsulate the lexer’s state. We define the member variables p, pe and eof which Ragel needs to keep track of the state of the buffer and initialise them in the constructor. If the data was being read in chunks from a buffered source you would need to keep on updating these pointers until you reached the end of the input storem.

Since we are using a tokeniser machine we need to set up the ts and te variables which Ragel will set to point to the matched token in the buffer before calling each action.

The final set of Ragel variables (act, cs, top, stack) define the internal workings of the state machine.

class Lexer
{
public:
    Lexer(std::string& input)
      : buffer(input) {

        %%write init;

        // set up the buffer here
        p = buffer.c_str();
        pe = p + buffer.size();
        eof = pe;
    }

    Token* next() {
        auto token = new Token();
    
        token->type = Token::None;
    
        do {
    
            if (cs >= ExampleLexer_first_final) {
                token->type = Token::End;
            }
    
            %%write exec;
            
            if (cs == ExampleLexer_error) {
                token->type = Token::None
                return token;
            }
            
        } while (token->type == Token::None);
    
        return token;
    }

private:
    // buffer state
    const char* p, * pe, * eof;
    // current token
    const char* ts, * te;
    // machine state
    int act, cs, top, stack[1];

    std::string buffer;
};

The one last thing we need to define is the macro which captures the token. This macro basically writes directly into the token variable defined as a local in the next() method.

#define CAPTURE_TOKEN(t) \
  token->value = std::string(ts, te-ts); \
  token->type = Token::t

And there you go. All done. We can now pass a simple buffer to our lexer and keep on calling next() until we run out of tokens. This lecture will return the tokens one at a time as it reads them. You can have Ragel tokenise the whole input if you want by removing the fbreak calls from the actions and looping until you receive either None or End.

The full code for this article is in this Gist.

Joining the Disqusion

Tue, 17 Feb 2015 00:00:00 +0000

I have finally got around to adding commenting to this blog. In the end it turned out to be quite simple.

I chose to use Disqus as it was such an easy option to embed, and has support for many different blogging platforms. To embed it in Jekyll was as simple as copying the sample HTML from the Disqus website.

There are some subtleties to getting this all working properly with Jekyll though. If articles can appear on more than one page of the site you’ll need to make sure you set up a unique ID for each article and you’ll probably want to add a permalink to the article’s page as well.

Luckily with Jekyll this was quite simple. Each post already has a unique ID {{ post.id }} and you’re permalink URL can be found by adding the URL of the site to the {{ post.url }}. Putting this all together you get something like this:

  <!-- DISQUS -->
  <script type="text/javascript">
    /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
    var disqus_shortname = '{{ site.disqus_shortname }}'; // required: set your short name in config.yml
	var disqus_identifier = '{{ post.id }}';
	var disqus_title = '{{ post.title }}';
	var disqus_url = '{{ post.url | prepend: site.url }}';

    /* * * DON'T EDIT BELOW THIS LINE * * */
    (function() {
    var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
    dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
    (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
    })();
  </script>
  <div id="disqus_thread"></div>

Feel free to use this to embed Disqus into your site.

Sneakernet Push

Sun, 26 Oct 2014 00:00:00 +0000

I’ve been living without internet recently. That hasn’t stopped me writing code though. If anything it has accelerated the process. I’m mainly using Git these days. It’s more than at home working without a connection to my repos on GitHub. Every now and again I like to back things up though. That’s where the “Sneakernet Push” comes in.

Start by creating a new empty git repository somewhere on the drive you’re going to transfer your data with:

$ mkdir usb.git
$ cd usb.git
$ git init --bare

Add the new repo as a remote so you can push to it easily:

$ git remote add usb file:///Volumes/USB/usb.git

Now you can just push the branches you want to the new git repo on the USB stick:

$ git push usb master:master

The next step is a litre more difficult. Go forth and seek out some wild internet. From there you can get those precious bytes into the cloud with a couple of commands. Open up the usb.git folder, ad the internet remote and push the commits up to it.

$ git remote add origin http://github.com/myname/myrepo
$ git push origin master

Sit back, relax, grab a beverage and begin contemplating your next commit.

Testing Times

Tue, 21 Oct 2014 00:00:00 +0000

It’s been a hectic few months. I’ve just moved from the blistering sunshine of the south of England to the chilly embrace of the north. That means I’ve not been able to spend that much time writing code outside of work. Now things have calmed down a bit I decided to integrate google test into a C++ application I am developing. This is the story of just how easy that was.

There is a lot of code in the world. A lot of it is C++. Roughly 85% of that is unit testing frameworks. Seriously, the choices are in no way restrictive. Amongst the most highly praised at the moment is the Google C++ Testing Framework, or GTest for short.

Getting started with the framework is dead simple. Download the source, ./configure, make, and you’re done! It’s just as easy to integrate it into your project too. GTest has auto-discovery of unit tests so all you need to do to create your first test is compile and link in a file with a test definition. Something like this should suffice:

#include <gtest/gtest.h>

TEST(MyAwesomeTest,ThatPasses) {
    auto test_is_awesome = true;
    ASSERT_TRUE(test_is_awesome);
} 

With that compiled and linked I’ll reveal the next neat feature GTest has up it’s sleeve. If you’re writing tests for a command line app you can get guest to run unit tests with a call to RUN_ALL_TESTS(). But what if you’re not writing a standalone tool but a dynamic library? Well GTest has a minimalist test harness available as a library alongside the main libgtest too. Just link -lgtest_main on the command line along with your dynamic library and you should have yourself a working unit test runner.

One of the nicest features of GTest for me is the ability to EXPECT_ as well as ASSERT_, allowing you to continue to check more things in a test, even if one of your assumptions has failed. This gives you a bit more insight when a test fails as to the underlying cause of the failure.

I’ve barely touched the surface of what GTest can do here, quite deliberately I should add. There are plenty of resources out there to help you get started. GTest has tonnes of neat features, like test fixtures; parameterised tests; and what it calls ‘death tests’, which test program termination. Next time you find yourself looking for a C++ unit test framework GTest definitely deserves some considering.

The First Solution and The Best Solution

Sat, 31 May 2014 00:00:00 +0000

There is a saying. I’m not sure where it comes from. I’m not even sure if I made it up entirely. But I find it entirely appropriate when developing software.

The first solution is usually not the best solution. But it’s better than no solution at all.

Programming is all about solving problems. It’s about breaking difficult things into smaller things you already know how to do. Most of the time though it doesn’t really matter how good your solution is. Especially if it gets the job done.

A similar quote, attributed to Voltare, that’s probably a bit more popular goes something like this:

Best is the enemy of Good.

Obviously you could go to town with this idea and create some of the least maintainable software that man has ever seen.

Maximum Viable Product

If best is the enemy of good then Minimum Viable Product (MVP) is the enemy of getting anything half decent done at all. I don’t have a problem with MVP in theory. The idea of choosing to ship the smallest and simplest solution first and then build on it is a logical one. Developers get to see their software out in the real world quicker, users get something to play around with earlier, and the company’s bottom line gets a bit of a boost.

It’s very easy to take MVP too far though. Usually this is based on a mis-interpretation of exactly what the word ‘viable’ means.

The Tale of The Minimal Commute

A poor lonely user wants to get to work. He finds himself walking for hours a day and wishes there was a simpler solution. Deciding to commission some software engineers to solve his problem he draws up a list of requirements:

I need a wheeled vehicle
I should be able to get to work with it.

Seeing the requirements the software engineers flesh them out to a full design specification:

The deliverable should be a vehicle
The deliverable should have 4 wheels
The deliverable should have an engine
The deliverable should be capable of travelling 17.6 miles 5 days in a row without refuelling

All going well so far. It looks like our user is going to get a nice small car to drive to work in. Then MVP hits.

“Do we really need 4 wheels in the first version?” asks the first developer. “Could we not get away with something simpler, like 2 wheels” he proposes. After much discussion the team eventually decides that 2 wheels sounds like a good solution for the first version.

“The engine sounds complex” suggests a second developer. “I’m worried about how long that will take. Could we not get away without an engine in the first version and add one in later?”. Again the proposition is discussed and it is decided maybe the user could just push the vehicle himself initially.

The discussion continues over several days until it is decided that the user won’t push the car himself, but rather he will rely on some dogs to drag it. And also due to the technical complexity of balancing something on two wheels they will use two skis instead.

Time passes. Eventually the user is presented with the finished product. It’s not quite what he had in mind.

“It’s not quite got all the features of the final version” says the lead developer, patting the husky-sled earnestly. “We did manage to get some good features in though, like this bespoke cup-holder” he continues. Showing off the intricately carved wooden device as he does so.

“It’s not quite what I was looking for” the user replies. “It is a nice cup holder though. And I suppose I could get a new job in Svalbard. Then I would travel to work over snow and ice each day and this would work great”.

“I knew you’d like it” the development manager smiles. “Just wait until you see version 2.0”.

Loosing Focus

Mon, 03 Mar 2014 00:00:00 +0000

The air is warm. Too warm perhaps. There is almost no wind at all. What breeze there is does nothing to cool. Lights twinkle on the skyline in intricate patterns. I slide them out of focus. Snap.

Help, I'm Learning (and It's Fun)

Wed, 18 Sep 2013 00:00:00 +0000

It all started one fateful day, as these things have a habit of doing. A friend was playing a game on his laptop where he launched rockets into space. He informed me that it was called “Kerbal Space Program”. I thought nothing of it until I came across the game again in the Seam store. Now I’m hooked.

The game is still in beta, but there are still hours of fun to be had. The mechanics of the game are pretty simple. You take control of the fledgling space program of the great Kerbal nations. From a selection of stock parts, or extra ones if you use any of the multitude of mods, you can create endless rockets, planes and landers. With them you can do pretty much whatever you like. Land on other planets, dock in orbit, build massive space stations or just fly around in the planet’s atmosphere. In short you can explore.

The game is far from simple though, and it probably isn’t for everyone. Before I began playing the game I already had an interest in space exploration and technology. I saw a man jump from the edge of space, I follow @spacex on twitter, I watch space launches live and sometimes I spend hours watching obscure and grainy footage of almost unknown satellites being raised into orbit. I thought I knew my apogee from my perigee. How wrong I was.

I always harboured a suspicion that rocket science is far simpler than rocket scientists would have everyone else think. It’s just oxidiser and fuel. Thrust and drag. If this game is anything to go by I am far from the mark with this suspicion. I have learnt the most efficient way to raise periapsis. I have learnt that a more powerful engine is not always better, and neither is a more efficient one. I have learnt that simple things like circular orbits, docking and planetary (or münar) transfers are not quite so simple after all. I am still learning.

You can learn facts about space missions all you like. You can watch videos and think that it’s pretty impressive that they were taken on the moon forty years ago. That’s not quite the same as directly getting to grips with orbital mechanics. Until you have spent 20 minutes bashing two spacecraft together in a 94km orbit in a vain attempt to get one docking port close enough to the other you don’t quite appreciate the sheer accomplishments of people such as Neil Amstrong and the entire NASA team who made Gemini 8 a success. Loosing 3 Kerbals later on in the process when you forget to open the parachute on one of your spacecraft just adds insult to injury.

Despite its learning curve the game is a great deal of fun. Something about the cheery green Kerbals, their characteristic plucky attitude, and their blatant disregard for almost all forms of health and safety (I’m pretty sure half of them wouldn’t bother with space suits if they had the option) brings the game alive. You feel a ashamed when you let one down and he dies a fiery (or more usually explody) death, and you also feel pride as one sticks a flag into yet-another foreign planetary body.

I’ts not just the characters that make this game though. It’s the whole experience. From the frustration at the failed attempts to the explosions and the sense of accomplishment when something finally works. Kerbal Space Program is fun because it doesn’t cut any corners. It just tries to be as accurate as possible, however difficult that might make your life playing it. Give it a go. Worst case scenario you might learn something too.

A Splash of Colour

Tue, 02 Jul 2013 00:00:00 +0000

I began my text editing journey, as many do I am sure, with Notepad.exe. When I made the transition to OS X I began searching for an editor to replace it. The then-free Smultron was that editor initially. Since then I have used many editors, from the wonderful TextMate to the cheap-and-cheerful GNU nano. I even tried vim and, of course, Emacs.

In the endless search for the text editor I enjoy using most I always find myself returning to Emacs. There are a few important reasons for this, not least its configurability. Don’t take this as a dig against vim or TextMate. Both are powerful editors and very configurable. TextMate macros can be written in a variety of languages and, by default, it supports many Emacs-style keyboard bindings.

The king of text editing?

Configurability is an important factor for an editor. There are a lot of people on the planet who need to edit plain text files. While there is also a multitude of text editors the two don’t yet quite balance each other out. This imbalance is why programs like vim and Emacs always seem to win out again and again. A truly configurable and programmable editor allows someone to pick the one that most closely matches their desires, and fill in the remaining distance themselves.

Each different editor has a different set of features, from multiple cursors to advanced find-and-replace and powerful macro systems. The one thing that they all have in common is syntax highlighting. The text that these programs render has to be clear, readable, and pretty enough for you to stare at it for hours on end.

During my time tweaking my Emacs config, watching videos, and getting completely stuck I came across a neat collection of themes. The themes were in an older Emacs theme format though. Seen as this is the future and everything is open source I forked it, updated the themes to the new format and submitted a pull-request. I even got a bit carried away in the process and created a new theme of my own.

The results of my labours

Emacs is is by no means the easiest editor to pick up. It took me several applications of concerted effort to start to understand it. I am not going to pretend that I understand all of Emacs, I doubt anyone does, but over the years I have become more accustomed to it. It too has become more accustomed to me. I have my editor set up just as I want it, for now. When that changes I’m sure emacs will change with me.

Secure for the Common Case

Sun, 09 Jun 2013 00:00:00 +0000

“I’ll have to type in that thing again” my dad moaned. “They gave me a new laptop and I haven’t used it on the wireless yet.” he explained. The “thing” he was referring to of course was the wireless key. A completely incomprehensible string of numbers and letters that took up three lines on the back of the junk-mail envelope in his hand. I began wondering exactly how secure the house wireless password was.

The key itself was written on pieces of paper and haphazardly taped to the bottom of the router. It’s not really that much of a secret, provided you have access to the house already. It is not the ever so slight reduction in security that really matters here though. Someone’s ability to access the internet is the last thing you care about if they have broken into your house. Having a password so random and complex that you have to write it down everywhere, a password that you dread having to type in in case you make a mistake, is a level of complexity that is not needed. It is just an annoyance.

It all boils down to two questions. Is 9a05bbaf5c827459ae7711ce05 really more secure than Hundr3dw3ight Axolotl Tr33? Does the security gain justify the added pain of using the “random” one?

I would suggest that the answer to these questions is “possibly” and “probably not”. Not very definitive I know, it is hard to give definitive answers to these questions. You never really know what kind of threats you are going to be up against. Perhaps the first string is “more random”, whatever that means. Perhaps it contains more information. Perhaps it would be harder to attack with a brute force approach. Perhaps not. Susceptibility to brute force attacks depends upon the assumptions made by the attacker. Had they not realised that the password contained spaces, as many people’s don’t, then the second password would take a very long time to crack.

In computer science there is a saying “optimise for the common case”. It is used to argue that it is only worth optimising the bits of applications that people will use most often. Any time spent optimising other parts is not worth the gains. It is worth more to save a tenth of a second for instance in a browsers page load time than it is to save several seconds in it’s bug reporting screens.

There is more to it though. The phrase also discourages optimising some parts of the program at the expense of others, especially those parts that are used most often. You can’t optimise everything, but you can at least make the most common things as fast as possible.

Seen like this the most important feature of a good password, for WiFi or anything, is that it should be memorable. The “common case” for password use surely has to be remembering it and typing it in. You can’t make your password uncrackable, it just needs to be more difficult to break than any of those nearby. The barrier you have to beat doesn’t seem to be that high.

With security in mind all your wireless password needs to do in the most common case is to keep out the next-door neighbours who have run out of bandwidth on their own connection and attempt to piggyback yours. After all, if GCHQ wants to read your emails your wireless password isn’t going to stop them; it’ll just ask the people at the NSA.

How to Recognise a Tortoise

Fri, 07 Jun 2013 00:00:00 +0000

Computer vision is an area which I had a distant interest in for quite some time. I appreciate that the task of vision is a complex one and always imagined that there must be some great algorithms behind the task. The process of computer vision, like many areas of artificial intelligence, is one where it is so easy to view the actions of a computer as those of a person. It was this lingering interest, with a touch of reverence, that prompted me to choose a vision project in my final year of university. It completely changed the way I think about artificial intelligence.

My final year project performs statistical object recognition using Grassmannian Manifolds. It is an interesting technique. I enjoyed learning more and more about the field of object recognition throughout the process. The downside however was the more I learnt, the more areas I exposed to the light of knowledge the fewer areas there were for that great secret of artificial intelligence to hide. The secret of course is a disappointing one: it is all a sham. There is a man behind the curtain.

I don’t mean to belittle the achievements of artificial intelligence researchers here, there is a lot of great research going on out there. The problem I have is more one of perception. The more I learnt about the area the more the realisation dawned that I didn’t consider the process intelligence. The computers were just following an algorithm, carrying out a pre-determined process. Some would argue that the human brain is just a computer carrying out an incredibly complex program, however I feel that there is some form of difference. Until the algorithm, if there is one, that determines our actions is known the mist surrounding our actions will always separate my perception of humans and computers.

What About the Tortoise?

Intelligent or not I am proud of my application. It performs recognition using a statistical representation of images. If we want the computer to recognise an example object, such as George the tortoise, we first need to make sure that the program knows about tortoises. We do this using a set of images of several stock tortoises. The images of each tortoise are grouped together into a Maifold, a representation of them in a higher dimensional space.

When we want to recognise George we take some pictures of him too. With these images we compute his manifold representation too. From this representation we can decide what type of object George is. This stage is known as classification.

In my project a simple classifier is used that simply picks an object type based upon the closest objects that the program knows about in the higher dimensional space to the one it is trying to classify. This is referred as the K-Nearest Neighbour classifier by those in the know. It is simple, but effective. My application was ~88-92% accurate with the test data sets I used.

Can I See a Picture of George?

Sure can:

Cute isn’t he? Until next time, have fun!

Testing Testing Testing

Wed, 05 Jun 2013 00:00:00 +0000

Scripting languages are all the rage these days. I am a fan of Python, although you may prefer Ruby, Node.js or whatever it is the cool kids are using these days. The exact language that takes your fancy however doesn’t really matter. They almost all have one thing in common: Unit tests. This is not a startling realisation, nor is it restricted to the realms of modern scripting languages. I hear that even the assembly code running on the jet engine of your favourite airliner is unit tested (a reassuring fact). Startling or not what is important about unit testing with scripting languages is the ease at which the development cycle progresses. If only it could be as easy for other languages.

This ease of dynamic language test driven development is largely to do with automated test runners, such as nosy for Python. These tools are great, however I spend quite a lot of time programming in languages such as C and C++. I had to settle for manually running the test suite every time I made the change.

There is a limit on the number of times a programmer can repeat a task before he automates it. When I reached this limit I decided to introduce myself to Ruby, and Snooper was born.

Snooper is a lightweight test automation tool, it monitors files and folders while you work and re-runs your tests when you change something. Snooper doesn’t care what language you’re using or what framework you are testing with, it’s all configureable.

All snooper does is watch the filesystem for changes and runs a command when a given set of files changes. Dependant on the success or failure of the command a prettified status bar is printed, with plenty of colour to ensure that you can see the test status from far off. It is a small set of features, but a neat one (even if I do say so myself).

I hope you find it as useful as I do. Feel free to tweet me any cool uses of Snooper. Until next time, keep on testing!