blag/pages/posts/2017-08-01-delimcc.md

---
title: Delimited Continuations, Urn and Lua
date: August 1, 2017
---

As some of you might know, [Urn](https://squiddev.github.io/urn) is my
current pet project. This means that any potential upcoming blag posts
are going to involve it in some way or another, and that includes this
one. For the uninitiated, Urn is a programming language which compiles
to Lua[^1], in the Lisp tradition, with no clear ascendance: We take
inspiration from several Lisps, most notably Common Lisp and Scheme.

As functional programmers at heart, we claim to be minimalists: Urn is
reduced to 12 core forms before compilation, with some of those being
redundant (e.g. having separate `define` and `define-macro` builtins
instead of indicating that the definition is a macro through
a parameter). On top of these primitives we build all our abstraction,
and one such abstraction is what this post is about: _Delimited
continuations_.

Delimited continuations are a powerful control abstraction first
introduced by Matthias Felleisein[^2], initially meant as
a generalisation of several other control primitives such as
`call-with-current-continuation`{.scheme} from Scheme among others.
However, whereas `call-with-current-continuation`{.scheme} captures
a continuation representing the state of the entire program after that
point, delimited continuations only reify a slice of program state. In
this, they are cheaper to build and invoke, and as such may be used to
implement e.g. lightweight threading primitives.

While this may sound rather limiting, there are very few constructs that
can simultaneously be implemented with
`call-with-current-continuation`{.scheme} without also being expressible
in terms of delimited continuations. The converse, however, is untrue.
While `call/cc`{.scheme} be used to implement any control abstraction,
it can't implement any _two_ control abstractions: the continuations it
reifies are uncomposable[^3].

### Delimited Continuations in Urn

Our implementation of delimited continuations follows the Guile Scheme
tradition of two functions `call-with-prompt` and `abort-to-prompt`,
which are semantically equivalent to the more traditional
`shift`/`reset`. This is, however, merely an implementation detail, as
both schemes are available.

We have decided to base our implementation on Lua's existing coroutine
machinery instead of implementing an ad-hoc solution especially for Urn.
This lets us reuse and integrate with existing Lua code, which is one of
the goals for the language.

`call-with-prompt` is used to introduce a _prompt_ into scope, which
delimits a frame execution and sets up an abort handler with the
specified tag. Later on, calls to `abort-to-prompt` reify the rest of
the program slice's state and jump into the handler set up.

```lisp
(call/p 'a-prompt-tag
  (lambda ()
    ; code to run with the prompt
    )
  (lambda (k)
    ; abort handler
    ))
```

One limitation of the current implementation is that the continuation,
when invoked, will no longer have the prompt in scope. A simple way to
get around this is to store the prompt tag and handler in values and use
`call/p`[^4] again instead of directly calling the continuation.

Unfortunately, being implemented on top of Lua coroutines does bring one
significant disadvantage: The reified continuations are single-use.
After a continuation has reached the end of its control frame, there's
no way to make it go back, and there's no way to copy continuations
either (while we have a wrapper around coroutines, the coroutines
themselves are opaque objects, and there's no equivalent of
`string.dump`{.lua} to, for instance, decompile and recompile them)

### Why?

In my opinion (which, like it or not, is the opinion of the Urn team),
Guile-style delimited continuations provide a much better abstraction
than operating with Lua coroutines directly, which may be error prone
and feels out of place in a functional-first programming language.

As a final motivating example, below is an in-depth explanation of
a tiny cooperative task scheduler.

```lisp
(defun run-tasks (&tasks) ; 1
  (loop [(queue tasks)] ; 2
    [(empty? queue)]    ; 2
    (call/p 'task (car queue)
      (lambda (k)
        (when (alive? k)
          (push-cdr! queue k)))) ; 3
    (recur (cdr queue)))) ; 4
```

1. We begin, of course, by defining our function. As inputs, we take
   a list of tasks to run, which are generally functions, but may be Lua
   coroutines (`threads`) or existing continuations, too. As a sidenote,
   in Urn, variadic arguments have `&` prepended to them, instead of
   having symbols beginning of `&` acting as modifiers in a lambda-list.
   For clarity, that is wholly equivalent to `(defun run-tasks (&rest
   tasks)`{.lisp}.

2. Then, we take the first element of the queue as the current task to
   run, and set up a prompt of execution. The task will run until it
   hits an `abort-to-prompt`, at which point it will be interrupted and
   the handler will be invoked.

3. The handler inspects the reified continuation to see if it is
   suitable for being scheduled again, and if so, pushes it to the end
   of the queue. This means it'll be the first task to execute again
   when the scheduler is done with the current set of working tasks.

4. We loop back to the start with the first element (the task we just
   executed) removed.

Believe it or not, the above is a fully functioning cooperative
scheduler that can execute any number of tasks.[^5]

### Conclusion

I think that the addition of delimited continuations to Urn brings
a much needer change in the direction of the project: Moving away from
ad-hoc abstraction to structured, proven abstraction. Hopefully this is
the first of many to come.


[^1]: Though this might come off as a weird decision to some, there is
a logical reason behind it: Urn was initially meant to be used in the
[ComputerCraft](https://computercraft.info) mod for Minecraft, which
uses the Lua programming language, though the language has outgrown it
by now. For example, the experimental `readline` support is being
implemented with the LuaJIT foreign function interface.

[^2]: [The Theory and Practice of First-Class
Prompts](http://www.cs.tufts.edu/~nr/cs257/archive/matthias-felleisen/prompts.pdf).

[^3]: Oleg Kiselyov demonstrates
[here](http://okmij.org/ftp/continuations/against-callcc.html#traps)
that abstractions built on `call/cc`{.scheme} do not compose.

[^4]: `call-with-prompt` is a bit of a mouthful, so the alias `call/p`
is blessed.

[^5]: There's a working example [here](/static/tasks.lisp) ([with syntax
highlighting](/static/tasks.lisp.html)) as runnable Urn. Clone the
compiler then execute `lua bin/urn.lua --run tasks.lisp`.

<!-- vim: tw=72
-->