--- title: Delimited Continuations, Urn and Lua date: August 1, 2017 --- As some of you might know, [Urn](https://squiddev.github.io/urn) is my current pet project. This means that any potential upcoming blag posts are going to involve it in some way or another, and that includes this one. For the uninitiated, Urn is a programming language which compiles to Lua[^1], in the Lisp tradition, with no clear ascendance: We take inspiration from several Lisps, most notably Common Lisp and Scheme. As functional programmers at heart, we claim to be minimalists: Urn is reduced to 12 core forms before compilation, with some of those being redundant (e.g. having separate `define` and `define-macro` builtins instead of indicating that the definition is a macro through a parameter). On top of these primitives we build all our abstraction, and one such abstraction is what this post is about: _Delimited continuations_. Delimited continuations are a powerful control abstraction first introduced by Matthias Felleisein[^2], initially meant as a generalisation of several other control primitives such as `call-with-current-continuation`{.scheme} from Scheme among others. However, whereas `call-with-current-continuation`{.scheme} captures a continuation representing the state of the entire program after that point, delimited continuations only reify a slice of program state. In this, they are cheaper to build and invoke, and as such may be used to implement e.g. lightweight threading primitives. While this may sound rather limiting, there are very few constructs that can simultaneously be implemented with `call-with-current-continuation`{.scheme} without also being expressible in terms of delimited continuations. The converse, however, is untrue. While `call/cc`{.scheme} be used to implement any control abstraction, it can't implement any _two_ control abstractions: the continuations it reifies are uncomposable[^3]. ### Delimited Continuations in Urn Our implementation of delimited continuations follows the Guile Scheme tradition of two functions `call-with-prompt` and `abort-to-prompt`, which are semantically equivalent to the more traditional `shift`/`reset`. This is, however, merely an implementation detail, as both schemes are available. We have decided to base our implementation on Lua's existing coroutine machinery instead of implementing an ad-hoc solution especially for Urn. This lets us reuse and integrate with existing Lua code, which is one of the goals for the language. `call-with-prompt` is used to introduce a _prompt_ into scope, which delimits a frame execution and sets up an abort handler with the specified tag. Later on, calls to `abort-to-prompt` reify the rest of the program slice's state and jump into the handler set up. ```lisp (call/p 'a-prompt-tag (lambda () ; code to run with the prompt ) (lambda (k) ; abort handler )) ``` One limitation of the current implementation is that the continuation, when invoked, will no longer have the prompt in scope. A simple way to get around this is to store the prompt tag and handler in values and use `call/p`[^4] again instead of directly calling the continuation. Unfortunately, being implemented on top of Lua coroutines does bring one significant disadvantage: The reified continuations are single-use. After a continuation has reached the end of its control frame, there's no way to make it go back, and there's no way to copy continuations either (while we have a wrapper around coroutines, the coroutines themselves are opaque objects, and there's no equivalent of `string.dump`{.lua} to, for instance, decompile and recompile them) ### Why? In my opinion (which, like it or not, is the opinion of the Urn team), Guile-style delimited continuations provide a much better abstraction than operating with Lua coroutines directly, which may be error prone and feels out of place in a functional-first programming language. As a final motivating example, below is an in-depth explanation of a tiny cooperative task scheduler. ```lisp (defun run-tasks (&tasks) ; 1 (loop [(queue tasks)] ; 2 [(empty? queue)] ; 2 (call/p 'task (car queue) (lambda (k) (when (alive? k) (push-cdr! queue k)))) ; 3 (recur (cdr queue)))) ; 4 ``` 1. We begin, of course, by defining our function. As inputs, we take a list of tasks to run, which are generally functions, but may be Lua coroutines (`threads`) or existing continuations, too. As a sidenote, in Urn, variadic arguments have `&` prepended to them, instead of having symbols beginning of `&` acting as modifiers in a lambda-list. For clarity, that is wholly equivalent to `(defun run-tasks (&rest tasks)`{.lisp}. 2. Then, we take the first element of the queue as the current task to run, and set up a prompt of execution. The task will run until it hits an `abort-to-prompt`, at which point it will be interrupted and the handler will be invoked. 3. The handler inspects the reified continuation to see if it is suitable for being scheduled again, and if so, pushes it to the end of the queue. This means it'll be the first task to execute again when the scheduler is done with the current set of working tasks. 4. We loop back to the start with the first element (the task we just executed) removed. Believe it or not, the above is a fully functioning cooperative scheduler that can execute any number of tasks.[^5] ### Conclusion I think that the addition of delimited continuations to Urn brings a much needer change in the direction of the project: Moving away from ad-hoc abstraction to structured, proven abstraction. Hopefully this is the first of many to come. [^1]: Though this might come off as a weird decision to some, there is a logical reason behind it: Urn was initially meant to be used in the [ComputerCraft](https://computercraft.info) mod for Minecraft, which uses the Lua programming language, though the language has outgrown it by now. For example, the experimental `readline` support is being implemented with the LuaJIT foreign function interface. [^2]: [The Theory and Practice of First-Class Prompts](http://www.cs.tufts.edu/~nr/cs257/archive/matthias-felleisen/prompts.pdf). [^3]: Oleg Kiselyov demonstrates [here](http://okmij.org/ftp/continuations/against-callcc.html#traps) that abstractions built on `call/cc`{.scheme} do not compose. [^4]: `call-with-prompt` is a bit of a mouthful, so the alias `call/p` is blessed. [^5]: There's a working example [here](/static/tasks.lisp) ([with syntax highlighting](/static/tasks.lisp.html)) as runnable Urn. Clone the compiler then execute `lua bin/urn.lua --run tasks.lisp`.