blag/pages/posts/2017-08-01-delimcc.md

title	date
Delimited Continuations, Urn and Lua	August 1, 2017

As some of you might know, Urn is my current pet project. This means that any potential upcoming blag posts are going to involve it in some way or another, and that includes this one. For the uninitiated, Urn is a programming language which compiles to Lua¹, in the Lisp tradition, with no clear ascendance: We take inspiration from several Lisps, most notably Common Lisp and Scheme.

As functional programmers at heart, we claim to be minimalists: Urn is reduced to 12 core forms before compilation, with some of those being redundant (e.g. having separate define and define-macro builtins instead of indicating that the definition is a macro through a parameter). On top of these primitives we build all our abstraction, and one such abstraction is what this post is about: Delimited continuations.

Delimited continuations are a powerful control abstraction first introduced by Matthias Felleisein², initially meant as a generalisation of several other control primitives such as call-with-current-continuation{.scheme} from Scheme among others. However, whereas call-with-current-continuation{.scheme} captures a continuation representing the state of the entire program after that point, delimited continuations only reify a slice of program state. In this, they are cheaper to build and invoke, and as such may be used to implement e.g. lightweight threading primitives.

While this may sound rather limiting, there are very few constructs that can simultaneously be implemented with call-with-current-continuation{.scheme} without also being expressible in terms of delimited continuations. The converse, however, is untrue. While call/cc{.scheme} be used to implement any control abstraction, it can't implement any two control abstractions: the continuations it reifies are uncomposable³.

Delimited Continuations in Urn

Our implementation of delimited continuations follows the Guile Scheme tradition of two functions call-with-prompt and abort-to-prompt, which are semantically equivalent to the more traditional shift/reset. This is, however, merely an implementation detail, as both schemes are available.

We have decided to base our implementation on Lua's existing coroutine machinery instead of implementing an ad-hoc solution especially for Urn. This lets us reuse and integrate with existing Lua code, which is one of the goals for the language.

call-with-prompt is used to introduce a prompt into scope, which delimits a frame execution and sets up an abort handler with the specified tag. Later on, calls to abort-to-prompt reify the rest of the program slice's state and jump into the handler set up.

(call/p 'a-prompt-tag
  (lambda ()
    ; code to run with the prompt
    )
  (lambda (k)
    ; abort handler
    ))

One limitation of the current implementation is that the continuation, when invoked, will no longer have the prompt in scope. A simple way to get around this is to store the prompt tag and handler in values and use call/p⁴ again instead of directly calling the continuation.

Unfortunately, being implemented on top of Lua coroutines does bring one significant disadvantage: The reified continuations are single-use. After a continuation has reached the end of its control frame, there's no way to make it go back, and there's no way to copy continuations either (while we have a wrapper around coroutines, the coroutines themselves are opaque objects, and there's no equivalent of string.dump{.lua} to, for instance, decompile and recompile them)

Why?

In my opinion (which, like it or not, is the opinion of the Urn team), Guile-style delimited continuations provide a much better abstraction than operating with Lua coroutines directly, which may be error prone and feels out of place in a functional-first programming language.

As a final motivating example, below is an in-depth explanation of a tiny cooperative task scheduler.

(defun run-tasks (&tasks) ; 1
  (loop [(queue tasks)] ; 2
    [(empty? queue)]    ; 2
    (call/p 'task (car queue)
      (lambda (k)
        (when (alive? k)
          (push-cdr! queue k)))) ; 3
    (recur (cdr queue)))) ; 4

1. We begin, of course, by defining our function. As inputs, we take a list of tasks to run, which are generally functions, but may be Lua coroutines (threads) or existing continuations, too. As a sidenote, in Urn, variadic arguments have & prepended to them, instead of having symbols beginning of & acting as modifiers in a lambda-list. For clarity, that is wholly equivalent to (defun run-tasks (&rest tasks){.lisp}.

2. Then, we take the first element of the queue as the current task to run, and set up a prompt of execution. The task will run until it hits an abort-to-prompt, at which point it will be interrupted and the handler will be invoked.

3. The handler inspects the reified continuation to see if it is suitable for being scheduled again, and if so, pushes it to the end of the queue. This means it'll be the first task to execute again when the scheduler is done with the current set of working tasks.

4. We loop back to the start with the first element (the task we just executed) removed.

Believe it or not, the above is a fully functioning cooperative scheduler that can execute any number of tasks.⁵

Conclusion

I think that the addition of delimited continuations to Urn brings a much needer change in the direction of the project: Moving away from ad-hoc abstraction to structured, proven abstraction. Hopefully this is the first of many to come.

---

1. Though this might come off as a weird decision to some, there is a logical reason behind it: Urn was initially meant to be used in the ComputerCraft mod for Minecraft, which uses the Lua programming language, though the language has outgrown it by now. For example, the experimental readline support is being implemented with the LuaJIT foreign function interface. ↩︎

2. The Theory and Practice of First-Class Prompts. ↩︎

3. Oleg Kiselyov demonstrates here that abstractions built on call/cc{.scheme} do not compose. ↩︎

4. call-with-prompt is a bit of a mouthful, so the alias call/p is blessed. ↩︎

5. There's a working example here (with syntax highlighting) as runnable Urn. Clone the compiler then execute lua bin/urn.lua --run tasks.lisp. ↩︎