I’ve gone one more round with my indentation-sensitive Erlang scanner/parser. For a while, it looked like I was winning, but eventually, I had to admit defeat in the face of funs and record constructors.

Again, the approach I thought I’d take was to add indentation tokens to the normal scanner, and then add rule clauses to the normal grammar to make it understand both indentation-sensitive code, and all the code you’re accustomed to writing. The normal parser is an LR(1) grammar, which means that the additions have to be quite symmetrical in order to work.

This seemed to work in just about all the places that mattered, but I was stumped by funs and record constructors. The main with funs was that you could no longer write them in the conventional way (failing one of my preconditions), and record constructors usually cannot be written without including a few commans in a way that was …uhm, less than obvious.

I include the test module, which pretty much illustrates what works, and what doesn’t. At this point, I doubt that I can achieve better than a 90% solution, which is probably just good enough to eventually drive people crazy.

It’s been fun, though. I’m have no particular craving for indentation-sensitive syntax myself, so I thought I’d tackle this as a learning experience. Not for the first time, I feel like concluding that retrofitting concepts onto existing programming languages is usually very difficult to do well.

The code is at http://svn.ulf.wiger.net/indent/branches/0.3

-module(test).

-record(r,{a,b}).

-compile(export_all).
-scan(indentation).

f(X) ->
    X+2

g(X) ->
    X+4
.          % dot can either be 'outdented' or
           % terminating last line

g1(X) ->
    X+4.

h(X) ->
    Y = case X of
          a ->
            {a}
          b ->
            {b}
    end          % end is optional
    Y

h1(X) ->
    Y = case X of
          a ->
            {a}
          b ->
            {b}
    Y

h2(X) ->
    case X of
        a ->
            1
        b ->
            2

h3(X) ->
    case X of
        a -> 1;  % must use semicolon here 
        b -> 2

i(a) -> a
i(b) -> b

j(A,B) ->
    {A
    B}

j1(A,B,C) ->
    {A
     B     % indents must be greater than 1;
     C}    % else they count as aligned

k() ->
    S = "a string "
        "spanning multiple "
        "lines"
    {S}

l0() ->
    fun(1) -> 1; (2) -> 2 end.

l1() ->
    fun            % must break line here and indent.  
      (1) ->
            1
      (2) ->
            2

%%% This, alas, doesn't work 
%%%
%%% l2() ->
%%%     fun(1) ->
%%%             1;
%%%        (2) ->
%%%             2
%%%     end

m() ->
    #r{a = 1, b = 2}.

m1() ->
    R = #r{a = 1}
    R#r.a

%%% indentation syntax works poorly for record assignment.
%%% Both commas are needed 
m2() ->
    R = #r{a = 1,
           b = 2},
    R

test() ->
    2 = f(0),
    4 = f(2),
    4 = g(0),
    4 = g1(0),
    8 = g(4),
    {a} = h(a),
    {b} = h(b),
    {a} = h1(a),
    {b} = h1(b),
    1 = h2(a),
    2 = h2(b),
    1 = h3(a),
    2 = h3(b),
    a = i(a),
    b = i(b),
    {a,b} = j(a,b),
    {a,b,c} = j1(a,b,c),
    {"a string spanning multiple lines"} = k(),
    F0 = l0(), 1 = F0(1), 2 = F0(2),
    F1 = l1(), 1 = F1(1), 2 = F1(2),
    {r,1,2} = m(),
    1 = m1(),
    ok.

(See The full article for more details).

Panel discussion (Download)
[See post to watch Flash video]

I finally got around to editing out those few details in the discussion that initially prevented me from posting the video.

My own favourite extract from the discussion:

Simon: “If a boat is sinking with 100 people on board and you scatter some life belts in the region, then they’ll probably swim and cling on to them, right? Just because life belts float. So functional programming is good stuff, right, and is like this granite, then we’ll end up with people clinging on to it.”
Joe: “But … granite sinks.”
Simon: “Uuhm, what sinks?”
Joe: “Granite - sinks”
Simon: “…So I shouldn’t mix metaphors!”

(Thanks, Bjarne Däcker, for producing the graphs).

I decided to introduce another token, ‘GAP’, to denote an empty line. Most likely, the scanner, in its current state, will not be able to handle empty lines with white space in them, etc, and the code is starting to look a bit confused. Oh well…

The toplevel rule for a function now becomes:


form -> function dot : '$1'.
form -> function 'GAP' : '$1'.


and the rule for alternative function clauses is as before:

function_clauses -> function_clause : ['$1'].
function_clauses ->
   function_clause ‘;’ function_clauses : ['$1'|'$3'].
function_clauses ->
  function_clause ‘OUT’ : ['$1'].
function_clauses ->
  function_clause ‘END’ function_clauses : ['$1'|'$3'].

The first two rules are the original rules for indentation-insensitive code. The last two are for the indentation tokens. The ‘OUT’ token is for symmetry, to match the ‘IN’ token after the arrow in function_body. Remember that indentation tokens are normalized in the scanner.

The test program now looks like this:

-module(test).

-compile(export_all).
-scan(indentation).

f(X) ->
    X+2

g(X) ->
    X+4
.

h(X) ->
    Y = case X of
          a ->
            {a}
          b ->
            {b}
    end
    Y

i(a) -> a
i(b) -> b

test() ->
    2 = f(0),
    4 = f(2),
    4 = g(0),
    8 = g(4),
    {a} = h(a),
    {b} = h(b),
    a = i(a),
    b = i(b),
    ok.

A little bit better. The ‘end’ tokens are still needed, though. One thing at a time…

I made a copy of erl_scan.erl (named erl_scan_ind.erl) and made it figure out indentation tokens. Then I added to the Erlang grammar in erl_parse.yrl. All the old rules remain, but some new rules were added to account for indentation tokens. For example:


clause_body -> '->' exprs: '$2'


becomes:


clause_body -> '->' exprs: '$2'
clause_body -> '->' 'IN' exprs 'OUT' : '$3'.


The indentation tokens I used were:

• ‘IN’ for indent
• ‘OUT’ for outdent (one for each matching indent)
• ‘ALIGN’ for when the next line keeps the same indentation
• ‘END’ when indentation goes back to zero

So a sequence of expressions could be written without commas, based on the following rule:


exprs -> expr : ['$1'].
exprs -> expr ‘,’ exprs : ['$1' | '$3'].
exprs -> expr ‘ALIGN’ exprs : ['$1' | '$3'].


My test program, which I was eventually able to compile, looked like this:

-module(test).

-compile(export_all).
-scan(indentation).

f(X) ->
    X+2
.

g(X) ->
    X+4
.

h(X) ->
    Y = case X of
          a ->
            {a}
          b ->
            {b}
    end
    Y
.

(Note especially that the final ‘end’ must be aligned with the Y, rather than the ‘case’. Perhaps this could be avoided…?)

The ending dots don’t have to be on their own line. Getting rid of them was too hard for me, since ‘dot’ is the end token for the Erlang grammar.

The -scan(indentation). attribute tells epp to switch to the indentation-sensitive scanner. -scan(normal). tells it to switch to the normal scanner.

I soon realised that I had to normalize the indentation tokens at the end of the scan. A few oddities were introduced, like inserting an ‘OUT’ token before each dot (and corresponding additions to the grammar). But for the most part, the additions to the grammar seemed fairly logical. The parser seems to handle all the old code, even though I should perhaps try recompiling the whole OTP source tree before making such a claim.

The code (based on OTP R12B-1) can be found at http://svn.ulf.wiger.net/indent/trunk

The grammar is still contaminated with some debug statements, which allowed me to print the productions as they were identified. They should of course be removed eventually.

I’m not convinced that this is really a good idea, but at least I had fun doing it.

• Simon Peyton-Jones, Microsoft Research
• Satnam Singh, Microsoft Research
• John Hughes, Chalmers
• John Launchbury, Galois

Some notable people in the audience were Seif Haridi, Joe Armstrong, Thomas Arts, Mary Sheeran, Koen Claessen

I want to extend my sincere thanks to all the speakers and distinguished guests, and post the seminar slides and videos of the talks for your enjoyment. Some Ericsson-specific content from the seminar has been omitted.

You may view them as streaming video, or download them and watch them locally (using e.g. VLC, QuickTime or the Wimpy FLV Player). I’ve noted some trouble viewing this many embedded flash streams on a single page, so I’ve posted each talk individually under the FP Seminar category.

Links follow, in the order in which they were presented:

Simon Peyton-Jones:
“Taming Effects - The Next Big Challenge”

Satnam Singh:
“Declarative Programming Techniques for Many-Core Architectures”

John Hughes:
“Testing with QuickCheck”

Simon Peyton-Jones:
“Composing Contracts - An Adventure in Financial Engineering”

John Launchbury: “High-Assurance Software”

(Update)
Panel discussion

(See The full article for more details).

John Launchbury
“High-Assurance Software” (Download) (Slides)
[See post to watch Flash video]

(See The full article for more details).

Simon Peyton-Jones
“Composing Contracts - An Adventure in Financial Engineering” (Download) (Slides)
[See post to watch Flash video]

(See The full article for more details).

John Hughes
“Testing with QuickCheck” (Download) (Slides)
[See post to watch Flash video]

(See The full article for more details).

Satnam Singh
“Declarative Programming Techniques for Many-Core Architectures” (Download) (Slides)
[See post to watch Flash video]