Session: Let's build a LISP lexer together - in Python

Over the course of the last 12 months, I have written several lexers and parsers. I gained some experience and want to introduce everyone to the idea of writing a basic lexer and parser. LISP [or its syntax which is based on S-expressions] is famously the simplest syntax, if you want to support nested structures. The syntax is based on a prefix notation where every instruction is wrapped by parentheses and the arguments are whitespace-separated: (function arg1 arg2).

In fact, Peter Norvig's article on writing a LISP interpreter in python is quite famous: https://norvig.com/lispy.html

In this coding dojo, we will join together in a session to write a lexer for LISP. First, I am going to introduce you to the challenge in a small presentation. I will also provide utilities for error messages and depending on the time, the parser [which is trivial to write] is provided or will be developed together. The core effort is developing the lexer (= tokenizer) in a communal effort.

In the end, we also built a tiny interpreter based on a function table.