Skip to content

Hacking the CPython interpreter

This blog post is inspired by the talk CPython Meanderings by James Powell.

Note: some of the details in this blog post are slightly out-of-date, as the Python interpreter has changed a little bit. You should still be able to follow along in spirit, though.

The CPython interpreter is not a hugely complicated software project, which means it's not too hard to add on to as a learning project. Thanks to the strong philosophy against premature optimisation, little interpreter behaviour is obscured.

Python is also largely very well-documented and has a friendly core development community which are great places to go for help. However, you can get very far with a little file searching and debugging.

If you know a little about the internals of Python already, particularly the bytecode compilation model it uses, and can follow a little C, you'll have no problem with this. I won't go into detail about Python bytecode, but here are some recommended reading links:

So what are we going to do with CPython? Well, since this is not really a practical approach to extending the language, we might as well go with something silly. So have you heard of Code Golf? It's a "sport" where you aim to write a program to complete a particular task, but with the code as short as possible. Python is not normally hugely competitive, as languages go, so I thought I'd add something to it to make it better. When golfing, best practices truly go out the window, such as correct error handling. Python, however, is not as forgiving with errors as (say) JavaScript or a Shell language, so let's "fix" that!

I'm going to add a short-circuiting inline exception catching operator, as an expression-based shorthand for try: .../except: ....

Let's define the operator's behaviour a bit more specifically:

  • we'll call it ?, since the ? character is not used anywhere in vanilla Python syntax
  • it will have looser precedence than or
  • it will still always print the exception when it occurs to make debugging easier

Since this feature is actually a terrible idea, I'm going to call it Exception Mishandling.

By the way, I'm calling this project Whython (because, why would I do this?), so I'll be naming things using a Py_Y prefix.

A final warning: this post will walk you through almost exactly the steps I took, including all the troubleshooting that came with it. I hope to show you that you could have worked all of this out yourself by reading relevant documentation, and some educated guesswork informed by just a bit of knowledge of conventions used in C and Python. I will skip over some typos and other obvious mistakes I made though.

Initial setup

First, get yourself a copy of the Python source code. For these purposes I'll be using Python 3.9.5 taken directly from the 3.9 branch on GitHub.

1
2
3
4
$ git clone [email protected]:python/cpython.git
$ cd cpython
$ git fetch 3.9
$ git checkout 3.9

Now, let's create a branch for our new feature:

1
$ git checkout -b exception_mishandling

And finally, create an initial build from here. I'm passing the --with-pydebug flag to configure to give us some helpful debugging information. (Again, this isn't something I magically knew - you can find a list of the options with ./configure --help, and then scroll through to look for debugging-related ones)

1
2
3
4
$ ./configure --with-pydebug
$ make
$ ./python --version
Python 3.9.5+

Parsing

The first step in the Python interpreter pipeline is parsing. Let's add our ? operator to the parser.

The Python parser, since version 3.9, uses a PEG grammar which generates a parser. Instructions for working on it are helpfully given in the devguide.

First, let's define the question mark token in Grammar/tokens:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
--- a/Grammar/Tokens
+++ b/Grammar/Tokens
@@ -53,6 +53,7 @@ ATEQUAL                 '@='
 RARROW                  '->'
 ELLIPSIS                '...'
 COLONEQUAL              ':='
+Y_EXC_MISHANDLING         '?'

 OP
 AWAIT

Now we'll add the grammar rule for a ? expression. I don't fully understand this, to be honest, but it's largely copied from some of the existing boolean and binary operators' definitions. It also references a _Py_ExcMishandle action, which I think will create the AST node.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
--- a/Grammar/python.gram
+++ b/Grammar/python.gram
@@ -380,6 +380,9 @@ lambda_param_maybe_default[NameDefaultPair*]:
     | a=lambda_param c=default? &':' { _PyPegen_name_default_pair(p, a, c, NULL) }
 lambda_param[arg_ty]: a=NAME { _Py_arg(a->v.Name.id, NULL, NULL, EXTRA) }

+y_exc_mishandle[expr_ty] (memo):
+    | a=disjunction '?' b=y_exc_mishandle { _Py_Y_ExcMishandle(a, b, EXTRA) }
+    | disjunction
 disjunction[expr_ty] (memo):
     | a=conjunction b=('or' c=conjunction { c })+ { _Py_BoolOp(
         Or,

But it needs to be used somewhere else in the grammar, because otherwise it would be unreachable. Since we're adding it with lower precedence than disjunction (an or expression), we'll replace all other uses of disjunction with y_exc_mishandle:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
--- a/Grammar/python.gram
+++ b/Grammar/python.gram
@@ -327,8 +327,8 @@ expressions[expr_ty]:
     | a=expression ',' { _Py_Tuple(CHECK(_PyPegen_singleton_seq(p, a)), Load, EXTRA) }
     | expression
 expression[expr_ty] (memo):
-    | a=disjunction 'if' b=disjunction 'else' c=expression { _Py_IfExp(b, a, c, EXTRA) }
-    | disjunction
+    | a=y_exc_mishandle 'if' b=y_exc_mishandle 'else' c=expression { _Py_IfExp(b, a, c, EXTRA) }
+    | y_exc_mishandle
     | lambdef

 lambdef[expr_ty]:
@@ -522,9 +525,9 @@ kvpair[KeyValuePair*]: a=expression ':' b=expression { _PyPegen_key_value_pair(p
 for_if_clauses[asdl_seq*]:
     | for_if_clause+
 for_if_clause[comprehension_ty]:
-    | ASYNC 'for' a=star_targets 'in' ~ b=disjunction c=('if' z=disjunction { z })* {
+    | ASYNC 'for' a=star_targets 'in' ~ b=y_exc_mishandle c=('if' z=y_exc_mishandle { z })* {
         CHECK_VERSION(6, "Async comprehensions are", _Py_comprehension(a, b, c, 1, p->arena)) }
-    | 'for' a=star_targets 'in' ~ b=disjunction c=('if' z=disjunction { z })* {
+    | 'for' a=star_targets 'in' ~ b=y_exc_mishandle c=('if' z=y_exc_mishandle { z })* {
         _Py_comprehension(a, b, c, 0, p->arena) }
     | invalid_for_target

We also need to define the new Y_ExcMishandle AST node type in Python.asdl:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
--- a/Parser/Python.asdl
+++ b/Parser/Python.asdl
@@ -53,6 +53,7 @@ module Python

           -- BoolOp() can use left & right?
     expr = BoolOp(boolop op, expr* values)
+         | Y_ExcMishandle(expr try_expr, expr except_expr)
          | NamedExpr(expr target, expr value)
          | BinOp(expr left, operator op, expr right)
          | UnaryOp(unaryop op, expr operand)

This AST item is the action that gets called in the generated parser, I think.

Finally, we need to regenerate the parser code, as described in the devguide: 

1
2
3
4
$ ./configure --with-pydebug  # recreate Makefile
$ make regen-token
$ make regen-ast
$ make regen-pegen

Testing it out

1
2
$ ./configure --with-pydebug  # not sure if this was actually necessary to do again
$ make

And just like that, it actually compiled, with few warnings and no errors! I did scan through the compiler output, though, to see how bad GCC thought my code was so far, and I found only one noticeable problem, repeated a few times:

1
2
3
Python/ast.c:223:5: warning: enumeration value ‘Y_ExcMishandle_kind’ not handled in switch [-Wswitch]
Python/compile.c:4993:5: warning: enumeration value ‘Y_ExcMishandle_kind’ not handled in switch [-Wswitch]
Python/symtable.c:1535:5: warning: enumeration value ‘Y_ExcMishandle_kind’ not handled in switch [-Wswitch]

Some issues in compile.c were to be expected, because we haven't implemented any support beyond parsing yet. I didn't know what symtable.c was until now, but it turns out it's an intermediate step in compilation that works out variable scoping.

However, an issue in ast.c seemed more pressing. It was in the validate_expr function, which seemed to be responsible for checking semantic issues in the syntax that couldn't be identified by the parser.

Just for fun, I decided to try our newly built Python anyway: - python --version worked - python alone worked and gave me a REPL with most functionality working fine

I knew for a fact simply trying to run some code with a ? operator in it would crash the interpreter. Instead I decided to try, first of all, using the ast module to see if this expression could be correctly parsed:

1
2
3
4
5
$ python -m ast << EOF
1 ? 2
EOF
python: Parser/pegen/pegen.c:1161: _PyPegen_run_parser: Assertion `PyAST_Validate(res)' failed.
Abort (core dumped)

Yikes. I got greedy - I guess the AST validation was necessary after all.

AST validation

Let's find the definition of the function that errored:

1
2
3
4
$ grep -r PyAST_Validate
-*- snip -*-
Python/ast.c:PyAST_Validate(mod_ty mod)
-*- snip -*-

and analysing the functions it calls, we see that validate_expr was almost certainly the culprit: at the bottom of the function is a handler for if the switch-case doesn't match anything:

1
2
PyErr_SetString(PyExc_SystemError, "unexpected expression");
return 0;

So, drawing heavy inspiration from the nearby expression handling code for binary operations, I came up with this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
--- a/Python/ast.c
+++ b/Python/ast.c
@@ -227,6 +227,9 @@ validate_expr(expr_ty exp, expr_context_ty ctx)
             return 0;
         }
         return validate_exprs(exp->v.BoolOp.values, Load, 0);
+    case Y_ExcMishandle_kind:
+        return validate_expr(exp->v.Y_ExcMishandle.try_body, Load) &&
+            validate_expr(exp->v.Y_ExcMishandle.except_body, Load);
     case BinOp_kind:
         return validate_expr(exp->v.BinOp.left, Load) &&
             validate_expr(exp->v.BinOp.right, Load);

And parsing it now works!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
$ make
$ ./python -m ast << EOF
1 ? 2
EOF
Module(
   body=[
      Expr(
         value=Y_ExcMishandle(
            try_body=Constant(value=1),
            except_body=Constant(value=2)))],
   type_ignores=[])

Compiling

If we take the next step in the Python interpreter's pipline, and try to compile some code that uses our operator, we'll see an unexpected result:

1
2
>>> compile("1 ? 2", "string", "eval")
<code object <module> at 0x7fe4b6af0110, file "string", line 1>

It seemed to compile just fine! Weird, huh? Now, if we run it, we're surely not going to get a success...

1
2
3
>>> eval(_)
python: Python/ceval.c:2055: _PyEval_EvalFrameDefault: Assertion `EMPTY()' failed.
Abort (core dumped)

And so the rabbit hole goes deeper. I want to see why that compiled successfully, and what bytecode it produced:

1
2
3
4
5
>>> from dis import dis
>>> compile("1 ? 2", "string", "eval")
<code object <module> at 0x7fe4b6af0110, file "string", line 1>
>>> dis(_)
  1           0 RETURN_VALUE

I think I know why it crashed now: it tried to return the top of the stack, but nothing had been pushed on to it. I'm also guessing that the reason the code is so short is because the compiler didn't know how to handle our new AST node, but it added the RETURN_VALUE because all code must end in it, or at least it does by default.

Interestingly, if we try to compile it in exec mode instead of eval mode, it does crash while compiling:

1
2
3
>>> code = compile("1 ? 2", "string", "exec")
python: Python/compile.c:5507: stackdepth: Assertion `depth >= 0' failed.
Abort (core dumped)

Anyway, let's get on to dealing with compilation.

Symbol table

First I want to have a look at the symtable.c we found earlier. The switch statement our compiler was complaining about is here:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
switch (e->kind) {
case NamedExpr_kind:
    if(!symtable_handle_namedexpr(st, e))
        VISIT_QUIT(st, 0);
    break;
case BoolOp_kind:
    VISIT_SEQ(st, expr, e->v.BoolOp.values);
    break;
case BinOp_kind:
    VISIT(st, expr, e->v.BinOp.left);
    VISIT(st, expr, e->v.BinOp.right);
    break;
case UnaryOp_kind:
    VISIT(st, expr, e->v.UnaryOp.operand);
    break;
-*- snip -*-
}

As before, we can pretty much copy from BinOp's implementation without too much bother:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
--- a/Python/symtable.c
+++ b/Python/symtable.c
@@ -1540,6 +1540,10 @@ symtable_visit_expr(struct symtable *st, expr_ty e)
     case BoolOp_kind:
         VISIT_SEQ(st, expr, e->v.BoolOp.values);
         break;
+    case Y_ExcMishandle_kind:
+        VISIT(st, expr, e->v.Y_ExcMishandle.try_body);
+        VISIT(st, expr, e->v.Y_ExcMishandle.except_body);
+        break;
     case BinOp_kind:
         VISIT(st, expr, e->v.BinOp.left);
         VISIT(st, expr, e->v.BinOp.right);

and as far as I can tell, that's all we need to do here.

Understanding try-except

Now let's look into how we can compile a ? b expressions into bytecode.

Control-flow graphs

The devguide page for the CPython compiler informs us of the two major remaining steps: transformation of AST into a Control Flow Graph, and then assembly into raw bytecode. The control flow graph is very close to the final bytecode, but does not have jump instructions "resolved", so-to-speak; they are described as labels only. The CFG is implemented using "basic blocks", which are just "chunks" of bytecode, but the jump instructions point to other basic blocks instead of using offset numbers. When the compiler emits an instruction, it does so by appending it to the active basic block, which is changed using compiler_use_next_block.

By the way, I just inferred most of this information while reading the source code in compile.c. It's surprisingly simple to follow.

Exception handling and frame blocks

Let's disassemble a try-except statement in the simplest case, from which we can take inspiration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
>>> dis("try: a\nexcept: b")
  1           0 SETUP_FINALLY            8 (to 10)
              2 LOAD_NAME                0 (a)
              4 POP_TOP
              6 POP_BLOCK
              8 JUMP_FORWARD            16 (to 26)

  2     >>   10 POP_TOP
             12 POP_TOP
             14 POP_TOP
             16 LOAD_NAME                1 (b)
             18 POP_TOP
             20 POP_EXCEPT
             22 JUMP_FORWARD             2 (to 26)
             24 RERAISE
        >>   26 LOAD_CONST               0 (None)
             28 RETURN_VALUE

Each Python stack frame has a stack of blocks, which represent loops and try-except syntactic blocks. They are used to keep track of where to jump if an exception occurs, and where to jump with the break and continue statements (and the corresponding BREAK and CONTINUE opcodes).

Again, I learnt this by just examining CPython's code (ceval.c, the main interpreter loop), and reading the documentation for [dis], I didn't just magically find this out.

Here's a simplified and annotated bytecode template that we need to write for an exception mishandling expression like a ? b, and that also explains the blocks in more detail:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Line       Byte Opcode                   Argument

                # Push frame block telling interpreter where to jump in case an exception occurs
 1            0 SETUP_FINALLY            8 (to 8)

                # Evaluate contents of try block (in this case just the expression a)
              2 LOAD_NAME                0 (a)

                # If we reached this point, the try block was evaluated without error. We need to remove the frame block
                # so that future code doesn't keep the exception handler
              4 POP_BLOCK

                # Skip the except handler block because no error occurred.
              6 JUMP_FORWARD            12 (to 18)

        >>      # This is where we get if an error occurs. Here the interpreter will push a special frame block,
                # "EXCEPT_HANDLER", which is just used to record the fact we're inside an except block so Python can
                # reset its internal state afterwards.

                # The interpreter will also push 3 things: exception type, exception object, and traceback object. We
                # don't care about any of these, so pop them all.
              8 POP_TOP     # type
             10 POP_TOP     # exc
             12 POP_TOP     # tb

                # Evaluate contents of except block (in this case just the expression b)
             14 LOAD_NAME                1 (b)

                # Tell Python we've finished in the except block, and to reset its internal state (clear `sys.exc_info`).
             16 POP_EXCEPT

        >>      # Here is where we'll jump to if no error occurred.

                # End of the expression, so return the relevant value
             18 RETURN_VALUE

Back to compiling

All that remains is to write the code to compile it. Here's the function template we'll need:

1
2
3
4
static int
compiler_exception_mishandling(struct compiler *c, expr_ty e) {
    return 1;
}

The function returns either 0 or 1 to indicate whether compilation succeeded (0 for success, 1 for failure).

As a sanity check, we'll assert that the expression is the right kind. I'm only adding this because all the other compiler functions have it.

1
assert(e->kind == Y_ExcMishandle_kind);

Now, to save myself the effort of writing the meat of the function from scratch, I'm going to copy the set of functions called by compiler_try_except by getting a trace of all the functions called with a debugger.

1
2
$ make
$ gdb ./python

I'll start the interpreter first, so we don't break on a whole load of things we don't care about, as it starts up: 

1
2
3
4
(gdb) run
Python 3.9.5+ [...]
Type "help", "copyright", "credits" or "license" for more information.
>>> ^C

(get back to GDB by pressing Ctrl+C)

How I'll break on the first line of compiler_try_except (the line number may be different for you depending on what version you start from and what modifications you've made already), and type in a try-except statement that roughly corresponds to the desired behaviour of exception mishandling: 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
(gdb) break Python/compile.c:3080
Breakpoint 1 at 0x55555569632d: file Python/compile.c, line 3080.
(gdb) continue
Continuing.

>>> try: 1
... except: 1
...

Breakpoint 1, compiler_try_except (c=0x7fffffffd570, s=0x555555c4a960) at Python/compile.c:3080

My plan from here is to just keep stepping through the compiler function using next and copy all the functions that are called into compiler_exception_mishandling, tweaking them as necessary:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
3080        body = compiler_new_block(c);
(gdb) next
3081        except = compiler_new_block(c);
(gdb) 
3082        orelse = compiler_new_block(c);
(gdb) 
3083        end = compiler_new_block(c);
(gdb) 
3084        if (body == NULL || except == NULL || orelse == NULL || end == NULL)
(gdb) 
3086        ADDOP_JREL(c, SETUP_FINALLY, except);
(gdb) 
3087        compiler_use_next_block(c, body);
(gdb) 
3088        if (!compiler_push_fblock(c, TRY_EXCEPT, body, NULL, NULL))
(gdb) 
3090        VISIT_SEQ(c, stmt, s->v.Try.body);
(gdb) 
3091        ADDOP(c, POP_BLOCK);
(gdb) 
3092        compiler_pop_fblock(c, TRY_EXCEPT, body);
(gdb) 
3093        ADDOP_JREL(c, JUMP_FORWARD, orelse);
(gdb) 
3094        n = asdl_seq_LEN(s->v.Try.handlers);
(gdb) 
3095        compiler_use_next_block(c, except);
(gdb) 
3097        if (!compiler_push_fblock(c, EXCEPTION_HANDLER, NULL, NULL, NULL))
(gdb)
3100            excepthandler_ty handler = (excepthandler_ty)asdl_seq_GET(
(gdb)
3102            if (!handler->v.ExceptHandler.type && i < n-1)
(gdb) 
3104            SET_LOC(c, handler);
(gdb) 
3105            except = compiler_new_block(c);
(gdb) 
3106            if (except == NULL)
(gdb) 
3108            if (handler->v.ExceptHandler.type) {
(gdb) 
3113            ADDOP(c, POP_TOP);
(gdb) 
3114            if (handler->v.ExceptHandler.name) {
(gdb) 
3167                cleanup_body = compiler_new_block(c);
(gdb) 
3168                if (!cleanup_body)
(gdb) 
3171                ADDOP(c, POP_TOP);
(gdb) 
3172                ADDOP(c, POP_TOP);
(gdb) 
3173                compiler_use_next_block(c, cleanup_body);
(gdb) 
3174                if (!compiler_push_fblock(c, HANDLER_CLEANUP, cleanup_body, NULL, NULL))
(gdb) 
3176                VISIT_SEQ(c, stmt, handler->v.ExceptHandler.body);
(gdb) 
3177                compiler_pop_fblock(c, HANDLER_CLEANUP, cleanup_body);
(gdb) 
3178                ADDOP(c, POP_EXCEPT);
(gdb) 
3179                ADDOP_JREL(c, JUMP_FORWARD, end);
(gdb) 
3181            compiler_use_next_block(c, except);
(gdb) 
3099        for (i = 0; i < n; i++) {
(gdb) 
3183        compiler_pop_fblock(c, EXCEPTION_HANDLER, NULL);
(gdb) 
3184        ADDOP(c, RERAISE);
(gdb) 
3185        compiler_use_next_block(c, orelse);
(gdb) 
3186        VISIT_SEQ(c, stmt, s->v.Try.orelse);
(gdb) 
3187        compiler_use_next_block(c, end);
(gdb) 
3188        return 1

Here's that cleaned up a little (I removed the unexecuted branches, except the error handling ones, which I just copied), and with some of the variables adapted to use the AST types we defined):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
static int
compiler_exception_mishandling(struct compiler *c, expr_ty e) {
    basicblock *body, *except, *orelse, *cleanup_body, *end;
    assert(e->kind == Y_ExcMishandle_kind);
    body = compiler_new_block(c);
    except = compiler_new_block(c);
    orelse = compiler_new_block(c);
    end = compiler_new_block(c);
    if (body == NULL || except == NULL || orelse == NULL || end == NULL)
        return 0;
    ADDOP_JREL(c, SETUP_FINALLY, except);
    compiler_use_next_block(c, body);
    if (!compiler_push_fblock(c, TRY_EXCEPT, body, NULL, NULL))
        return 0;
    VISIT(c, expr, e->v.Y_ExcMishandle.try_body);
    ADDOP(c, POP_BLOCK);
    compiler_pop_fblock(c, TRY_EXCEPT, body);
    ADDOP_JREL(c, JUMP_FORWARD, orelse);
    compiler_use_next_block(c, except);
    if (!compiler_push_fblock(c, EXCEPTION_HANDLER, NULL, NULL, NULL))
        return 0;
    except = compiler_new_block(c);
    if (except == NULL)
        return 0;
    ADDOP(c, POP_TOP);
    cleanup_body = compiler_new_block(c);
    if (!cleanup_body)
        return 0;
    ADDOP(c, POP_TOP);
    ADDOP(c, POP_TOP);
    compiler_use_next_block(c, cleanup_body);
    if (!compiler_push_fblock(c, HANDLER_CLEANUP, cleanup_body, NULL, NULL))
        return 0;
    VISIT(c, expr, e->v.Y_ExcMishandle.except_body);
    compiler_pop_fblock(c, HANDLER_CLEANUP, cleanup_body);
    ADDOP(c, POP_EXCEPT);
    ADDOP_JREL(c, JUMP_FORWARD, end);
    compiler_use_next_block(c, except);
    compiler_pop_fblock(c, EXCEPTION_HANDLER, NULL);
    ADDOP(c, RERAISE);
    compiler_use_next_block(c, orelse);
    compiler_use_next_block(c, end);
    return 1;
}

I could've also got this by copying the code in compiler_try_except and manually simplifying bits of it.

Finally, let's add a handler for expressions of this kind to the compiler function for arbitrary expressions, in the switch-case pointed out to us by gcc earlier:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -4996,6 +5037,8 @@ compiler_visit_expr1(struct compiler *c, expr_ty e)
         ADDOP(c, DUP_TOP);
         VISIT(c, expr, e->v.NamedExpr.target);
         break;
+    case Y_ExcMishandle_kind:
+        return compiler_exception_mishandling(c, expr);
     case BoolOp_kind:
         return compiler_boolop(c, e);
     case BinOp_kind:

Trying it out

Let's try out this handywork!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
$ ./configure --with-pydebug
$ make
$ ./python
>>> import dis
>>> dis.dis("1 ? 2")
  1           0 SETUP_FINALLY            6 (to 8)
              2 LOAD_CONST               0 (1)
              4 POP_BLOCK
              6 RETURN_VALUE
        >>    8 POP_TOP
             10 POP_TOP
             12 POP_TOP
             14 LOAD_CONST               1 (2)
             16 POP_EXCEPT
             18 RETURN_VALUE
             20 RERAISE
             22 RETURN_VALUE

Looks not bad! And the compiler didn't even crash! However, there are a few things wrong compared to the bytecode template we came up with earlier: - why are there extra RETURN_VALUE instructions? - why is there a RERAISE instruction?

After scouring the CPython codebase for instances of RETURN_VALUE, and a bit of Google-fu, I found that they came from the peephole optimiser, a system which makes simple optimisations to the bytecode after its main compile step. One of the things it does is replace unconditional JUMP instructions that point to RETURN_VALUE instructions, with RETURN_VALUE instructions directly. We can see the code without them if we compile the expression in a single-expression context (as is used in the Python REPL) which prints the expression instead of returning it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
>>> dis.dis(compile("1 ? 2", "", "single"))
  1           0 SETUP_FINALLY            6 (to 8)
              2 LOAD_CONST               0 (1)
              4 POP_BLOCK
              6 JUMP_FORWARD            14 (to 22)
        >>    8 POP_TOP
             10 POP_TOP
             12 POP_TOP
             14 LOAD_CONST               1 (2)
             16 POP_EXCEPT
             18 JUMP_FORWARD             2 (to 22)
             20 RERAISE
        >>   22 PRINT_EXPR
             24 LOAD_CONST               2 (None)
             26 RETURN_VALUE

Now as for the RERAISE instruction, it looks like it will always be skipped, so it must be a weird side-effect of copying the code for a try-except statement. I'll come to removing that later.

Let's see what happens if we actually try to use this 1 ? 2 expression:

1
2
>>> 1 ? 2
1

Wow! Just like that, it worked! Let's try the case in which an error occurs, by dividing by zero:

1
2
3
>>> 1/0 ? 2
python: Objects/object.c:411: PyObject_Repr: Assertion `!_PyErr_Occurred(tstate)' failed.
Abort (core dumped)

That's irritating - and worse, all it tells us is that a PyErr_Occurred, which is vague and tells us nothing about the root cause.

I spent a long time floundering here: adding all sorts of debugging instrumentation to my copy of the interpreter to work out what exactly was going wrong, but I got nowhere.

The final insight I needed to solve the problem came from comparing the disassemblies of these two functions:

1
2
3
4
5
6
def f():
    try: return a
    except: return b

def g():
    return a ? b

These should be identical in behaviour, but let's look at their bytecode:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
>>> dis.dis(f)
  2           0 SETUP_FINALLY            6 (to 8)
              2 LOAD_GLOBAL              0 (a)
              4 POP_BLOCK
              6 RETURN_VALUE

  3     >>    8 POP_TOP
             10 POP_TOP
             12 POP_TOP
             14 LOAD_GLOBAL              1 (b)
             16 ROT_FOUR
             18 POP_EXCEPT
             20 RETURN_VALUE
             22 RERAISE
             24 LOAD_CONST               0 (None)
             26 RETURN_VALUE
>>> dis.dis(g)
  2           0 SETUP_FINALLY            6 (to 8)
              2 LOAD_GLOBAL              0 (a)
              4 POP_BLOCK
              6 RETURN_VALUE
        >>    8 POP_TOP
             10 POP_TOP
             12 POP_TOP
             14 LOAD_GLOBAL              1 (b)
             16 POP_EXCEPT
             18 RETURN_VALUE
             20 RERAISE
             22 RETURN_VALUE

Besides the extra LOAD_CONST in the first, the substantial change is the addition of a ROT_FOUR in the working one. It turns out this is necessary.

The fix, then, was simple: insert a ROT_FOUR instruction after the handler expression is visited:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -3214,6 +3214,7 @@ compiler_exception_mishandling(struct compiler *c, expr_ty e) {
     if (!compiler_push_fblock(c, HANDLER_CLEANUP, cleanup_body, NULL, NULL))
         return 0;
     VISIT(c, expr, e->v.Y_ExcMishandle.except_body);
+    ADDOP(c, ROT_FOUR);
     compiler_pop_fblock(c, HANDLER_CLEANUP, cleanup_body);
     ADDOP(c, POP_EXCEPT);
     ADDOP_JREL(c, JUMP_FORWARD, end);

Next steps

Where do I go from here?

Documentation and tests

This feature has no unit tests and is undocumented. Were this an official addition to Python (God help us), it would need those.

Printing error messages

For debugging purposes, it's probably useful to print the error tracebacks even though they're ignored.

Ignoring SystemExit and KeyboardInterrupt

It might be worth letting these two non-errors pass through silently, because they (particularly KeyboardInterrupt) might cause some confusing and unreliable behaviour.

Conclusion

I hope this blog post has shown you that large codebases can be approachable, with the right tactics. "Fake it till you make it" applies, and you can achieve a lot just by guessing and copying-and-pasting existing code. Along the way, you're sure to learn something useful, so you might not need to guess as much next time!

If you want to play with this modified version of Python, you can do so here!

Update: I've been keeping my fork somewhat up-to-date with more recent versions of Python, and added another couple of features to it which I've not blogged about. Check the GitHub repo for more information.

First published: 2021-09-19
Last updated: 2023-01-03