Some Pointers

ceval.[ch] (VM engine)
compile.[ch] (Bytecode compiler)
frameobject.[ch] (execution frames)
opcode.h (bytecodes)
code.h (PyCodeObject)
pystate.[ch] (interpreter state)
pythonrun.[ch] (entry point)

Particularly useful pieces of documentation:

Notes on the implementation

Unless otherwise noted, the source file in question is Python/ceval.c.

Control Flow

The calling sequence is: main() (in python.c) -> Py_Main() (main.c) -> PyRun_FooFlags() (pythonrun.c) -> run_bar() (pythonrun.c) -> PyEval_EvalCode() (ceval.c) -> PyEval_EvalCodeEx() (ceval.c) -> PyEval_EvalFrameEx() (ceval.c).

PyRun_FooFlags() also calls PyParser_ASTFromQuux() to obtain an AST which run_bar() then passes to PyAST_Compile() (in compile.c) to get a PyCodeObject for PyEval_EvalCode().

EvalCodeEx() does some initialization (creating a new execution frame, argument processing, and some generator-specific stuff) before calling EvalFrameEx() which contains the main interpreter loop.

Threads

PyEval_InitThreads() initializes the GIL (interpreter_lock) and sets main_thread to the (threading package dependent) ID of the current thread. Thread state switching is done using PyThreadState_Swap(), which sets _PyThreadState_Current (both defined in pystate.c) and PyThreadState_GET() (an alias for _PyThreadState_Current) (pystate.h).

The actual thread switching occurs by releasing the GIL (Python doesn't dispatch threads at all; it just releases the GIL, giving the operating system permission to wake up a different thread - which the operating system may or may not chose to do. After some time, the original thread will try to reacquire the GIL. Assuming the OS applies fairness, it will not get it back if a different thread was also waiting for it, so our thread will block - and then the OS will dispatch (at latest)). See Periodic Tasks below.

Async Callbacks

Asynchronous callbacks can be registered by adding the function to be called to pendingcalls[] (see Py_AddPendingCall()). The state of this queue is communicated to the main loop via things_to_do.

State

The global state is recorded in a (per-process) PyInterpreterState struct and a per-thread PyThreadState struct. In principle, multiple interpreter states are supported per process (and the current interpreter is identified by thread). However, there are many limitations and quirks in the multiple-interpreter code.

Each execution frame's state is contained in that frame's PyFrameObject (which includes the instruction stream, the environment (globals, locals, builtins, etc.), the value stack and so forth). EvalFrameEx()'s local variables are initialized from this frame object. A lot of stuff also lives on the regular C stack, which exists in parallel to the frame object stack.

Instruction Stream

The instruction stream looks as follows (c.f. assemble_emit() in compile.c and dis.py for the inverse operation): A byte stream where each instruction consists of either

a single byte opcode: OP
a single byte opcode plus a two-byte immediate argument: OP LO HI
a special opcode followed by the first two bytes of the argument, followed by the real opcode and the remaining two bytes of the argument: EXTENDED_ARG ARG ARG OP ARG ARG

Opcode Prediction

One neat trick used to speed up opcode dispatch is the following: Using the macros PREDICT() and PREDICTED() it is sometimes possible to jump directly to the code implementing the next instruction rather than having to go through the whole loop preamble, e.g.

case FOO:
      // ...
      PREDICT(BAR);
      continue;

PREDICTED(BAR);
case BAR:
     // ...

expands to

case FOO:
      // ...
      if (*next_instr == BAR) goto PRED_BAR;
      continue;

PRED_BAR: next_instr++;
case BAR:
      // ...

Main Loop

Variables and macros used in EvalFrameEx()

The value stack:

  PyObject **stack_pointer;

The instruction stream:

  unsigned char *next_instr;

NEXTOP(), NEXTARG(), PEEKARG(), JUMPTO(), and JUMPBY() simply fiddle