Code Generation in Python: Dismantling Jinja

Author

  • Armin Ronacher

Why?

  • Isn’t it evil?
  • A security problem?
  • Bad for performance?
  • Not if you do it right.

Security

  • Code Injection

  • Pollute namespace
    • Change local variables
    • Can evaluate code in different namespace

Performance

  • Alternative: Write an interpreter
  • Too slow
  • Not suitable

Eval 101

  • Compile function to make code objects
  • evan can work on a namespace
  • Using ast module, can alter underlying structure
  • Can use ast to add in line numbers to nodes
  • Don’t pass strings to eval/exec, but use code objects
  • Explicit compliation and namespaces, to fix problems

Jinja

  • Jinja and Django have C inspired scoping rules

  • Pipeline
    • Lexer
    • Parser
    • Identifier analyzer
    • Code generator
    • Python source
    • Bytecode
    • Runtime
  • Only runtime is necessary

Scoping

  • Context objects are dict-alike
  • Slow
  • Resolve in context ahead of time

Code Generation

  • Low level
  • Target byte-code
  • High level
  • AST generation
  • Bytecode doesn’t work on appengine, and is implementation specific
  • Would be nice to map jinja to bytecode
  • Ast is limited, easier to debug, and doesn’t segfault

Tale of Two Pieces of Code

  • scope in a function is faster than global scope
  • lookup via index instead of name
  • local dictionary isn’t generally used
  • semantics can be mapped to fast execution environment
  • Jinja context is data source
  • Django context is data store
  • You cannot modify context in Jinja

jsonjinja

Q&A

  • If you had the chance to redo would you use ast? * Yes, there are utility libraries that help this
  • ctypes for line numbers? * put special line number variables, monkey patch traceback * works in everything tested, including pypy * Some problems on some architectures.