Advanced Celery

Author

  • Ask Solem
  • Work at VMware, on RabbitMQ team

Overview

  • Flexible and Reliable message queue system
  • Granularity: the less computation, the more fine grained the task is * Can reuse connections, etc
  • Chunking * Grouping fine-grained tasks to reuse resources

Chords

  • Sync primitive
  • Known as a barier
  • Callback the body with the results of the headers
  • Native support in Redis, with good enough fallbacks for others
  • demo of parallel summariazation using chords
  • Can use this to implement MapReduce

Blocking

  • Is bad
  • Timeouts

Routing

  • Smart routing
  • CPU based routers would be nice

Cyme