Eric Ries' list of the benefits of small batches:

• faster feedback
• problems are instantly localized
• reduced risk
• reduced overhead

Eric's post

I agree with these and have been experiencing the same thing with the project I'm currently working on. If you have a persistent atomic queue server, it's easy break work into small pieces which can be transparently distributed.

In this model, the chain of events normally held in a stack on a single worker is replaced by a chain of queues from which any number of workers can process items and place them in the next queue.

For my project, each queue has four subqueues - "waiting", "active", "done", and "error". Workers atomically pop an item from waiting to active and a qtime and qexpireTime are set on the item. On completion, the item is popped from active to done. On failure, the item is popped to the error subqueue. If the worker crashes, the active item will eventually expire and get pop back to the waiting queue. I'll post later on how we're using this with Amazon EC2.