Today marks the release of the second generation of PastPages’ code base, nicknamed “bradlee.” The screenshotting system has been rewritten to make it faster and cheaper by shedding dependencies and introducing a task queue. Here’s a quick rundown:
- Firefox -> Webkit
- Selenium -> PhantomJS
- Xvfb headless server -> Nothing!
- One-by-one screenshot script -> Concurrent Celery queue
- Memcached -> Varnish
The result is that a significantly less powerful server now completes a screenshotting run in half the time the old server did before. That saves money in addition to time.