What happens when a request arrives?

Pixies and/or gnomes not acceptable explanation

Need to understand how requests are queued, multiplexed, managed

Lets take a look at a few

• Apache
• Tomcat
• Nginx
• Passenger
• Mongrel
• Node.js
• Mochiweb

Lots more, but I’ve worked with these, and won’t sound as clueless talking about them.

(Yes, I know some of those technically aren’t web servers, but humour me…)

Apache

Like a city train station with multiple platforms

Each request is a train

Can handle as many trains at once as you have platforms

Conductor controls which trains go onto which platforms

What happens if you have too many trains at once?

If all platforms are full, trains have to queue up and wait

Service slows down

If trains take too long to finish on platform, services get cancelled

Sometimes on platform

Sometimes the trains waiting outside

BUT - A few slow trains on some platforms don’t slow other platforms down though

But, if the queue of trains gets too long

You may as well just give up and go home. You’re not getting in

Apache

• Master OS level process (conductor)
• Spawns multiple OS level worker processes (platforms)
• Worker processes handles requests serially

Apache

• Workers isolated, config can resource throttle if desired
• More secure. Workers not started as privileged user. No shared memory
• Workers can crash without affecting other requests
• OS level processes heavy on CPU/memory
• Finite workers. FAIL if all are blocking on resource that won’t timeout

Web servers are boring unless they’re doing something

Apache doesn’t do much on it’s own

But has lots of open source community activity

Many modules to plug in to do almost anything

• Caching
• Custom headers
• Authentication
• Static file serving
• Whatever

Being on the beaten track has it’s advantages

Apache / Passenger / Rails

Passenger is a module that lets you run Rails apps on Apache

Passenger

• Loads Rails/app code once. Forks workers to save memory
• Apache still does heavy lifting, Passenger does rails
• Each Passenger worker process single threaded
• Makes app deployment easy, as all workers are managed by Apache
• Passenger has own worker/master pool. Not quite as battle hardened as Apache

Tomcat

Conceptually the same as Apache

Separate platforms, trains come in

Can still hang with too many long requests

Tomcat

But under the hood it’s all happening in the same place. Not separated at all

Ok most of the time

Tomcat

Can cause problems if you’re not careful to keep things separated in memory

Tomcat

• Basically same as Apache process model
• Uses thread pool inside JVM
• Acceptor thread delegates to worker threads
• This is lighter on resource usage
• Threads are cheaper than OS processes

Shared state concurrency is easy to screw up though

Tomcat

THIS WILL HURT YOU

private static final DateFormat dateFormatter = new SimpleDateFormat();

SimpleDateFormat isn’t threadsafe. Instance shared between all threads accessing code

Just a matter of time till…

Shared state concurrency bugs will bite you in the ass

Well, maybe not as dramaticly. You wish

Will probably just manifest as cryptic to debug NullPointerException, or as corrupted data you don’t notice till months later…

Tomcat / JRuby / Rails

• Pool of JRuby instance objects sitting inside JVM
• Needs DB connection pool
• Shared code/data across threads (but not with multiple instances)

Tomcat / JRuby / Rails

• Can single thread - sort of like Apache/Passenger. Set JRuby instance count to max concurrent requests
• OR you can configure it with 1 instance, and multithread Rails
• Newbie error : default is 1 JRuby instance, Rails multithread off - single threaded execution == Slow

Another concurrency model: Supermarket with lots of checkouts

Simple conceptually

Each checkout has own queue

Works well (mostly…)

Mongrel /Rails

• Start up multiple mongrel processes manually (supermarket checkouts)
• Best practice for Rails apps for a while, but Passenger more used now
• Each has full Rails stack loaded (more memory usage)
• Each single threaded with own queue
• Needs front end round robining requests to mongrels
• Long running requests mean wait for all queued on that mongrel

Supermarket checkout problem

Some guy paying for groceries with small change slowly

Everyone else at that checkout has to wait

If you choose the wrong queue, you’re stuffed

Mongrel / Rails

Suffers from “guy paying with small change”

Means more monitoring/restarting required when slow requests block it up

Restart kills all requests in queue for mongrel. Users see errors as a result

Forget about trains and pedestrians for a minute

Think about a waiter at a restaurant

• Single waiter can serve multiple customers
• Serve orders to multiple chefs in kitchen
• Time-slices between tasks, just don’t spend too long any where at a time
• Don’t need whole order at once: drinks, entree, main all come out separately

Nginx

• Light, single threaded, event driven (waiter)
• Can do huge request concurrency
• Does simple stuff super fast (static file serving etc)
• Delegates complicated activity to background workers (Passenger / Mongrel etc)
• Great for long polling / comet / websockets

Nginx hides single thread event driven model in black box

Don’t have to care about it day to day. Just use it and it works

Node.js

Node puts it in your face

Node.js

• Not web server, but stdlib includes nice, easy to use http server
• Whole stack is async. Your code has to be too
• Perfect for long polling / comet / websockets. Great performance / concurrency possible
• Dumb blocking code can hang whole server
• Async model with lots of Javascript callbacks is mindf**k

In theory, single waiter could serve millions of customers

Would be slow, but requires no more resource usage

Can’t do this with Apache/Tomcat style multi-process. Limit to how many “train platforms” you can build

But what if you COULD build millions of train platforms?

And do it cheaply and quickly?

And pack them into your available space really efficiently?

Mochiweb

Lets you build millions of “train platforms”

Actually, you build a new platform for a single train

Then tear it down

All really cheaply

Mochiweb

• Built in Erlang
• Simple, low level
• Super concurrency high performance
• Amazing for long polling / comet / websockets
• Actually event driven under the hood
• But can code like process model
• Erlang still not on “beaten track” though

Yaws

Another Erlang web server, similar to Mochiweb

Apache dies at ~4,000 concurrent reqs. Yaws still ticking up around ~80,000.

Total throughput does not decrease as load increases.

Work done to get 1,000,000 concurrent requests on a single box with Mochiweb

Done!

Questions?

Julian Doherty

@madlep