New Sun Labs Technical Report:

by Tim Blackman, Jim Waldo

October 8, 2009 - Project Darkstar is an investigation into building a server infrastructure that will allow transparent scaling over multiple cores and multiple machines for on-line games and virtual worlds. These applications provide a particular challenge, since they require very low latency for the individual clients even when the server infrastructure is handling hundreds of thousands or millions of distinct players.

A central component of the Project Darkstar infrastructure is a datastore that is shared among all of the server nodes. To minimize latency, each server node keeps a cache populated by requests on the centralized store. The challenge is to find a caching scheme that minimizes latency.

The answer involves an unusual trade off, and one that is especially suited to games. If users are willing to let the game forget a small amount of their recent game play when something goes terribly wrong -- a server crash -- the benefit can be substantially faster performance and also the ability to handle more players. This is a trade off that users probably wouldn't be willing to make for a real world application, but seems worth the risk for a game. In the transactional framework provided by Project Darkstar, we therefore trade off durability to achieve low-latency, while maintaining consistency at the global level.

This scalable storage scheme is one of the pieces that will let Project Darkstar help game developers take advantage of modern multi-core architectures and high throughput networks.