Fun and Games and Business

Solving Hard Problems on the Way to a Million-Player Game

By Al Riske

08.June.07- Jeff Kesselman wants to be clear about one thing: Games are a business. Sure, they're fun and, sure, you may find yourself looking up at 4 a.m. only to realize you've been playing all night, but that doesn't change the fundamental fact.

This is business. Serious business.

"Games are the second biggest media business today," Kesselman points out.

Bigger than the music business. Second only to the movies, in all their various forms and formats.

"I came from the business, and if you come from the business you know it's all about the money," Kesselman says. "Yes, there's the flash, but it's just like the movie industry. From the outside it looks like it's about glamour. From the inside it's all about the bottom line."

He makes the point because his baby, Darkstar, is not what you'd expect from a business-to-business business like Sun.

Not at first glance anyway.

Look closer.

It's exactly what you'd expect from Sun.

Darkstar may be a game server, a first for Sun, but as Kesselman, the chief architect, points out: "It is designed to solve all the hard things about doing massively scalable, reliable, fault-tolerant, persistent online game play."

Kesselman's involvement with games dates back to his high-school years, when pen-and-paper role-playing games such as Dungeons & Dragons were popular. In college, he wanted to learn how to create computer games, but no one was teaching that, so he designed his own major -- image generation -- by combining courses in computer science, film production, and art.

For the next 15 years, he wrote code for a number of startups and contributed to a wide range of commercial games, including The Horde, Titan, and Blazing Dragons.

Then, in 2001, he came to Sun.

At first he was doing performance tuning on the Java Development Kit, but he soon hooked up with Chris Melissinos, who had just been appointed business development manager for video games and electronic entertainment.

Melissinos needed a server strategy, and Kesselman had one.

In fact, he had been working on a prototype, part time, for years.

"I went through about three versions and none of them was complete. Then I ended up at Sun, and at Sun I learned how enterprise systems work," Kesselman recalls. "While the way enterprise systems are built is not right for game developers, the concepts of how to get five-nines reliability, how to get transactional guarantees, and so on -- these things were very, very useful."

Just as important, Kesselman knows how game developers think -- what they know and what they don't want to know.

"The kind of programming they do, and have always done, is near real-time programming. Which means they sit in a loop, and within that loop they say: Read the controllers, figure out the next update, draw it, do it again. That's their world. It's one mono-threaded loop," he says.

What Darkstar brings them is the power of multithreading -- dividing programs into multiple threads, or tasks, that can be processed in parallel by today's multicore chips -- without the usual bugs.

"We provide the developer with a very simple model that makes it look like they're developing mono-threaded, event-driven code (because that's what they know), but we make it parallelize," Kesselman says.

He also knows what players care about. Like dupe bugs and rollbacks.

A dupe bug is a glitch that allows two players to possess a single object at the same time. A duplicate. It's a common problem when playing massive, multiplayer games online.

"From the point of view of a programmer with any kind of database background, that's a break in referential integrity. But game developers don't know about databases. They don't want to know. So we keep things very high level and invisible to them," Kesselman says.

In other words, they don't need to know about race-conditions or deadlock avoidance, With Darkstar, it just happens.

Rollbacks are the worst.

Say you and about 50 other players have spent the past three hours trying to take down a particularly fierce dragon, but what dies instead is the server you're playing on.

"When it comes back up, it comes up from ground state and that dragon is back where he was at the start," Kesselman says.

"What Darkstar does is, it has a layer of processing nodes that people connect to and the data lives in a [network] cloud above that. As players affect things, that data is fetched, modified, and put back. Now the server can die. No big deal. It has no state. The state's all in the cloud. Players simply reconnect to another server and keep going."

"The core to this, the thing that makes it all work, the thing that's the most labs-y about it, is called the object store. Which is that cloud. It is a very, very fast transactional database of objects -- distributed in memory. Now the persistence is effectively invisible. Developers don't have to think about persistence. The objects just are persistent," Kesselman says.

By solving that basic problem, Darkstar opens new possibilities. Developers and players can now start to think about how they can change their virtual worlds.

"In the old world, you might be able to build a castle in memory, but when that died, you'd lose your castle. Now that won't happen. You can change the environment and it stays that way. So games can potentially become more dynamic," Kesselman says.

"This is actually a complaint of a lot of the players -- that the games all feel very fixed and static. It's more like the game playing you than you playing the game in a lot of ways. The game says, 'Do this, do this, do this. Oh, good, you did that. Nice player.' What the players are really yearning for is a game that's more open-ended, where they can start developing their own goals and setting up their own trading house or whatever. So that's another thing we're trying to enable in this system."

Kesselman notes that Darkstar is becoming an open-source project, which has heightened already-strong interest in the project -- and should increase revenue opportunities for Sun.

"First of all, we have some very, very big customers, most of which I cannot name because they're on need-to-know, but I can mention NCsoft because we went public with them. NCsoft is one of the biggest massive multiplayer online role-play game publishers in the business," he says.

"So that's one kind of Darkstar user -- someone who sets up a Darkstar backend and uses it for commercial production. Well, they're going to have to build a machine room, which means there's machine room business there. They're going to need support for it. Which means more services business."

In fact, Darkstar stands to stimulate the entire game industry.

"Today it takes $30 million, base, to build a massive multiplayer online game. That's sort of a standard figure in the industry. That's compared to $10 million, which is the base for developing a single player game. The difference is they develop their server stack custom, from the ground up, and have to build a machine room for it. Because it's custom, they need their own group to run it. There's a huge expense in that. Part of what Darkstar does is, by standardizing this and allowing different games to share the same backend (like an app server in a hosting environment), it makes hosting possible," Kesselman says.

"We've got a bunch of small to mid-size developers who would love to get into this game but can't risk $30 million. So for them what we're looking at is a situation where Darkstar is hosted. The hosting provider runs their beta, and when they're ready to go live, the host flips a switch and takes part of the revenue for it. They can also run billing for them, provide these other services, and basically do all the back-office stuff."

In fact, Sun recently announced something called the Playground.

"It's our hosted prototype environment. We've been taking applications from game developers and frankly we have way too many, but we tell people if you don't get into this round there will be another round. It's very exciting. Among those that applied are some very big names that anybody who plays games would know."