Ah, multiplayer. First you make the actual game then you add multiplayer support ontop of it, right? After all, multiplayer just means that the game needs to run on different devices simultaneously and somehow show the same game.
We all know that things are not as simple as that. I’d daresay that you either plan for multiplayer from the get-go or you’re going to be in a hell of trouble later on. It’s not like you can just get your normal game going and then add it later on without having planned for It in the first step.
Also, a general rule of thumb is that a multiplayer game takes three times as much effort to realize than a corresponding single player experience.
I knew pretty much that multiplayer would be part of my game since RTS is pretty much all about multiplayer competition. Obviously, when you first start your game, it will initially be a single player experience. But, whenever I was working on a part of the (early) game, I was always keeping multiplayer aspects in the back of my head for all design or implementation decisions.
Multiplayer is not Multiplayer
One of the key messages is that there’s no such thing as the multiplayer support. If you have coded multiplayer for a turn-based game, you don’t really have a whole lot to build upon when you next start a FPS shooter.
Similarily, if you’re doing an FPS shooter (or any other action-based game), you’ll have to use a totally different model and solve other problem than I had for my RTS game.
That having said, all my experiences are obviously from an RTS multiplayer game. By no means I’d be able to carry over the code to an action-packed game without substantial modifications.
Challenges of RTS Multiplayer
Still, when I started the multiplayer part of my game, it was really a challenge. First of all, I got to say that I had a pretty clear picture in my mind how I wanted to implement the multiplayer aspect of my game. A lock-step simulation dynamically adapting to changing network conditions (latency adjustment) and frame rates. I got a pretty solid understanding about network programming and having a professional history in VoIP, RTP and QoS, terms like lateny, jitter and packet loss, network protocols like UDP or implementing custom protocols ontop of it was nothing new for me.
I’ve even written a pretty detailed design document describing the lockstep model I wanted to implement, the protocol exchanged between the clients and the used algorithms to adapt for changing network conditions and such.
Still, I was running into real challenges. Basically, when you want to deal with real-time multiplayer, you need to deal with:
- Synchronization: commands sent from one client must be synchronized to all others and replayed “at the same time”.
- Latency: sending (and receiving) commands takes time but this delay should not be noticeable if you can help it. You need to adapt to changing latency.
- Jitter: the time it takes for transmitting packets is not constant, it may change rapidly from 100ms to 500ms and later go back to normal, and you need to deal with that.
- Packet loss: guess what, packets get lost in the Internet and you need to make sure they are resent or you’re missing out on crucial information. Ontop of it, packet loss causes additional latency (you first need to detect that the packet wasn’t received, then resend it and all this takes time).
- Determinism: simulation on all clients must be deterministic or the games will quickly start diverging between clients.
- Frame rates: different devices can sustain different frame rates but slower devices will drag faster ones down if you don’t take it into account.
- Matchmaking: how to make players connect to each other?
Is Life Deterministic?
While my initial implementation was able to handle most of the challenges (thanks to the detailed design doc), I was particularly running into quite a few problems with making the game deterministic. Even though I was planning for multiplayer support and therefore opted for a fixed simulation timestep, no out-of-sync random number generation and generally deterministic behaviour, a few subtle sources of determinism were causing me a real headache.
I remember that a particular (efficient) implementation of line-of-sight calculation had a very subtle side effect that resulted in non-deterministic behaviour: a fighter would pick a different target on one client than on the other one for a few frames and eventually go totally out of sync between different devices, causing side effects to other fighters and within a second or two, two devices would show a totally different scenario.
The fact that it is hard to debug multiplayer games means that you have to trace down these sources using endless trace files trying to figure out why the heck the game desyncs. I was really blaming floating point arithmetic or generally starting to think that computers may have become non-deterministic lately (Turing, where are you?).
Multiplayer programming can also provide some more philosophical insights into time being relative, for example. If on one device your frame rate drops, and as a result, you cannot keep up your simulation, then you need to avoid the spiral of death (simulation takes even longer and your frame rate drops some more which makes the simulation take even longer). You do so by cutting off (real-)time and not use your whole real-time for simulation, so simulation time and real time diverge (the game will basically slightly slow down instead of getting totally frozen). In addition, adjusting the simulation rate will remedy some of these problems (but end up in more jerky but at least steady movement). On another client, things may be fine (e.g. better hardware) but it must do the same simulation or it will be ahead in time.
The other device may, however, experience network issues causing latency to increase. This must be taken into account to avoid jerky movements or the game getting stuck for noticeable periods of time. However, other devices must be aware of this even though they may not experience the same problem. When the network conditions get better, we may improve response times but only gradually so things stay predictable for the player.
Very funny situations you run into! Obviously, things get a lot easier if you’re doing a turn-based multiplayer game so many challenges do not apply there.
Anyway, over a couple of weeks of hard work, I must have ironed out these problems but the multiplayer code in my game still gives me the creeps. Every now and then, I got to do some regression testing on the multiplayer part and the simple fact that I might discover a new nigh-impossible-to-track-down bug really scares me.
Testing multiplayer games is a lot more difficult than testing single player games. First of all, you need quite a bit more test devices or how are you going to test a 2v2? Ontop of it, you can’t play on 4 devices simultaneously, heck not even on two for a challenging game.
Even more difficult is testing for different network conditions. Overall, you can’t always get a real game going with real bad network conditions. After all, who has a friend in all corners of the world at his fingertips he can test with anytime?
That means, testing means simulating bad network conditions and we all know simulation is often only half the truth. I’ve simulated quite a few varying network conditions, like high latency (1 second of round trip latency means the game will have really noticeable lag), high random packet loss, high consecutive packet loss (yuck, game gets unplayable but not a realistic scenario), or network conditions that vary a lot (alternating between low and high latency).
I’ve also connected devices with one (artificially) slowed down to low frame rates dragging down the other to check out whether frame rate adaptation can compensate for changing frame rates.
Overall, the testing aspect means a lot of extra effort and I still can’t be sure how the game will react in all situations it will be exposed to when the player base really grows and people across the world need to connect to each other.
Security always takes time both, on the programming side and the execution side. However, multiplayer means there’s always the risk of players trying to score an unfair win by cheating. Usually, this needs some technical background but without any measures, it is totally possible to cheat if you are creative.
I don’t plan to totally encrypt everything because encryption and decryption is too much of a performance hit. However, I’ve used digital signatures to make sure that the content is not tampered with, neither while it was sent over the network nor on the device itself (e.g. configuration files holding balance changes).
On iOS, multiplayer usually means Game Center support and I was counting on Game Center supporting me in getting crucial things done, most importantly matchmaking and NAT punch-through.
In fact, Game Center is fantastic, I wouldn’t want to do my game without it but it still has quite a few short-comings that were hard to deal with.
First of all, matchmaking in Game Center offers you little control on how to match teams up. I really would have loved to see an arranged team setting where I really have control who’s gonna play against who. Teams do not show up in Game Center. Apparently, Apple thought all games are going to be cooperative.
Also, it is extremely hard to implement a League concept ontop of Game Center where you can be matched based on your skill.
Scores in Game Center are another issue. In contrast to OpenFeint, they can only go up which means you cannot implement an experience system where you’d lose some of your points after losing a game.
Overall, Game Center still provides a lot of functionality that I wouldn’t have wanted to implement on my own but a few things could be improved.
It can be said that multiplayer adds a lot to your game. You can play with/against your friends, the replay value dramatically improves, you can socialize with other players, invite friends and it opens up new game modes.
However, it can also be said that you need to add a lot to your game as well to make multiplayer happen. So the bottom line is: even if you have a pretty good clue about network programming, I still came to the con-clusion that real-time multiplayer is not for the faint-hearted.