Important link: We released an open source proof of concept metagame and lobby development library on GitHub
Most of the games use online services these days. Many of them are multiplayer or at least are being run like a service and use lots of online/social features. Previously I wrote an article about choosing networking middleware for games and it became very popular. Since I mainly work as contract developer/consultant, I always don’t get the time to share what I learn/do or am not allowed to do so. However after working on an online pvp game using one of the public game backend providers and using another for a small project of our own team and more importantly developing a backend server for games using Microsoft Orleans in the last 2 years, I feel myself in a good position to write about public and custom backend servers and their advantages and disadvantages, about if you should make the backends in your game engine (like what we had with uLobby at MuchDifferent) or write them in platforms like Erlang, Orleans, Akka and …
The products which are marketed as backend servers or services for games usually try to provide features or building blocks of creating the features which are a part of the metagame of your product. Things like leaderboards, achievements, challenges and quests. They also provide basic features like authentication, cloud data storage and real-time messaging between players which you usually need to build your game. The main game logic of your product on the server is usually coded either in your game engine or a specially written software for the exact purpose. Some products like Playfab simply allow you to run whatever you want (if it is an executable file) on their servers for your authoritative game server logic. Some of them try to create a scripting system which allows you to write authoritative logic which means a more light-weight server but usually it means you don’t have many game engine features available at your fingertips and also you have to choose their language of choice and debugging is not as easy as it should be either. Here we intend to mostly talk about the first set of features and only a bit at the end about the game logic part.
There are 3 factors which are very important when choosing your backend. First of all the public services are usually very cheap at the beginning and they will only charge you more when your game really takes off which is ideal for many companies and types of games. Custom solutions like ours or the open source equivalents are usually cheaper in the long run but at the beginning might have a cost associated with them, especially because you need to run them on your own servers and pay for the servers yourself. Even if it is one machine only at the beginning, still it is some cost. Also when using custom solutions made by yourself or others, you need to manage the servers yourself which again is more costs.
Now you might question , why you should want to write your own or buy a custom backend solution, the answer is flexibility. The public services have to provide their features in a way which means they can run their service in a cost effective manner and also the features they provide might differ a lot from what you need.
They themselves differ very widely with each other. GameSparks for example provide more complex querying capabilities and now it doesn’t use MongoDB as database anymore so will not have the scaling issues while Playfab uses a different data model (it always has been more scalable due to database choice and more interesting architecture IMHO). Let’s say you want to build a game which runs tournaments on a weekly basis and provide prizes. Both of the popular public platforms have features which assist you in doing it but as soon as you have a special requirement like small prizes between you and your friends then you have to do a lot to make the system work with your design.
Also as said some of their features are simply less powerful because implementing them in a more powerful/correct manner would mean their system will be open to bad usage from developers. For example, unidirectional friend lists are there because transactions are expensive and if they allow you to run them then you might mess up with their service which is very important when they are providing a lot for free. Ability to run free form queries is tricky for the same reason and lots of special kinds of timers is discouraged because of this as well. Now when you make your own custom solution, you still need to be careful with transacitons, queries, your data sizes and many other things but you can choose where to spend your budget specificly and where you want to spend your resources or money if needed.
How to write your own custom solution
If you decide to create your own custom solution or buy/use an open source or commersially available backend other than the public service providers then you have a few broad choices to make which affects you a lot. One choice is what technology to use and the other is how to scale, monitor and maintain the system.
One approach which was popular back then for some medium size games was building the bakcned part in your game engine and run multiple instances of it to scale it and use a powerful database behind it. When I was working at MuchDifferent, our products were made for such a system. We had uLobby which could be used to create lobby and metagame features of a game inside Unity and we had uGameDB which was a client for the powerful NoSQL database Riak. We made medium size games using this approach successfully and many our customers did the same in relatively big games as well. Not that the approach does not have limitations.
The advantages of the approach are the fact that you are inside your game engine and have lots of technical know-how and the communication between your game servers and metagame servers is very easy using the same library and RPC mechanisms. Back then before existence of gRPC and similar technologies it was a challenge for a relatively small team to develop their own RPC mechanism which works between different platforms and is flexible and high performance enough for multiplayer online games. Also this approach probably is more interesting if you are making a multiplayer authoritative server based game and are already using your game engine on the server.
The disadvantages are the fact that you don’t have any of the usual features which you need to implement for bigger games. Things like fault tolerance, gossip protocols (for clustering and finding other nodes) and distributed computing mechanisms usually don’t exist in game engines and you have to develop them yourself. If you need to run two instances of your lobby and friends of a player on lobby A are in lobby B and they try to send messages to each other, you are responsible for understanding the fact and relaying the messages.
We have developed a proof of concept system for a friend team and cleaned it up a bit and released it in github as an open source project. This is called MidNight and is available on my personal github page here. It is called midnight because it tries to handle the chaos of the players and many systems at work in a lobby. It contains basic authentication features, friend lists and game server instance management. There is a sample and minimal documentation but we are available for help if needed, just create an issue and we’ll jump in.
The other approach is to use frameworks which are created for the purpose of making distributed scalable server software. These days most cloud providers have PaaS offerings like managed Kubernetes setups on all major providers and Service Fabric on Azure. Even if you don’t want to use these for good reason or not, frameworks like Akka, Microsoft Orleans or the Erlang programming language makes it a lot easier to work on these kinds of software. I previously wrote about how we use them.
Using PaaS offerings might lock you in and also using Kubernetes is not necessarily beneficial if you don’t have a big team and are not developing things in too many languages. Micro service architectures have their own challenges when it comes to handling long running processes, transactions or cross domain business logic and they are much harder to debug. I personally see an actor as a sort of micro service (the concept is from a much smarter person) and am very happy with running a cluster of servers using an actor framework without any micro-service in the common sense whatsoever. You still develop each small part in a different actor while you avoid many of the complexities of running a micro service based system.
Our backend uses Orleans and has 3 processes other than Couchbase which is our on-premise database choice. One process is the Orleans silo which hosts the actors and can form a cluster and is stateful. Another is a front-end for games to connect to which is a stateless socket server which can scale independently of the Orleans silo and the third is a rest server front-end which can be hosted on IIS or using OWIN to serve requests of our web-based panel which is an angular based app.
To make management a bit easier we developed everything as console applications which make them easy to debug but used Topshelf to make them windows services which automatically start after crashes and also automatically start if the OS restarts after an update or some other event. Orleans took care of connecting failed and consequently restarted processes to each other for continuing execution. Now with Orleans availability on .NET core we can even runthe system on Linux and benefit rom Orleans transactions and also all perf improvements in .NET core and that is another reason to choose te tech you use carefully so you can rely on it to improve over time.
When you do this and write your own backend or use one with source code available, you can choose and you can choose a lot. You can choose to have quests for each user based on his/her friend’s units, reward them based on number of friends which completed something in a day and run custom match making logic which looks at weekly leaderboards so players close to each other are matched which makes their games more intense.
These are just some examples and some of them might be easier to do in one or another of the public providers. Playfab allows you to run custom match makers on your own servers for example which we actually did but this is something between writing your own stuff and using a public provider because it is essentially a mix of both and that is a good power to have. GameSparks also allows you to run js code in many many places and based on timers and it results in lots of flexibility on many fronts but overall, I guess you get the point. Especially when it comes to more fundamental stuff like what protocol to use, and other foundational things, having your own solution can make a huge difference.
Well if you decide to write your own dedicated/authoritative server for your game then the main choice is to use the game engine for the server as well or use some compact simulation software on the server. It is a big topic but in general if you need lots of the features of the game engine and/or your time to market is very important, then you should definitely go forward with the game engine but if you are making something which has to be as cost-effective as possible, then if you have the right people, you should try to code it on the server using a lightweight framework which can share its core with the client part of your game if the technologies partially match.
The decision of what to do regarding all of these is not easy at all in many cases. Attempting this at the wrong time can cause your project to fail and your team to endure hard times but almost always for teams bigger than 5 developers or time frames larger than 8-10 months, it is an option which should not be dismissed lightly. Be aware that it would mean at least one of your developers should exclusively work on this but the good thing is that for next titles, lots of it can be reused unless you decide to build a game which is very different in many dimensions.
At the end I just would like to say that we do contract work and consulting on Unity and distributed server systems both. We would be happy to talk to you if you are into an interesting project and like quality.