About the Book Link to heading

This book, tentatively titled ‘Game Engine Internals’, is an introductory to intermediate level programming book dealing with real world game development problems that you can sink your teeth into. It aims to be the systems engineering equivalent of a basic mathematics or physics programming book and is therefore intended for readers interested in the guts of game programming.

Tackling the game development process from a fresh perspective - that of the fuel feeding modern game engines - it examines the fundamental role that data plays in the game development process from the very lowest level data records, up through a game’s tools, out via the run-time of a game’s engine, and into players’ hands. The book exposes the hidden internals of modern game engines using applied examples in common game genres to help illustrate and teach the concepts presented.

Specifically, “Game Engine Internals”:

  • Introduces you to the basics of data record and data format definition, data serialization, and the flow of data production in games.
  • Teaches you both traditional and modern game object creation techniques including the use of archetype, prototype, and template definitions as well as data driven component composition.
  • Explores various ways you can use data to control a game’s run time including: game asset customization, data binding, function binding, and simple scripting.
  • Shows techniques you can use to build data introspection into your code, and then, ways you can apply that introspection to automate tasks such as serialization, data validation, and generic data editing.
  • Provides primers on player initiated load/save, multi-threaded loading, and data migration. It even shows you some ways you can usefully integrate relational databases into your games.

By specifically eschewing graphics, sound, input, and many of the other topics covered in typical game engine programming books, this book can give you a solid foundation from which to layer on those topics, or a way to weld the pieces of game engine knowledge you already have into a workable whole.

As an introductory book this book definitely doesn’t require you to be a programming guru, nor any sort of expert at developing game engines or game system architectures. You do need a good grasp on C++, as well as common data structures such as linked lists, queues, and trees, but given those basics you will find this book useful. One note: I do make the occasional use of templates and also of STL, but, if you’re not familiar with those aspects of C++ don’t let that scare you off. A good C++ book, or even the internet by your side, should help to see you though.

About the Outline Link to heading

The following outline is a talking outline. Each chapter description below starts with a summary sentence fragment describing the major task the reader undertakes during the chapter. That fragment is followed by one or more paragraphs describing the purpose, contents, and structure of the chapter. Each chapter description ends with the key terms and concepts that readers will have encountered during that chapter in the final book.

The chapters I present here move in roughly increasing complexity, later chapters building on earlier chapters. I’ve tried to include those topics, which, based on my experience as a game developer I believe are the most useful to new developers. The last three chapters, however, look forward. They are issues that have arisen largely only in the newest of games.

My intention is to keep the book to a digestible 300-350 pages.

A first draft of Chapter 8: “Templates”, is available here: templates.pdf

Chapter 1: Game Objects Link to heading

Building the hook upon which game data hangs: the game object.

Game objects are the most fundamental unit of a game’s processing logic. They are the conceptual representation of a real world analog - a unique, tangible, entity that artists, designers, programmers, and even players, can all recognize - and yet defining just what is and what is not a game object can be incredible hard to do. Is a tank a game object, how about the turrets of a tank, or its treads? When the tank explodes into pieces, is each piece its own game object? Were those pieces game objects all along?

This short chapter introduces a simplified game object class, and looks at what kinds of data come together to create game objects, namely: assets and data records. It examines the lifetime of game objects from construction to destruction. It shows you how to build a simplified game main loop to help frame the other code examples used throughout the book.

key concepts: assets, data records, game objects, instances, the main loop

Chapter 2: Data Records Link to heading

Controlling the properties of game objects with data.

This chapter walks you though the creation of your own data records, shows you the options you have available when mapping one or more data records to one or more data files on disk, and teaches you how to create “catalogs” to search for records at run time.

Using a simple “Beserk!” style arcade game as an example, it shows how to map data records to game objects, how records can select concrete object types, how catalogs can support both uniform, and non-uniform types, and what pros and cons of different catalog methodologies have. This chapter shows you how to permit data records to reference each other. Finally, it illustrates methods to present data to designers.

key concepts: data records, catalogs, nicknames, references, type registries

Chapter 3: Text Serialization Link to heading

Using YAML to store game data

While you should now have a basic grasp on how to setup a game to support customized data, you’re missing an important piece of the puzzle. How exactly does that data get read into your game, and how, after you modify it, does it get written back out?

This chapter teaches you the YAML data format and shows you a simple interface to save and load YAML data files. The goal of this chapter is not, truly, to teach the implementation details of YAML serialization - off the shelf tools will largely suffice for that - instead it’s to teach how such systems can be used to perform common tasks and solve common problems that occur during the serialization process. This chapter therefore teaches the transformation of data from text friendly to run time friendly representation, how to use serialization hooks to link up with appropriate game systems, and how to use fixed names to better persist references between data records.

Space permitting the chapter will also look briefly at the most common plain text format: XML. What it looks like, how it compares to YAML, two common ways its used for game data structures ( name-pair, and member-as-tags ), and where to find more information about using XML.

key concepts: data activation, serialization, yaml, xml

Chapter 4: Game Assets Link to heading

Employing content from third party tools to present game objects to players.

Game assets comprise the majority of a game’s data. Everything from geometry of subterranean caves to the hair raising battle cries of the exiled garden gnomes that call the caves their home, assets are the primary face of a game to its players. More technically, an asset is any content created outside of your game’s code base. Typically constructed in a third-party application, assets tend to be mainly graphics or sound data, but can include behind the scenes data such as multi-language string tables.

Because content specific algorithms change so much over time I avoid discussions of things like how to triangle strip your 3d models, and instead focus on teaching you how to weld assets of all types into your game.

Specifically, I’ll walk you through the following key tasks:

  • Importing and registering assets
  • Extracting ( and storing ) data from assets
  • Exposing asset customization hooks to designers
  • Managing the loading and unloading of asset data
  • Obtaining references to assets at run time
  • Handling bad or missing assets
  • Separating the mechanics of game play from the raw asset data itself

key concepts: assets,data extraction, data customization, mechanics, working sets

Chapter 5: Data Pipeline Link to heading

Understanding the processes that prepare game data for runtime usage

In the first few chapters you’ve learned how to manipulate the basic elements of game data, now I’ll show you how those elements fit together in the larger context of game development. Specifically, this chapter covers the data pipeline: the tools and processes your team uses to bring new data into your game.

Using a main menu system as a concrete example, the chapter follows the evolution of user interface data from initial conception to eventual product release. While much more descriptive than any of the other chapters, using this use case I will tie the discussion down to actual tasks and code samples wherever possible.

This chapter introduces the data pipeline. It examines who typically uses the data pipeline, and what their roles are over the course of data development. It shows how you, as a programmer, interface with the pipeline. This chapter illustrates the importance physically separating your single host development environment from your potentially several target run-time environments. It examines potential host and target data formats. It presents typical host and target data storage mechanisms.

This chapter follows the transformation of data through exporters, importers, converters, and packagers. It describes how tracking, error management, and reporting fit into the big picture. It emphasizes the importance of quick iteration in the data development cycle.

key concepts: data pipelines, data storage, hosts, targets

Chapter 6: Archetypes Link to heading

Sharing simple designer customized object types

This chapter is the first of three game object data patterns. Each follows the structure from the Gang of Four’s design pattern book - examining why the pattern is needed, how the pattern works, and how the pattern is constructed. Each chapter uses a unique game genre to show you how to employ the pattern, and closes by relating popular games that have successfully employed the pattern

This first pattern, the archetype pattern, is probably the most commonly used data pattern in games. It extracts common data from game objects and stores that data in a separate, standalone, shareable structure. This chapter explains the archetypes pattern. It shows how to use it to supply both type invariant data, and game object default data. It teaches you how to protect game code from changes to archetype layout. Examining a simple real time strategy game, this chapter demonstrates how you can employ archetype data to help your designers define, then balance the strength of competing enemy factions.

key concepts: archetypes, default data, data sharing, invariant data

Chapter 7: Prototypes Link to heading

Creating highly customizable game object instances

The second of three game object data patterns, prototypes create new game objects by cloning old game objects. This chapter shows how the prototype design pattern gives designers detailed control over the properties of game objects with a minimum of programmer work. It teaches you some simple methods for cloning game objects including: load time cloning and run time cloning. It examines the difference between shallow and deep copying. Using a “Mario” style platform game as an example, this chapter shows how you can use cloning to implement a simple enemy generator.

key concepts: cloning, prototypes, generators

Chapter 8: Templates Link to heading

Creating and managing highly varied game object types

The last of the game object data patterns, templates provide a way for hierarchies of data types to work together to create new game objects. This chapter explains the template pattern, and its key components: partial specification and hierarchy definition. It examines how templates can be combined together, and applied to new instances. Using a role playing game as an example, it walks you through specifying a simple game world using actor placeholders, defining actors and their templates, saving and loading template hierarchies in YAML, and, finally, using templates to guide actor creation.

Chapter sample available: templates.pdf

key concepts: data hierarchy, partial specification, placeholder objects, templates

Chapter 9: Examining Data with Introspection Link to heading

Procedurally manipulating programming defined data structures

In the previous chapters, each time you wanted to save a data structure, you had to manually provide the necessary code. Such rote typing can be both time consuming and error prone. This chapter provides an alternative.

Moving in stages, I introduce an interface for parsing the contents of C++ classes, plus a markup system that you can use to enable the parsing of your own game’s classes. Using the task of data serialization as an example, this chapter teaches the mechanics for accessing the structure and contents of a class including: individual members, base classes, arrays, lists, enumerations, and aggregated structures. It also teaches how to outfit automated load code with parse-style translation and event hooks.

While hopefully you will find the introspection mechanism presented here directly useful, the primary purpose of this chapter is to encourage you to think of your class structures not as dead code, but as pliable data in and of themselves. Structures capable of supporting many types of generic programming.

Space permitting this chapter wraps up by introducing some real world examples of introspection and introspection generation in four rough categories: native support, scripted extensions, schema based code generation, and code analysis.

key concepts: introspection, reflection, schema

Chapter 10: Automating Tasks with Introspection Link to heading

Solving common tasks using introspection

Since implementing introspection involves its own investment, if it were only useful for serialization it might not be worth the effort. It has, however, many disparate applications. This chapter teaches you three: template specification, generic property editing, and network packet marshalling. It introduces metadata markup for customizing introspection based algorithms, and looks at ways to use that markup to direct single structures to multiple ends.

The chapter rounds itself out by providing a popular alternative to run-time introspection: namely introspection based code-generation. It walks back through the network packet marshalling example to show you how code generation works. It weighs the basic pros and cons of run-time introspection vs. code generation, leaving the ultimate choice up to you.

key concepts: class markup, code generation, introspection

Chapter 11: Activating Data Link to heading

Transferring program control from code to data

Data by itself does nothing. It must, in some way, interface with code in order to affect a game. These interactions can be separated into two categories: “active code”, and “active data”. Systems written from the perspective of active code, parse and manipulate data to achieve their goals. Systems written from the perspective of active data, however, let data direct code.

This chapter walks quickly through a handful of mechanisms used to activate data. It gives you broad exposure to how different kinds of active data work, and gives you a basic handle on the common techniques behind their usage.

Using a point and click adventure game as an example, this chapter teaches: exposing game events in data, filtering events based on predefined queries, calling programmer defined functions based on data, and binding data sinks to sources.

key concepts: events, data binding, filters, function binding, sources and sinks

Chapter 12: Designer Defined Data Link to heading

Moving the contents and composition of game objects to the design team

This chapter is a companion to Chapter 11. It shows three key ways you can move the definition of game objects from code to data, namely by adding custom data to existing objects, creating custom data structures for new game objects, and building new game objects out of predefined components.

First, the chapter introduces variant data, shows a way to define it, and teaches you a way to serialize it. It shows how you can use variants to allow designers to tag game objects with custom data, and how you can use tags to provide additional event handling customization.. Extending that concept it teaches you the use of named tags ( aka. attributes ), and a way you can turn complex attribute based event filtering into relatively simple queries.

Second, the chapter looks at how you can use the attribute mechanism to give designers the ability to build custom build data structures. It shows how such structures provide unique new data sources for generic data sinks, especially in user interface.

Last, the chapter looks briefly at component objects. Components provide a way for designers to build custom object types out of smaller pre-defined pieces. Component objects can be used either as an extension to, or an alternative for, customized event handling and designer defined data structures. This section teaches you a simple mechanism for supporting component objects, giving you a way to freely experiment with the concept.

key concepts: attributes, component objects, custom objects, tags, variants

Chapter 13: Saving and Reloading Games Link to heading

Finding ways to track player progress

From adventure games to fast paced action games few players expect to have play through their previous progress the very next time they pick up a game. This chapter looks at several different ways to save and reload a game, proceeding in a somewhat, chronological/historical order.

  • Accomplishments: Using key moments in game play to reconstruct progress.
  • Memory dumps: Saving and loading wholesale chunks of run time memory to completely reconstruct game state.
  • Object traversals: Walking active game objects to extract/restore object state. Notes on reconstructing state while avoiding state transitions ( the gun reload at reload problem ).
  • Journals: Recording game object changes on the fly. Replaying moves at load time. Notes on the similarity to network prediction / correction. Remember to watch out for those random seeds.

Each section in the chapter covers a single save game methodology from the above list. Each presents a brief real-world scenario, followed by a basic implementation of the method at hand.

key concepts: accomplishments, derivable state, journals, memory dumps, object traversals, save games.

Chapter 14: Binary Serialization Link to heading

Optimizing data size and loading speeds with binary representations of game data

As earlier chapters have shown, text based data formats provide easy to read, easy to maintain data. Text formats do have their drawbacks however. They are often both large in size and slow to load. This chapter seeks to teach you the skills necessary to build an alternative: your own, custom, binary data format.

This chapter looks at two forms of binary data, streams and blobs, ways to serialize them, and reasons why you may prefer one over the other. It explains issues common to all binary data and ways to solve them, including: endian order, alignment, pointer persistence, and versioning,. Further, the blob data section also looks at how to address the lack of parse style registration hooks, how virtual functions cause problems and what to do about them.

key concepts: binary serialization, blobs, streams

Chapter 15: Supporting Change Link to heading

Migrating data formats and supporting downloadable content

No game is designed in detail before the programming begins. Given that game programmers must be able to handle the unexpected, what techniques can you use to help make changes to data as painless as possible?

This chapter looks at data migration: what does it mean, when does it occur, where can it be implemented. It discusses different levels of data migration: from the trivial addition or removal of data structure members to the renaming of members; from the machinations of changing container classes, to the more complex splitting and combining of existing data structures.

It explains how to detect changes to existing assets, and how to defend against changes to third party tools and sdks.

It wraps up by looking downloadable content: how to support the integration of new data after having already shipped, and how to protect code from bad or broken data.

key concepts: downloadable content, migration, validation, versioning

Chapter 16: Streaming Data Link to heading

Implementing an uninterrupted game play experience

Each new generation of console hardware brings both more storage space and more processing power. Memory size, however, while also increasing, has never kept pace. Teams often find themselves faced with a hard choice: break up levels and introduce loading screens, or introduce the complexity of streaming. This chapter will focus on providing, first, a general overview of the issues involved in streaming, and then, teaching you the mechanics of loading data from a separate thread.

Streaming issues covered:

  • Load invocation: area movement, and on demand requests
  • Slow media: esp. dvd seeking, and dvd throughput
  • Memory management: fragmentation, shared resources, resource location
  • Issue hiding: priming, proxies, portals, priorities
  • Advantages of data packaging, compression

Loading mechanics presented:

  • Creating and destroying threads
  • Locking and unlocking resources
  • Loading resources in stages: resources states: pending and ready resources; resources stages: loaded, active, destroyed, and errored.
  • Handling packaged resources

key concepts: dependencies, packages, streaming, threaded loaded

Chapter 17: Databases Link to heading

Leveraging relational storage to increase productivity

Its almost ironic that the cutting edge in game programming comes from the business technology of the 70s and 80s but, as games grow in size, that irony is becoming a common place reality.

Based on my own experiences in the game industry, I’ve seen that the biggest stumbling block on the road to creating a modern game involves not raw the graphics, animation, or sound technologies but the management and delivery of large numbers of highly detailed assets and data records.

This chapter looks at ways developers are incorporating relational databases into their games to help handle the ever increasing amounts of game content. It concentrates on three key uses: the tracking of game assets and associated metadata during production, the use of embedded databases, such as SQLite, at run time, and the external storage of key game data especially as illustrated by Massively Multiplayer Online Games.

Specifically, it addresses: the basic elements of a database management system, where a database fits into your pipeline, how you can store asset status, how you can track links between your assets and your data records, how you can run useful queries about your assets, what is SQLite, how SQLite can increase a data set’s flexibility, how to store data records in a db, what are transactions, and what’s happening behind the scenes in an MMO.

This chapter assumes no prior knowledge of either SQL or relational databases. It also does not seek to be an in depth primer on these things. Instead, it’s meant to give you a handle on some basic tasks relational databases can help accomplish, and spur your interest in database technology.

key concepts: asset tracking, data storage, embedded databases, relational databases, SQL