Software engineering, file formats, build processes, and packaging...

[MikaelKalms]

This is a story about software engineering, file formats, build processes, and packaging. All framed within the Project Management Triangle.


Developing a game is developing a software suite, and a dataset that will go along with the software. Users will use the software (= run the game executable) to manipulate the dataset (= use mouse/keyboard/joystick to control the game).

A good game requires that the capabilities of the software matches the dataset that is being created. The software is usually refined in parallel with the dataset. In order words, the game engine and the game worlds are tailored to each other, to some extent.

The programming side of making a game corresponds fairly much to developing other kinds of software. You usually use mature languages, standardized tools, and techniques which were pioneered back in the 1960s to create a set of source code, which when built creates a single game executable.

Creating the content is less straight-forward. Sometimes there are tools that do the job well (Maya, Photoshop, SoundForge, Cubase etc). For other kinds of content, there are no good tools, so the game developers develop their own.

Raw source format is not a good format for distributing a game engine to end users. One could ship the code as-is, but that would require people to have the right compilers and software SDKs available on their machines. Distributing the code in the form of a game executable is more practical.

Raw source format is not a good format for distributing the game content either. It is convenient for editing, but the game engine will usually want the content in a different format before using it. The raw source format is often bulky, and the conversion step is lengthy. Therefore, game developers usually create custom tools which "cook" the data -- convert it from source format to something that suits the game engine better.



Cooking is good, and bad.

No cooking gives you the advantage that you can change a source file, restart the game, and the effect happens immediately in the game. It is usually easy to figure out which files on-disk contribute to what in-game.

With no cooking -- or just too little cooking -- you get very long loading times. You usually get more memory fragmentation. You also lack mechanisms to validate the integrity of the data; if you want to see that it's consistent, you have to play through the full game and exercise all aspects of the game.

Cooking gives you the advantage that you can do lots of sanity checks on the data before even starting the game. You can improve loading times a lot (especially off media with slow seek times such as DVD), and you get less memory fragmentation.

You can also create extra datastructures, which improve runtime performance. (A good example of this is the BSP & PVS trees that were used in FPses back in the 90s.)

With too much cooking, you find that it is difficult to change anything when data already has been cooked. If you want to edit just one tiny little detail, you have to re-cook ALL the data. It is difficult to tell which files on-disk contribute to what in-game.


Now let us consider the Frostbite engine and where it comes from.

It was initially used to create BFBC1. This game was released only for consoles. This means that the team which developed

BC1 had to do a certain amount of cooking - mainly to avoid excessive load times and memory fragmentation. The hard memory and performance constraints of running on a console also made it more important to pre-compute some things, and to package data into suitable archives.

With this foundation, we essentially had a game engine which solved a lot of time-consuming problems for us when we began on BFBC2 PC. Loading times would be under control, it's easy to figure out which files go into which archives, and which files/archives belong to which level. These are things which are often overlooked when not using a game engine that has been used to ship games on DVD.

We also wanted a way to automatically patch the game. Making an auto-patcher that works properly under Windows XP, Vista & Win7, knows about limited users & UAC, and can handle restarting mid-way through a patch at all takes a huge amount of time.

Therefore we took the auto-patcher from BF Heroes and modified it slightly. Voila, the game could now pull down updates from the internet and apply them to its own datafiles. We were all set.

Or so we thought.



Some complex systems seem simple on the surface; it is only when you look under the hood that they turn out to be tricky.

The tools which "cook" the game's datafiles takes in a set of original files, which are roughly 80GB in size. Most of the files here are organized by function ("this is a mesh, this is a texture, this is an animation, ...") rather than location (on which level[s] it is used). The tools will process the dataset once per level, extract the subset that applies for the current level, and convert it to a format suitable for the game engine.

This allows the tools to do some per-level precomputations easily; for instance, since all the pixel/vertex shader combinations that will be used throughout the entire level is known, the tools pre-generate all these combinations and store them in a "shader database". (BF2142 generated & compiled the shader combinations during first load - that's one reason why first launch of a level was very slow there.)

After this is done for all levels, there are a bunch of archives for each level. This is ideal for quick loading times and no memory fragmentation, but it wastes diskspace unnecessarily. The result is probably about 15GB in size.

In order to make a better tradeoff between diskspace and load times, an extra processing step has been added; all the level-archives are compared, and any datafiles which appear inside many of these level-archives are moved to a level-common archive. So when loading a level, everything inside the level's own archives and the level-common archive has to be processed. This reduced the total data size down to about 6GB.

This technique allowed BFBC2 to fit on a DVD, both for the console and the PC versions. It is not perfect by any stretch, but dealing with these large amounts of data is time consuming, and therefore you don't try every idea under the sun - rather, try what seems most likely to work first, and then keep on until the end result is good enough.



So this is all awesome when shipping the game. Where do the problems begin?

When you begin creating patches.

First off, it turns out that the tools that "cook" the data don't produce binary-identical data all the time. The result after cooking is always functionally identical, but the bit-content may differ slightly. Items in unordered lists change order, uninitialized fields contain random data, that sort of stuff.

Why didn't this get caught sooner? Because you only notice problems of this kind when re-cooking the same data several times, from scratch. And re-cooking all BC2 data takes over 48 hours for a high-end computer. And locating & correcting all places where cooking isn't deterministic down to the bit level would take a lot of time (both calendar time and effective development time). Perhaps that time is better spent elsewhere?

So. If different "cooking" runs produce slightly different results, it is suddenly difficult to look at version A and version B of the data and answer the question, "what are the differences between these two datasets?". It's easy when looking at the source data, but when looking at the cooked data there are a lot of changes which have no effect on the final game experience.

There are about 40.000 source files, and this results in well over 100.000 cooked files. Going through those by hand is not an option. Writing a perfect filter, which knows which differences are benign and which are for real, will take as much time and effort as making the cooking 100% deterministic. Neither that is an option.

So you make a filter which does something in between; be smart, create rules for the file types you know about, and when in doubt - assume that the change is for real. Err on the side of caution.


Then you realize that those shader databases were never designed to be extendable. What happens when a new object is added to a level in a patch? Its mesh & textures are included, no sweat, but what about the shader combinations? How does one create add something to the shader database, when the shader database is an opaque binary block whose entire contents may change when just one object is added to the level?
(One shader database is about 5MB. There are three shader databases per level - one each for DX9, DX10 and DX11.)


And finally, the patch system itself. Yes, it can replace portions of files on-disk. But due to its heritage (from BF Heroes), it is not able to open BFBC2's archive files and apply differences to individual files within the archives.
The only straightforward way is to make all patched-in content arrive in archives on the side of the original archives.



Given the above scenario, we end up with the situation that we have today.

Each patch gets larger than the previous, because the game drifts ever further away from what was shipped on the DVD. Changes that require shader database updates make the patch balloon in size. And we have to be careful and clever when selecting which file types to include and which to ignore when creating the patch.

And that's where we finally ran into real problems. It was too difficult for one person to identify which changes were required and which were not, and how to update the patch generation process to accommodate the latest set of changes. Most of the delay of Client R8 was because there are very few people at DICE who have the in-depth knowledge of the far-spanning corners of the game engine *and* the cooking tools *and* the patch generation process, to work out what is going wrong, why, and how to fix it.

The new content did work a long while ago - but the patch was back then approximately 7GB large. The patch had to get down to less than 1GB, or else some customers in Australia and South Africa would not be able to download it due to bandwith caps.

[As an aside - how about distributing the patch as an installer, for those who prefer that option? We would love to do so, but creating an installer that does anything out of the absolutely most ordinary installer-y requires ridiculous amounts of development time.]


I think that we have a proper Client R8 patch ready for release, but the approach we have been using thus far has been taken as far as it can go. (A bit too far, considering the delays.) We want to continue supporting the game, but if we want to change anything else than the game executable itself, we will need to spend time on figuring out how to do so with a less error-prone patch generation procedure ... and preferably without generating as large patches.

This stuff keeps me awake at night. What are your demons made of?

[silentcyanide]

Thank you

[notodd1991]

nice

[CubanJalh]

thanks a lot for the info !

[MFMAdiuvo]

Given the amount of detail you went into, I'd be amazed if people still ask questions about why you can't release small patches repeatedly or why patches might take some time.

Thanks for the information! It definitely opens up what the world of game development really is like.

[AdrIneX]

Thank you, my eyes are a bit opened now.

[Nosos22]

I am impressed MK. I think out of the known personalities on the forums, you have it the worst. I would not like to think of the stress put upon you. It's too bad Dice is short staffed in experts regarding the Frostbite engine, it must make your life hell.

I am curious in how difficult it would be to change the Heroes patcher to a custom patcher. In the end would it save more time than spent? I assume MOH and soon the BFBC2 Vietnam will be using the Heroes patcher as well, so it's inevitable that it's going to be next to impossible to support MOH, BFBC2 and BFBC2 Vietnam at once. Imagine each requiring a patch at the same time. Who would do it?

[xyzeratul]

So how about this DICE?

Hire more ppl for patching! get some good coders for this game's patch support team.

and yes, I for one like patch as the old way, a normal installer, download from your server and run it, many games have that, and if the patch itself won't go wrong, the installer is fine.

old school install also is good for those ppl who have poor internet, my friends in China have to make their own patch for BC2 , cuz the auto patch drop every 2 min.
and u can even use BT to share them, this can make your file servers easier, I remeber in BC2's beta, many file servers were overload in that night.

[Osmium]

Originally Posted by MikaelKalms

We want to continue supporting the game, but we will need to spend time on figuring out how to do so with a less error-prone patch generation procedure ... and preferably without generating as large patches.

Try to make smaller patches at a regular base. If you feel the need to change some weapon stats it surely doesn't loads of QA. The more you put in the patch, the more can go wrong and the longer the community has to wait for balance changes and bug fixes. No need to release new content like the SPECACT skins at the same time, they cause the most trouble integrating in the game.

[Brorim]

thats one awesome reply Kalms .. Finally something that makes sense of all this.

straight talk is what works . Had you written this a month ago you would not have heard a peep from me anyways..
You have been doing a superhuman effort and you need to get a few more on the team that knows the intricate s of the issues with the patching..

Thanks Kalms for taking the time to clarify this.

[KryptonJo]

I can totally understand the delay and how complicated these things really are. Thank you Mikael =).

[tb00ner]

First of all I want to say total respect from me on this! You obviously have a lot of problems with this game since it's a console port...

But what you're basically saying is that expecting future patches for BFBC2 is a waste of time and energy? I'm too old to whine about bugs, glitches and so on, so I play the game "as is" and it gives me the satisfaction I need.

I dearly whish for the dream about BF3 to come true and pray to someone-I-don't-believe-in that it won't be a console port... That said it prolly won't be, based on your experiences from the BFBC2 port

[Frosty_AUT]

Quite an interesting read and explains a lot.

[C1trom]

Thank you very much for your insight.

You are the first DICE employee, who talked to us straight without the marketing lala, and evasive PR tactics.

From the sound of it, frostbyte is not the kind of engine I would wanna work with lol. Seems raw & rigid.

You should get a few trainees and force your knowledge into their heads, so you could share some of the work. Expecting one person to do all this, just wont work.

[EvilClownxx]

Thanks for all the hard work, it is appreciated by most Bc2 players.

Maybe people will stop screaming at DICE now?

[19071971]

Thank you for giving as such a detailed explanation, in my eyes you just redeemed yourself ( read DICE ).

This information while ago would surely cut the "flame" in the forums, but what is done is done, and we all know that the haters gona hate no matter what you do.

Just hope the patch corrects the performance problems some people are experience ( mines are due to slow connection and bad laptop ), and give the game a new start, although a couple of NEW maps ( new being the key word here) would be very good.

Thank you again for all your work both in the patch and in the post you just did.

[DaveyJonesOK]

w00t

[Jimknils]

Thanks a lot man, your work is certainly appreciated!

[rukqoa]

To solve the patch size problems, can you not just release patches every time a problem is solved?

The "datasets" that should be able to be manipulated easily (ie weapons stats, descriptions...etc) also shouldn't take such a large patch to fix. In short, balance patching should be able to be done as soon as you can figure out the values to change.

Also a suggestion, you can use an in-game P2P system to distribute patches! (instead of dl'ing my patch from your main server, I wouldn't mind downloading it from other computers that have already got the patch.)

[soulz2003]

Thanks for the detailed outlook on what it's like to go through development.

Good article .

[DirtyGhettoKid]

But maybe you forgot a little thing Kalms. If the game wasnt in such horrible shape at release you woulnt have to create those patches in the 1st place.

Frostbite sounds like a pretty bad engine under the hood now (unless the game is almost bug free at retail) no wonder why they dont/cant release mod tools.

Originally Posted by Osmium

Try to make smaller patches at a regular base. If you feel the need to change some weapon stats it surely doesn't loads of QA. The more you put in the patch, the more can go wrong and the longer the community has to wait for balance changes and bug fixes. No need to release new content like the SPECACT skins at the same time, they cause the most trouble integrating in the game.

Did you even read his post? And weapon stats are server sides so no need for client patch, only server restart.

[ubrytybtyd]

So real proplem is the people that work for dice? When game if complete everyone moves on and few stay fixing bugs and so WHY? When we do someting on work we all stay and fix things that broke or dont work.

[DivineIron]

Nice, big read. But now at least I understand why the patch has been delayed and how complicated it can get.

[rukqoa]

Originally Posted by MFMAdiuvo

Given the amount of detail you went into, I'd be amazed if people still ask questions about why you can't release small patches repeatedly or why patches might take some time.

Thanks for the information! It definitely opens up what the world of game development really is like.

Can you explain?
Also balance patches shouldn't be so big/complicated

[faaj]

wow i couldnt understand most of that post, but i think i know what you mean at the end. Anyway you should have definitely posted this a few weeks ago, however reading through this post i would have thought the patch would never have come out....