Archive for June, 2008

Deterministic Game Testing

Posted in Video Game Development on June 24, 2008 by Dan Amerson

Noel Llopsis recently wrote a great two-part article in Game Developer on deterministic playback systems for game state testing. It’s a fantastic read that hits a lot of the high points for testing. After reading it, I compared it with some of the systems we use at Emergent. In our system, we run a battery of tests every 24 hours on an automated system, so my comments are largely about automation and less about QA and user input testing which is well-covered in the articles. There are some differences which are warranted that I wanted to share because I think more information on testing and determinism is a good thing. Games and specifically 3D graphics are hard to test.

Using Fixed Timing

Noel mentioned recording the time and frame ID for each frame as a good way to control issues with timing across machines. For automated testing, I prefer using a fixed time step. The upside is that you don’t have to record the time. Once testing is turned on, you just override the clock and each frame takes 1/60 of a second. As noted in the article, this can also accelerate things if you turn off VSynch as well. For us, most of our tests are simply files that reproduce a single feature. The frame time is much less than 1/60 of a second, so we can slam through 100+ frames for a test file in less than a second. It’s a huge win for an automated test suite with thousands of tests that takes in excess of 20 hours to complete. The downside, obviously, is that you have no variability in your times and frame deltas. A fixed time won’t catch odd edge cases in your math. Given that a large portion of our tests are focused on rendering artifacts, as I’ll note below, this isn’t a huge deal for us, but you should consider it when planning any system for yourself.

Image Differencing vs. Perceptual Differencing

The first part of the article mentions a perceptual image differencing utility that determines how visible a change in the image will be to the human eye. In our tests, we use image differencing to compare known good results with the current test run. This technique will catch even the smallest change of a single bit in a single channel of the color of a single pixel. When there’s a failure, we actually have to amplify the results most of the time to see what went wrong. For game visual testing, a perceptual differencing is probably more than sufficient.

For our purposes as a middleware provider and because each file in our test only runs for about 2 seconds, the differencing actually serves us much better because it will catch small math errors throughout the pipeline which may only manifest as a tiny difference on screen. For example, a perceptual utility might not flag an error for an animated character that is one frame off because only a small number of pixels differ in the animation. If that error were to accumulate, it could cause problems. We should chase that down. The other issue as a middleware provider is that we have no idea what our customers will rely upon or are currently relying upon in our code. That minor math error may be a big problem for their assumptions, so we try to chase them down.

Using image differencing does lead to some real problems with false negatives in testing due to hardware changes which leads to my next point.

Use Fixed Hardware

Even if you think you’re doing everything the same, the hardware can still cause you problems. For example:

  • There are differences in internal floating-point precision between some Intel and AMD chips.
  • Ditto for graphics cards.
  • Filtering precision can differ for graphics cards.
    • Color interpolation may occur in 32-bit space vs. 16-bit space for textures.
    • Mipmapping calculations may differ from card to card.

I could add more, but the point is that you can’t rely on anything if you’re testing raw pixel outputs. You need to use fixed hardware. For consoles, this is no problem. For PC, however, make sure you are using the same CPU, GPU, and drivers. Obviously, this reduces your hardware coverage, but automated testing can’t solve everything.

Multithreading

Noel noted multithreading as one of the biggest pains in his article. I’ve left it for last as well. There’s a bunch of pain there, and I only have tips.

  • If you leave threading turned on, look out for thread local data. It will introduce non-determinism that you didn’t think of. One of the most common culprits is random number generation. Even if you seed with a consistent value, the seed of the RNG is usually stored per-thread for performance. If the number of calls to the RNG varies per-thread, you may have false negatives in your testing.
  • Consider a serial mode for your threading systems. We have a system called Floodgate for stream processing in Gamebryo. When we run with “-test” for some applications, we simply flip it into serial execution mode. Instead of running the tasks on worker threads, it simply runs them when submitted. It’s much slower and really a different configuration, but it’s much more testable.

Wrap It Up, B

I think that’s plenty of words, and I can see why Noel ended up writing a two part article. Regardless, I hope my additional ramblings add a bit of color to his work.

This post has not been fully tested. dba

Wanted

Posted in Miscellany on June 20, 2008 by Dan Amerson

There’s a movie coming out Friday called wanted. You can check out the official site at the link below. In fact, I encourage it since I’m planning to talk about the comic it’s based on a second.

http://www.wantedmovie.com/

Knowing that this movie was coming out, I read the comic recently. The comic was fair, and it had a good premise. Wesley is in a dead-end job. He’s got no backbone. His father was a bad ass and just died. Yes, in the comic a woman shows up and changes his world. She tells him about his father. He was called The Killer, and he was a member of a fraternity of supervillains.

You feeling it now? Sounds like the trailer up until then, but we’re talking supervillains, as in the opposite of superheroes. See, the main conceit of the comic is that around 1986, all the supervillains teamed up and killed all the superheroes in the world, all of them. The supervillains now run the world. They can do whatever they want, and they have some serious technology that lets them revise history to hide all their actions. It’s a fun premise for a book. I don’t think the comic fulfilled it completely, but it was a fun read. I had no idea how the movie would handle it. From the trailers, it looks like the answer is that it won’t. I’m not sure if that’s fortunate or unfortunate, but I’ll probably go check out the movie to see.

Leave me a comment if you see it. dba

Bad Day Selling a House

Posted in Life in General on June 6, 2008 by Dan Amerson

My house is on the market, and I think I’m doing a good job selling it. As part of that, I decided to get the AC checked out today. It’s unfortunately bad when the technician sees your unit and says, “Man, I wish I had a camera. I’ve never seen one this old.” It only goes downhill from there.

Now, I expected to have a few problems. I knew the unit was old when I bought the house. However, I wasn’t aware that it was 25 years old… at a minimum. (For those not in the know, the lifetime of an AC unit is 12 to 17 years.) Let’s list the bad news from today.

  • The air filter is under the house.
  • Since I thought there was no filter in this old unit, my air filter was 3 years old and mad dirty.
  • The condenser pump is dead causing water to pool under the house and grow mold.
  • My flue pipe rusted out, and exhaust from the gas furnace is venting under the house. This is know as a carbon monoxide hazard.

I guess I have to fix that stuff now. Luckily, I just have to crawl under the house with a new filter and pay about $500 for those repairs. Let’s all hope together that the actual AC unit doesn’t die. Apparently those cost about $5K which was news to me. In my mind, it was about $2,500.

Instant inflation. Mmmm.

dba

Gamebryo 2.5 is Gold!

Posted in Video Game Development with tags on June 5, 2008 by Dan Amerson

If it wasn’t obvious from the link on the right, I work for Emergent Game Technologies. More specifically, I work on Gamebryo. I’m the technical director for the runtime graphics components. I’m excited to announce that as of 4PM today, Gamebryo 2.5 is gold and will begin shipping. This is really exciting as we’ve been working on this release for a long time. It’s got a ton of new features for our customers.

  • Completely redesigned fleixble geometry system.
  • GPU Instancing.
  • Mesh modifiers for automatic parallel execution of tasks.
  • New terrain system.
  • Improved version of Floodgate with tighter engine integration.
  • More!

I’m really excited about what our customers are going to be able to do with this. I’ll blog more on the topic later, but I unfortunately don’t get to celebrate too long today as I’m behind on some other stuff. 😦

dba

Hello world!

Posted in Miscellany on June 4, 2008 by Dan Amerson

Welcome to my blog. I’m just starting to set up categories and formatting for now. More content will be forthcoming.