Ask any developer experienced with multithreading about his or her biggest obstacle to writing threaded code, and you'll probably get the same answer: debugging the stuff.

Havok found that creating their own debugging tool, the Visual Debugger, paid off surprisingly well. Gargan told me a little bit about how it works.

"We're embedding timer information as a timer stream directly into the code. Our builds are shipped with that timer information in them. Putting that into the code has given us the biggest tool that we use to analyze performance. That's been so invaluable. What we've done on all our builds is we have a direct TCP connection to our running physics engine and a developer can straight away, without giving us source code or anything else, they can take a running profile of the game. And we get a visual profile and that statistic profile that comes out with it. They can send us what's really a movie, and we can analyze frame by frame [to see] where the performance has gone."

In-house, even a high-end development company like Havok relies on the same tools we all know and love. "On the PC, we just use Visual Studio for multithreaded debugging. We find that a great, great tool. And in particular, one feature that I like is the ability to set watches on different variables. Not all debuggers support that, especially not on the PC platforms. And it's a great thing. If you've got a memory overwrite or something else and a different thread is running, at least it will say stop execution WHEN the thing went wrong. Because with multithreaded programming, often you find out something has gone wrong somewhere previously, where with sequential debugging you usually find out right now something has gone wrong. The ability to set a watch and let you know and stop execution immediately when things go wrong is great." In fact, "We think it's great to the point where we'll write our simulators where they run on the .NET environment because the debugging opportunities are a lot higher."

And of course any developer working on an AMD platform should be using AMD CodeAnalyst™. It gives you timer-based sampling, event-based sampling, thread profiling, all of which work on 32-bit or 64-bit, Windows or Linux. Best of all, it's a free download.