Dumping core

April 23, 2010

Your program just crashed, and you didn’t have a debugger attached. You can’t reproduce the crash after many attempt. How you do debug the problem?

Well, if your program had left a core dump, you could easily attach a debugger postmortem and get some kind of idea what state the program was in before it died. A core dump is essentially a dump of all memory in your program’s virtual address space: stack, heap, code and everything else.

On most systems, though, you won’t get a core dump when you crash, where a crash can come from a segfault (or any other signal), a call to abort(2) (such as via a failed assertion), a call to terminate() (such as via throwing an uncaught exception), or other similar avenues. Core dumps are rather large (after all, it’s all of the memory from the process) — they can easily be tens or hundreds of megabytes, even for simple programs, due to a large number of shared libraries being loaded. Your hard drive would fill up very quickly if every program that crashed left a core dump.

If you’re just poking around in the shell, you can enable core dumps with ulimit(1) to raise the core dump file size limit from 0 (the default) to something non-zero such as unlimited. This will cause any crashing programs spawned by that shell to leave core dumps. For example:

$ cat crash.c
int main(void)
    *(int *)1 = 2;  // cause a segfault
$ gcc crash.c -o crash
$ ./crash
Segmentation fault
$ ulimit -c unlimited
$ ./crash
Segmentation fault (core dumped)

Where the core dump ends up depends on your operating system. By default, Linux puts it in a file named core in the current working directory, and Mac OS X puts it in a file named /cores/core.<PID>, where <PID> is the process ID of the process that crashed. The exact name and location may vary by flavor and version of OS. See the core(5) man page for detailed discussion of core files on Linux.

Ok, so that’s all well and good if someone has the good nature to run ulimit before running your program, but few (if any) people will do so. If you want to say, “No really, I want core dumps!, you can call setrlimit(2) to set the limit for yourself and any child processes (which is all ulimit really does). Just make sure not to annoy your users by filling up their hard drives with core dumps. Which of course you won’t do because your code is perfect and never crashes anyways.

You’ve gone through the trouble of creating a core dump, but when your program crashes in some far away land, how do you actually get your hands on the core dump? You could ask your users to email it to you, but they’re not going to do that. They’re just going to complain on the Internet that your software sucks and that people shouldn’t use it. Some operating systems have a nice Crash Reporter or Error Reporting Service, but those send crash reports to first parties, something you might not want, and getting the crash data back to you is far from trivial.

One solution is to install your own error handlers in-process to catch things such as segfaults and instead of letting the operating system handle the error, you handle it yourself: you do your own stack trace, grab important data such as filenames, optionally pop up a UI asking the user if he wants to send an error report and for supplemental information, and sending the crash report your way. This is a lot of work, and it’s also dangerous: if your program has crashed, there’s no telling what state it’s in. Trying to do something like sending an email from a signal handler could easily fail — your heap might be corrupted, so you could crash again the moment you do something as mundane as try to allocated some memory. If you decide to go this route, a good place to start would be with signal(2)/sigaction(2) (*nix and OSX) or Structured Exception Handling (Windows).

A solution that I like better is out-of-process. Just let the process crash and dump core as before, but this time we’ll have a watchdog process running. The watchdog just waits for the main process to exit (normally or abnormally); if it sees an abnormal exit and a core dump, then it sends off the crash report into the ether. This is much safer, since you don’t have to worry about things such as a corrupted heap when sending a crash report. The only downside to this you now have twice as many processes running.

Here’s a full example of a watchdog with core dumps. The program forks, with the parent as the watchdog. The child intentionally crashes, and then the parent grabs the core dump if one was made.

#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <sys/resource.h>
#include <sys/wait.h>

int main(int argc, char **argv)
  // Try to enable core dumps
  struct rlimit core_limit;
  core_limit.rlim_cur = RLIM_INFINITY;
  core_limit.rlim_max = RLIM_INFINITY;
  if(setrlimit(RLIMIT_CORE, &core_limit) < 0)
    fprintf(stderr, "setrlimit: %s\nWarning: core dumps may be truncated or non-existant\n", strerror(errno));

  int status;
  case 0:
    // We are the child process -- run the actual program
    *(int *)1 = 2;  // segfault

  case -1:
    // An error occurred, shouldn't happen
    return -1;

    // We are the parent process -- wait for the child process to exit
    if(wait(&status) < 0)
    printf("child exited with status %d\n", status);
    if(WIFSIGNALED(status) && WCOREDUMP(status))
      printf("got a core dump\n");
      // find core dump, email it to your servers, etc.
  return 0;

If you compile and run this program, you’ll get a core dump from the child process, which the parent process will detect, and it can then do whatever it wants with it. Email it to you, upload it to a server, analyze it and trim it down before doing those, or anything else you can write code to do. All from the safety of an uncrashed process. If you run ulimit -c 0 before running this program, you’ll see the warning about setrlimit failing and you won’t get a core dump. This is because, if you look at the documentation for setrlimit, you’ll see that the soft limit can never exceed the hard limit, and the hard limit can only be decreased by unprivileged processes.

So there you have it. You now have a way to have your software dump core when it crashes and send those core dumps back to you without any extra hassle on the user’s part. Though depending on who your users are, it may still be a good idea to ask them if they want to send a crash report before actually doing so, since core dumps can easily contain private information in them. If you had anything like usernames or passwords in memory anywhere in your process, they’ll be in the core dump. So keep that in mind and take appropriate measures to protect users’ privacy. Encrypt the core dump if necessary. Maybe even attach a cryptographic signature to ensure authenticity.

Links for further enrichment:


It must be Retro Season again

February 16, 2010

Following in the success of Mega Man 9, Capcom has decided to release another sequel in the style of the old NES Mega Man games: Mega Man 10

(Not to be confused with Mega Man X)

Can’t wait. Though I will be avoiding easy mode, its existence will appease the people who thought Mega Man 9 was way too hard. Kids these days. They wouldn’t last more than 5 minutes in Battletoads before

And for the first^H^H^H^H^Hsecond time, Proto Man is playable! First not counting DLC. Or spin-off games. Speaking of which, why haven’t I bought the Mega Man 9 DLC? I really should. I just….haven’t. Don’t know why.

I vaguely remember reading somewhere that Proto Man’s character was supposed to be the original design for Mega Man, but the NES color palette didn’t have enough reds in it to make him look good, so the designers made him blue instead, and saved the red design for Proto. But, I can’t find a source for this fact, so take this with a grain of salt. I also don’t know much about the NES video hardware, so it’s entirely conceivable that it supports a customizable palette that can handle several shades of red at once.

Ok, so the Blue Bomber is back in town this March. Awesome. But you know else is back in town? The Blue Blur:

There was a pretty awesome leaked video showing actual gameplay, though that was quickly brought down by Sega, so I’ll have to settle for the interesting-but-not-a-lot-of-juicy-content official video above.

For the first time since the Genesis days of Sonic 3 & Knuckles, Sonic finally comes back to his roots for some good old-fashioned 2 (and-a-half) D speed platforming. I was a huge fan of the Sonic series back in the day, but nothing after the Genesis has been good enough for me. Though I hear that Sonic Rush was fairly decent; maybe I’ll get around to that some day, but it’s fairly low in my queue.

Yes, spring-summer 2010 is shaping up to be a fine, fine season of neo-retro-gaming.

Comments Off on It must be Retro Season again

Green Day: Rock Band

January 12, 2010

Ok, yeah, I know, I’ve not been updating this blog too frequently. I really should have written this entry a month ago, when the following announcement was made on the Spike Video Game Awards 2009:

Not included in the announcement was the fact that Demiurge Studios was working on Green Day: Rock Band. And I gotta say, it’s pretty freakin’ awesome working on it. I’d love to tell you more, but NDAs, yada yada yada, so I can’t say much else.

It’s strange, though, that I didn’t realize how big of a Green Day fan I was until we started working on this game and I listened to all of their albums. I loved a lot of their songs beforehand, but I didn’t know that those songs I loved were Green Day songs since I pay next to no attention to the artists behind particular songs. I don’t really buy much music, so most of the songs that I like that I’ve only heard on the radio, at parties, etc., I have no idea who plays them.

This makes the third Rock Band title I’ve worked on, the other two being Rock Band Country Track Pack and Rock Band Metal Track Pack. [Aside: Metal was originally announced to be released on Rocktober 13; we’d wrapped up work on it long before then. Then, lo and behold, on September 22, an email went around with “hey look, we’ve got a game on store shelves”. Turns out the release date got pushed forward by 3 weeks, and nobody at Demiurge realized it]. But this ain’t no wimpy little track pack — it’s a whole new game that’s an order of magnitude more challenging, and we’re working as hard as possible to make this game as awesome as possible.

Rock on!


Making p4 and Cygwin play nice

November 2, 2009

Say you’re a developer doing development on Windows, and say you’re using Perforce (better known as p4) for source control. Let’s also say you have *nix background and like using Cygwin to keep you from going insane. What are your options for interacting with p4?

Fortunately, Perforce provides a binary compiled against Cygwin. You can download that, drop it in your $PATH, and be on your merry way.

But I happen to like P4Win as a graphical client, and P4Win doesn’t play nicely with the Cygwin p4. Why? Client roots. With P4Win (or any other non-Cygwin client, for that matter), your client root is a Windows-style path, e.g. C:\path\to\root. But Cygwin sees things differently. It wants your client root to be something like /cygdrive/c/path/to/root.

So, if you have a client set up with P4Win and try to interact with the command line p4 in Cygwin, you’ll get messages like this:

'awesomecode.cc' is not under client's root 'C:\path\to\root'

You can get around this by using absolute paths, e.g.:

p4 edit $(cygpath -wa awesomecode.cc)

That gets tiresome very fast, though. But you, fearless reader, you are in luck! I have just the solution for you! It turns out that all p4 needs is a little persuasion to change its PWD environment variable. I whipped up a little C program that fixes up its PWD and exec’s the real p4, so it no longer gets confused.

You can download my program here. Enjoy!


If you’re gonna play the Gameboy, ya gotta learn to play it right

July 25, 2009

Ok, so that’s not quite how the lyrics for Kenny Rogers’ The Gambler go, but it’s an amusing case of misheard lyrics. The Gambler is just one of 21 great country songs you’ll find in the brand new Rock Band Country Track Pack, available now for the PS2, PS3, Xbox 360, and Wii.

Enough with the blatant advertising. Rock Band Country Track Pack was a joint venture between Demiurge Studios and Harmonix Music Systems that I’m proud to say I was a part of. I had a blast working on it, since I was already a huge fan of the Rock Band franchise. But prior to this past Tuesday’s release, we couldn’t tell anyone we were working on it, even though the game had been announced in May, thanks to some bigwigs in upper management who decided it wasn’t in their interests. Of course, now that the game is on the street and our logo and names are in the credits, we can talk freely.

Rock Band Country Track Pack now joins Brothers in Arms: Double Time in the elite class of retail games with my name in them. Mass Effect (PC) also has my name in it, as Demiurge credits all people in the studio in its products, although I only worked on that game for about two weeks. Word-Fu (iPhone) was another game I worked on, but it has very minimal credits. You can still find my (first) name at the top of the built-in high scores table — of course, that’s not my real high score. If they’d let me put in my real high score, that would have just been too depressing to people who bought the game who aren’t very good spellers.

What’s next for Demiurge? Only time will tell. Well, I could tell you, but then I’d have to kill you.


Spaces and Tabs

July 3, 2009

Like emacs vs. vi and what line you put your braces on, the issue of spaces versus tabs for indentation will always be a holy war among programmers. I’m from the always-use-spaces camp, but that’s not what this blog post is about.

Pretty much any editor worth its weight in gold (which is not very much) has a spaces/tab setting whereby you can hit the tab key and it inserts enough spaces to get you to the right indentation level, saving your wrists from tapping the space bar dozens of times and getting RSI. So if you’re a spaces guy like me, go with that setting.

And if you’re a tabs guy, use only tabs. Don’t mix the two, that will only lead to trouble when you’ve got someone else who uses a different tab size. You have to be extra-careful in order to make it look right at all tab sizes. Take the following, for instance, where [TB] is a 4-space tab, [-TAB--] is an 8-space tab, and _ is a space:

// 4 spaces:
[TB][TB]return Matrix(1, 2, 3,
[TB][TB]______________4, 5, 6,
[TB][TB]______________7, 8, 9);

// 8 spaces:
[-TAB--][-TAB--]return Matrix(1, 2, 3,
[-TAB--][-TAB--]______________4, 5, 6,
[-TAB--][-TAB--]______________7, 8, 9);

Getting this right is hard, especially since some editors (I’m looking at you, Visual Studio) like to re-indent things with tabs whenever they darn well feel like it.

Oh, and then there’s that whole semantic whitespace thing with Python. Can we please have our from __future__ import braces not throw a SyntaxError some time soon, Guido? Using tabs with Python is just asking for trouble. PEP 8 helps, but how many people actually read PEPs?

Well, I seem to have been rambling on about spaces and tabs on longer than I intended. Don’t mix spaces and tabs.

So the reason I started writing this post. Getting back to that. No really. I was editing a makefile the other day, obtained from a source which will remain nameless. Makefiles are notorious for being cryptic hard-to-maintain build systems filled with arcane syntax. Makefiles are so bad that there are a gazillion tools out there whose only purpose is to generate Makefiles: automake, cmake, qmake, Perforce Jam, and many more, though to be fair most of these tools are targeted at producing cross-platform software.

But let’s say you’re not using any of those fancy-pants tools because you’re a masochist, or more likely you’ve inherited a build system from a masochist. So you’ve got a makefile that you need to edit. I ran into this situation the other day. I fired up emacs, made my changes, and went to save it.

But something curious happened when I did so. emacs gave me the following message: “Suspicious line 50. Save anyways?” Its makefile-mode apparently installs a hook that checks your makefile’s syntax when you go to save it. Really handy. I wish other major modes did the same, although doing so may add an unacceptable delay to saving, particularly with languages with hairy syntax such as C++.

It turned out that line 50 had a mix of spaces and tabs indenting it. Hard tabs are meaningful in target specifications. In this case it didn’t end up mattering semantically, since it was in the middle of a multiline variable declaration, but I fixed it anyways to use spaces. I was also editing another makefile for which emacs gave a warning which had a blank line with spurious spaces. Again, not meaningful, but it was nice for the heads-up.

When editing a makefile in makefile-mode, emacs highlights trailing whitespace, but that’s not good enough for me. I happened to have a snippet of elisp code in my .emacs file that highlights all hard tabs. This made it especially easy to identify the problem in the offending line 50 in the makefile. I generally keep it on, but sometimes when I’m working with an all-tabs file it gets bothersome, so I turn it off. It’s amazing how code can have an awful mix of tabs and spaces all over the place as many different coders have touched it over time, each with their own preference for space and tabs.

Without further ado, the snippet. Just plop this baby in your .emacs file, and enjoy seeing your tabs burning bright on your screen:

; Draw tabs with the same color as trailing whitespace
(add-hook 'font-lock-mode-hook
  '(lambda ()
        '(("\t" 0 'trailing-whitespace prepend))

I found this snippet somewhere online, but I’ve lost the source. If you can point me to its source, I’ll gladly attribute it. And I just realized my syntax highlighter plugin doesn’t support any variant of Lisp. That’ll need to be fixed at some point, since there will likely be more elisp snippets posted here in the future. But until then, you’ll have to deal with broken syntax highlighting (currently highlighted as C++ code). At least there’s an even number of single quotes.

Update: Syntax highlight success! Source code for the highlighter can be found here.

Update 2/23/11: Plugin updated for Syntax Highlighter v3.x.

Comments Off on Spaces and Tabs

The Spoony Blog gets a facelift

June 14, 2009

I’m lazy. I’m also not a web programmer. Put 2 and 2 together, and you’ll infer that I have very little motivation to stylize this blog. But as both regular readers of this blog have probably figured out by now, I’ve gotten off my lazy ass and given this blog a new theme.

I recently stumbled across Charles Nicholson’s blog, liked the theme, and decided to shamelessly steal^H^H^H^H^H copy it. In adding the new theme, I’ve put in a few more tweaks as well. I updated to the latest WordPress version (currently 2.8.0), which has a much nicer dashboard. I threw away the useless calendar widget on the sidebar, and I’m now working on beautifying <code> tags.

Next up is installing a snazzy syntax highlighter. I really like the look of the Google syntax highlighter (also shamelessly stolen from Charles Nicholson), so I’m going to go with that. When I’m done with that, the following code snippet should be a little more than just monospaced text in the same color and style as regular text.

#include <stdio.h>

int main(int argc, char **argv)
    printf("Hello, world!\n");
    return 0;

Edit: Success!

Comments Off on The Spoony Blog gets a facelift

Ocarina of Time: Best. Game. Ever.

June 12, 2009

At least according to the users of GameFAQs. The Legend of Zelda: Ocarina of Time recently finished triumphing over Final Fantasy VII in the 128-game battle royale contest of Best. Game. Ever. 2009, which took place over the past two-plus months.

Each day, four games battled it out in the Poll of the Day, with the top two advancing in the bracket. In the final battle, Ocarina of Time edged out Final Fantasy VII. With the third- and fourth-place games from that match removed (also Zelda and Final Fantasy games, incidentally), Ocarina held onto its title in the bonus match, taking in 52.69% of the 155,838 votes cast.

GameFAQs has held some sort of tournament-style contest every year since 2002, with some years hosting multiple contests. There have been seven character battles, one game villains battle, two games battles, and one game series battle. The other games battle prior to this year took place in 2004. That battle was a 64-game 1-on-1 single-elimination tournament, in which Final Fantasy VII emerged victorious over Chrono Trigger in the finals, after defeating Ocarina of Time in the Division Finals [full bracket and stats] [bonus poll].

What does this turnabout mean for GameFAQs? I’d say it’s most likely a shift in user base demographics towards Nintendo. Nintendo games absolutely dominated the contest, what with a whopping four Mario games, four Zelda games, two Super Smash Bros. games, and a Pokémon game comprising 11 of the 16 final games, the other five being three Final Fantasy games, Resident Evil 4, and Metal Gear Solid 4.

Another surprises was Chrono Trigger getting narrowly defeated by Super Mario 64. There were also a few times where one game beat another in one round but then lost to it the next, such as Pokémon Red/Blue/Yellow and Metal Gear Solid, Final Fantasy X and Super Smash Bros. Melee, and Metal Gear Solid 4 and Super Smash Bros. Brawl. There were a lot of other results that differed greatly from something which happened in 2004, but I won’t continue boring you with statistics.

All in all, it was a fun contest with a number of surprises, but I’m starting to question the value of these contests. They were fun and exciting at first when they were completely unpredictable, but now they’re starting to get dull as people get better and better at understanding what GameFAQs visitors like. It was refreshing to have a games contest this year after not having had one for five years. I suspect, though, that the contests will continue so long as they continue to attract more visitors to the site. So what will see next year? I’m hoping to see something completely new and different, but odds are it’ll be Character Battle VIII, with some mildly interesting gimmick.

My personal Best. Game. Ever. is Final Fantasy VI. What’s yours?

Comments Off on Ocarina of Time: Best. Game. Ever.

Word-Fu is out!

April 11, 2009

Ok, Word-Fu has been out for over a month now, and I’ve been really lame about updating this blog. Word-Fu is an iPhone game developed jointly between Demiurge Studios and ngmoco:) that I was working on for a couple of months. The first release was on February 26, and the first update hit the iTunes App Store not too long ago. We’re now working an a second update, to include localizations into a few other languages (which is extra challenging for a word game).

So what is Word-Fu? Word-Fu is a Boggle-like spelling game, where you’re given a bunch of dice with letters on them, you shake them up with the iPhone’s accelerometers, and then touch them to spell words. Unlike Boggle, there is no constraint that consecutive letters must be adjacent—you can use the letters in any order. If you happen to get a particularly good set of letters (such as ACEIKORTS) and you know a lot of anagrams, you can just keep spelling more and more words. You get extra time for each word spelled, so you can keep playing for quite a long time and rack up your score.

You can even reuse letters in the same word, so if you get an A, an E, and an S, and you spell long words like ASSESSES with just those three dice. There were some discussions during development with very strong opinions regarding this feature—I was in favor of not being able to reuse letters—and we ended up supporting both modes of play. The mode where you can’t reuse letters is called Shaolin Style, and it can be turned on in the Options screen; it is disabled by default.

So if you’re a word game freak, what are you waiting for? Go check out Word-Fu in the iTunes App Store today! It’s currently $1.99, although there was a special promotional price of $0.99 for the first couple of weeks after the game launched.


Don’t steal Mac OS X

January 5, 2009

I was reading through the Mac OS X Internals book, and came across this interesting nugget in chapter 7. In OS X 10.4, a kernel extension maps an anti-piracy message into the 256 bytes of virtual address space of every running process at address 0xFFFF1600, which you can see for yourself by compiling and running the following code:

#include <stdio.h>
int main(void)
    printf("%.256s\n", (char *)0xFFFF1600);
    return 0;

The output:

Your karma check for today:
There once was was a user that whined
his existing OS was so blind,
he'd do better to pirate
an OS that ran great
but found his hardware declined.
Please don't steal Mac OS!
Really, that's way uncool.
   (C) Apple Computer, Inc.

Note that this only works in OS X 10.4, not in 10.5. I don’t know if it runs on older versions of OS X. I wonder if this message is also on iPods and iPhones.