Ed's Blog

A PhD Student's Musings

C Integers Are Tricky

I’ve had a lot going on lately, so sorry for the lack of posts.

Today I’d like to post about an interesting security vulnerability that was just discovered in MySQL. The problem is extremely serious: by simply attempting to login ~256 times, the user gains access. From a practical point of view, it’s amazing that such a serious vulnerability is present in such a popular software package. It’s even more astounding when you consider how simply the “exploit” is.

So, here’s the vulnerable C code:

1
2
3
4
5
6
7
my_bool
check_scramble(const char *scramble_arg, const char *message,
               const uint8 *hash_stage2)
{
...
  return memcmp(hash_stage2, hash_stage2_reassured, SHA1_HASH_SIZE);
}

Did you spot the bug? Here’s another hint, the fixed code:

1
2
3
4
5
6
7
8
9
#define test(a)         ((a) ? 1 : 0)

my_bool
check_scramble(const char *scramble_arg, const char *message,
               const uint8 *hash_stage2)
{
...
  return test(memcmp(hash_stage2, hash_stage2_reassured, SHA1_HASH_SIZE));
}

What’s going on is that C does not have a native bool type, and so MySQL uses char. The return value of memcmp, which is non-zero for non-matching comparisons, is cast using truncation to a char. For example, a return value of 0x100 would be truncated to 0, which is interpreted as a successful comparison. Oops! The real problem is that bools do not behave like integers, and as programmers we expect a boolean cast (0 maps to false, anything else maps to true), which C doesn’t provide. I found this bug very interesting, because at first glance it looked correct. Another argument for program analysis and verification.

If you found this interesting, you might enjoy this quiz on C integers. I only got 88%. (I guess you can tell I work on binaries!)

Comments