Being a software security researcher in academia, I spend a fair amount of time talking to non-academic security researchers. These conversations are extremely valuable: academics should be aware of the problems that industry perceives as important. Yet, these conversations have also shown that industry researchers hold some fundamental misconceptions about the nature of academic security research.
The research you see is “new”
Most people have never submitted a paper to an academic conference, and naturally have no idea what the timeline is like. Let’s take a look at the call for papers (CFP) at one of the highest regarded academic security conferences, IEEE Security and Privacy (which is also called “Oakland”).
Research papers were due on November 16th, 2011. Ideally, the idea and proof of concept for research is completed at least a month before the due date, to allow enough time for writing and experiments. So, let’s say the research is from about mid-October. Based on my personal experience, it can easily take six months to produce research for a paper, so we’ll say that research started in mid-April, 2011. The Oakland 2012 conference actually met in May 2012. So the research presented there was an entire year old! This also assumes that the paper was accepted in the first conference it was submitted to. By the time you see and hear about this work, the researchers have probably started writing their next paper, or have moved on to another topic.
This is one of the reasons that academia seems to struggle to keep pace with industry. It’s not that we are really behind, but it takes a long time for results to be disseminated. As a real world example, our group was criticized by some industry researchers when we released our paper on Automatic Exploit Generation for not handling defenses like DEP. At the time that AEG was published, I was already working on a separate paper on Exploit Hardening, or how to transform exploits so that they bypass defenses.
Academic prototypes are supposed to solve industry problems
If you’ve been in the security community long enough, you’ve probably heard at least one person complain that an academic paper did not release their prototype, or that a released prototype did not solve industrial problems (e.g., it sucked).
Industry focuses on tangible products – after all, that is why it is industry! So, it is unsurprising that in industrial circles, prototypes are often more important than the ideas employed in them. For example, there are dozens and dozens of fuzzers available, and yet there are few novel ideas employed in each of them. It is natural for industry researchers to transfer their expectations to academia as well: if this group’s prototype is so crappy, what use is the paper?
Unlike industry, academia values ideas over implementations. Here is a quote from the Oakland Call For Papers (CFP), which describes the type of papers they solicit:
Since 1980, the IEEE Symposium on Security and Privacy has been the premier forum for computer security research, presenting the latest developments and bringing together researchers and practitioners. We solicit previously unpublished papers offering novel research contributions in any aspect of computer security or privacy. Papers may present advances in the theory, design, implementation, analysis, verification, or empirical evaluation of secure systems.
Note the phrase novel research contributions. To write an academic paper, you must discover something new. Also note that nothing is mentioned about a prototype in the CFP. Academic prototypes usually support the novel contribution that is the basis of the paper, whereas in industry the prototype often is the contribution. Even academic systems papers must have a novel contribution, for instance a new design technique that enabled the system.
While the ideas and contributions of an academic paper may be widely applicable, the prototypes are often of limited use outside of the paper’s experiments. These prototypes are often written or modified on tight deadlines, and the code is often unmaintainable (or even unusable!) by anyone other than the creator. It’s usually not just a matter of releasing the code; researchers would need to spend significant time to clean and fix prototypes.
I have yet to hear a reasonable argument why academics should spend their time to create stable prototypes for industry. Is industry going to pay us for our time? Are they going to give us stock options to share the profits? No, of course not. It’s industry’s job to productize research. On the other hand, it’s academia’s responsibility to undertake more risky ideas and directions that may or may not lead to industry improvements. Because academia is not profit driven, we can think more outside the box, which leads to larger discoveries.
Academics do not understand exploits
Many industry researchers think that because academics target stack-based buffer overflows, and do not consider defenses, that this constitutes our entire knowledge of exploits.
Academics keeps their jobs by writing many papers. In general, it is preferable to write three incremental papers than one super novel paper that contains ten years of research. There’s a saying in academia that “Deans can’t read, but they can count”, meaning that deans do not understand the significance of papers, but can count how many of them are published in reputable conferences.
Academic researchers that value their jobs target a piece of a larger problem that is significant enough to be accepted by a reputable conference. For example, research in automatic exploitation is very new, and so it makes a lot more sense to target the low hanging fruit first: stack exploits, small or medium sized programs, and limited or no defenses. We will get to the harder cases in time.
These incremental papers are accepted because they contain novel research contributions. In other words, there is enough information there to raise the bar on public knowledge each time. Industry may not be able to convert these papers to products without more research, but that is not the goal of academic research. At the end of the day, we academics want to understand the world a little more.