Deep Fried Debugging

I was listening to the current episode of Deep Fried Bytes and was reminded of an important lesson. In case you haven’t heard of it, Deep Fried Bytes is a relatively new but very good development podcast. I highly recommend the podcast, it’s become a favorite on my Zune.

The hosts, Keith and Woody were interviewing members of the Microsoft.com support team. Yes, the guys who keep the actual Microsoft.com website up and running. Keith Woody asked them about a really challenging problem they hand, and one of the team recounted the tale of a site that had been in production about a year, when performance suddenly tanked. Naturally they went through the standard debugging questions, including “has anything changed in the code?” Since nothing had, they said “oh, well can’t possibly be the code” and went on to look at other things.

They went on to look at other things before finally, in desperation, coming back to the code. It turned out there was a scalability bug that had been there since day one, buried deep in a stored procedure. The select statement inside the stored proc caused a table scan. Not so bad when there were few records but after being up for a year the number of records was bogging down the stored proc.

I have been on many projects where a developer insisted the bug couldn’t possibly be in the code as it’s been running “perfect” and no recent changes have been made. The lesson to learn is never to rule out anything when looking for bugs. True, you should start with the most likely suspects, if no changes have been made to code then the probabilities of it being code are low as compared to say a hardware issue, but don’t rule it out completely. Get the entire team working in parallel. Let the developers look at the code, the DBAs at the database, admins at the server and network, and so on. Through teamwork, and being open to all possibilities you can achieve some deep fried debugging.