Briefly following up on the Twitter situation from my last post, Michael Arrington of TechCrunch posts a short list of extremely pointed questions for Twitter. In it he says...
So I have a couple of questions, too, based on a couple of discussions I’ve had with people who say they’ve seen Twitter’s architecture.
- Is it true that you only have a single master MySQL server running replication to two slaves, and the architecture doesn’t auto-switch to a hot backup when the master goes down?
- Do you really have a grand total of three physical database machines that are POWERING ALL OF TWITTER?
- Is it true that the only way you can keep Twitter alive is to have somebody sit there and watch it constantly, and then manually switch databases over and re-build when one of the slaves fail?
- Is that why most of your major outages can be traced to periods of time when former Chief Architect/server watcher Blaine Cook was there to sit and monitor the system?
- Given the record-beating outages Twitter saw in May after Cook was dismissed, is anyone there capable of keeping Twitter live?
- How long will it be until you are able to undo the damage Cook has caused to Twitter and the community?
Now, to a certain extent this is Mr Arrington realizing what type of posts sell and taking advantage of that fact at a normally slow time (the weekend). But there are a couple of interesting issues here.
On the technical issue: If half of this list is true than Twitter's infrastructure was built by a complete amateur (and that's being generous). Or maybe more accurately a programmer who didn't understand the first thing about IT infrastructure. Whatever the case it shows a business with no idea where to turn when it came to building a solid infrastructure.
To my eyes this is endemic of an industry that has (a) allowed academia to abandon relevant experience and (b) refused to regulate itself with certifications.
Computer Science programs focus almost completely on theory while Information Systems degrees focus largely on business so both create candidates with huge gaps in their knowledge. Beyond that the only certifications that exist are from individual vendors and those are more marketing than they are verification of knowledge. So you get an industry where it's impossible to identify who knows what they are doing (unless you ARE someone who knows what they are doing)
Everyone from Doctors to Plumbers have professional certifications to verify their knowledge. That's why you don't need to be a Doctor to hire one. The fact that Twitter, one of the most scrutinized companies right now, has such a painfully insufficient setup suggests it might be time for the IT industry to look into ways to verify its members.
On the Blaine Cook issue: Since the first serious tech job I ever had there has been a constant in the industry. Companies always insisted on Non-Disclosure agreements and tech workers always hated them. Recently that policy has started to lift and Mr. Cook is just some of the fall out.
After seeing the damage he's done I can't help but wonder if Twitter isn't a lesson in why they are necessary. At least after termination (you could have someone agree to it at the time they are hired). Transparency is great until your ex-employees turn on you and once that happens they can do some serious damage to the company as a whole.
NDAs after termination keep your ex-employees from destroying the morale of your current ones.
Anyway, its about time for me to get ready to go out for the night but I wanted to throw out those two quick impressions before they got forgotten amidst other weekend activities.