How Are You Staffing Your Startup?

I have, in the past, worked for startups of varying forms. I worked for a spinoff that ultimately failed but had the most awesome product I’ve ever seen (neural networks were involved, need I say more?), I helped a buddy very early on with his startup, which did great until angel investors crept in, destroyed his vision, and failed completely to understand the Long Tail vision my buddy was trying to achieve, and I worked for a web 2.0 startup which was pretty successful, and was subsequently purchased… by another startup!

Working in academia for 6 years also exposed me to people who are firing up businesses, or projects that accidentally become businesses, and some of those go nowhere, while others seem to be on the verge of NYSE listing now, while a year ago they were housed in the smallest office I’ve ever seen, using lawn furniture for their workstations.

Of course, I’ve also consulted for, and been interviewed by, a host of other startups – recently, even.

First, the bad news

The bad news is that most or all of these startups are headed by developers, and they have applied *only* dev-centric thinking to their startup. They’ve thought about how to solve all of the app-level issues, mapped out use cases, drawn up interfaces, hacked together prototypes, and done all kinds of app-level work. Then, they’ve hired more developers. Then more after that.

Some seem to have given almost zero consideration to the fact that their application might become successful, and its availability might become quite critical. They haven’t given much thought to things like backups or disaster recovery. They have no plan for how to deploy their application such that when it comes time to scale, it has some hope of doing so without large amounts of downtime, or huge retooling efforts.

They’ve also given very little thought to how to enable their workforce to communicate, access their applications and data remotely without huge security compromises, and generally provide the back end system services necessary to run a business effectively (though, admittedly, most startups don’t require much in the way of things like NFS, or even internally-hosted email in the very beginning).

In short, they’ve either assumed that systems folks’ jobs are so easy that it can be handled by the developers, or they think that scalability lives entirely in their code, or they’re just not thinking about system-level aspects of their application at all. And don’t even get me started about the databases I’ve seen.

I know of more than one startup, right now, months late in going live. None of them, at the time I spoke to them, had a systems person on staff, or a deployment plan that addressed things that a systems person would address. What they had was a lot of developers and a deadline. Epic fail.Yes, even if you use agile development methodologies.

The Good

The good news is that, while some companies hire no systems folks at all and flounder around system-related issues forever, others hire at least one or two good systems folks, and make them a part of a collaborative effort to solve systems problems in interesting ways, utilizing the knowledge and experience of both systems and development personnel to create truly unique solutions. When sysadmins and developers collaborate to solve these issues, I have learned that they can create things that will blow your mind (in a good way).

In fact, Tom Limoncelli wrote recently that systems administration needs more PhDs. Well, I suppose that would help, but I think we’d get really far, really fast, if we could just break down some of the walls between sysadmins and developers, give them a common goal, and let them hash it out. Sysadmins have an understanding of the system and network-related issues that developers aren’t likely to have. Developers, in most cases, can probably write better code, much more quickly, than a sysadmin. Developers and sysadmins working together, sharing their knowledge and communicating with each other, can solve systems problems in new, unique, creative, and very effective ways.

The End

In the end, issues facing startups now blur the line between development and system administration a bit more than in the past. There are problems that need solving for which there is no RPM or Deb package. These problems require some knowledge of how related or analogous problems have been solved in the past. A knowledge of the systems and development approaches that have worked, and why. Enough experience to have seen where and when things go bad, and why. It also requires creative and critical thinking. I think that good, senior systems and development people have these skills, and much more.

For whatever reason, it seems that the only time these two camps ever meet is on opposite sides of a deployment or application support problem. Perhaps this happens with enough frequency for people to think that the two camps can’t, or won’t, work together. They can, and they will. People with a love for technology will come together if the common goal involves furthering technology and its use, regardless of their background. Sure, it takes proper management like any other project, but it can be done.

If you’ve had experiences, good or bad, with dev/sysadmin collaboration, I’d love to hear your stories!

  • Chuck

    In college I worked as a systems administrator; post-college I went into game programming.

    I have a respect for both jobs, and having programming skills helped me tremendously in my college job. At this point in time though, since I’m now a developer by age, I could definitely appreciate working together with the sysadmin to make both of our lives easier.

  • Manuel Padilha

    I worked for 3 years as a sysadmin, regularly meeting developers “on opposite sides of a deployment or application support problem”. It wasn’t pretty.

    For the past year an a half I’ve joined a development team as a developer. It sure opened up my horizons. But I can say the opposite is also true, having me on board said team managed to tackle things like multi-plataform support, multi-threading, caching and network performance optimization, which would otherwise remain unexplored areas.

    Excellent post by the way.

  • Arjen Lentz

    In a small company, skill diversity is a great asset.
    In an organic-growth company, one could even say it’s a necessity.
    There are some downsides to skill diversity, but as long as you’re aware of it, I think it can work out fine.
    As the company grows, specialisation is something that tends to naturally happen; managing that in a smart way can again make things work better.

  • m0j0

    In a small company or department, I think skill diversity is a requirement, and I think these environments run more smoothly when there is skill diversity, precisely because there is some overlap in the skill sets of the staff there, which makes a lot of things easier both from a technical and managerial perspective. I believe I do point out that things will work in this mode… to a point. And I begin to define where that point occurs.

    I suppose I should note again that the environment I’m speaking to here is the recent startup environment. In my experience in dealing with relatively recent startups, the ones I’ve seen do well are thinking about systems-related issues up front, and have a systems person on staff (who probably also does a bit of coding or, in my case, database work). The ones who are floundering seem to be unaware that there are systems issues to address, or that there’s planning that can be done and precautions that can be taken in their code to insure that the application will easily scale when systems scaling maneuvers are performed later.

    Skill diversity only goes so far. The idea that a developer who has “seen a system or two” has the experience to foresee, plan for, and conquer even relatively short term (in web 2.0 time) scaling issues is almost certainly hubris. Perhaps not coincidentally, it’s also right in line with a lot of things I’ve heard people at startups say. I’ve mostly kept my mouth shut about it rather than confront people with big degrees and even bigger egos. But now, when I go to sites where there’s still the (completely stupid) ‘stealth-mode’ splash page up, months after their planned date for going live, I begin to think perhaps my own theory that skill diversity doesn’t get you all the way there, and systems people really can improve the outlook for a startup weren’t just selfishly-motivated wishings.

    I can’t remember if it was someone from flickr or livejournal who once said in a talk that, if they could do it all over again, they’d put more thought into the systems end of things. I would imagine that, somewhere, someone from twitter is uttering something similar right about now. 🙂

  • Nes

    The barrier between systems/database and developers happens in big companies often because the systems people have to handle stable production environments whereas developers are pushing the envelope in new projects and the production people are not willing to be more agile and flexible to try new things. There are usually checks and balances in place to avoid screwups in productive environments (for example multiple signoffs) that slow down development.

  • Grant Austin

    I used to work at a small software development company. One of my friends and coworkers there was a really amazing sysadmin. He was always working hard so he could get into development, even though he was way more valuable to the company in his current capacity. I think his reasons basically boiled down to two things: Most people see system administration as lower prestige than software development and, at small companies anyway, system administrators get paid significantly less than developers.