It's the Infrastructure, Stupid!

It's the Infrastructure, Stupid!

Summary: Companies need to invest in the future of their infrastructure, or there won't be a future for them.

SHARE:
TOPICS: Servers, Hardware
15

The System, by Rosscott

This past week we saw major outages on popular social media websites Facebook and FourSquare. FourSquare had two multi-hour outages on consecutive days, with the explanation that the servers were being upgraded to handle the excessive load being placed on them. As I write this, Foursquare is in the process of another outage, with a message on their website saying that the servers are being upgraded.

This is actually fairly common in startups. The company founders start out very small, build a product and get it running quickly to show investors so they can afford to build it out further. The problem is that many companies never get out of the mindset that it's okay to just make updates and have multi-hour outages after they've reached a level of success and popularity.

Or as I like to call it, "amateur hour".

Quite often infrastructure is overlooked as a critical aspect of running an online service or business. All too often company founders aren't technically savvy, and have no idea what an IT department really does. They invest in developers but overlook system and network administrators often to their detriment. The bean counters that control the company budget give short shrift to the IT team that's in charge of keeping your business online.

Sometimes an IT department is completely overlooked, and developers are expected to handle the sysadmin tasks; quite often they are unfamiliar with these responsibilities and underqualified to handle them.

This lack of attention to infrastructure results in underpowered, under-funded, and downtime-prone networks that are not up to the task of keeping an online business viable. Social media sites like Twitter, Facebook and Foursquare are perfect examples of what not to do when ramping up your business.

The two most egregious mistakes an online service can make are: 1) not planning for future growth; 2) Making substantial system changes during the busiest hours of the day. When you're trying to build up business relationships with other companies to generate revenue, and want to look good to your investors so they keep funnelling cash to you to grow, you don't take your site down for 8 hours during normal business operations to upgrade your servers without notice.

This bears repeating: DO NOT TAKE YOUR SERVERS DOWN DURING BUSINESS HOURS.

The only reason there should be downtime during business hours is due to an unforseeable emergency. If you need to upgrade your servers, do it in the off-hours. After 7pm at night in the busiest part of the world (usually the US). On weekends. Plan it in advance, and give your customers/users plenty of lead time to know that it's coming so they can plan accordingly. If you provide a new service that people have come to depend on, and you take it down for the entire work day without warning, it makes your company look like a bunch of incompetent idiots that have no idea what they're doing.

An even better idea would be to analyze the amount of growth you get right in the beginning and ramp up your infrastructure accordingly before a sudden surge in traffic finds you unable to meet the demand.

You know those parties you throw at trade shows and give away thousands of dollars worth of free swag? Stop doing that until your business is stable. How many dotcom-era T-shirts from long vanished companies are sitting in your clothes bureau right now? Spending money on good will and marketing is fine, but doing it at the expense of actually running a viable service is not. It's a good way to drain all of your VC money and have absolutely nothing to show for it.

The key ingredient in heading off a situation like this is investing in hardware, networking, and the people to support it the moment you get your first round of funding. Hire at least one system administrator, a generalist who can build out your beginning server farm and network connectivity with an eye on future growth.

Pinning your company's future on a handful of desktop computers with no backups and no failover shows a complete lack of foresight. If you can't host it yourself, rent virtualized servers in a datacenter and let their infrastructure be your backup. Usually with a service like that you can simply rent more virtual servers and bandwidth as needed.

Your IT group is essential to your online service. If there's too much for them to do and not enough people to handle it, it's probably time to expand the group. If your service is successful, your revenue will pay for it. A successful, stable service is looked upon favorably by investors and by the companies that want to do business with you. If your service is unreliable because you skimped on infrastructure, they will see you and your company as unreliable and untrustworthy.

I personally have worked at companies where they invested early in hardware and IT, and had good practices for rolling out software and hardware changes. They were structured and organized and they continue to be successful (or got bought out by bigger, more successful companies) to this day. Downtime almost never occurred unless it was planned and during a maintenance window scheduled ahead of time.

I have also worked at companies where the infrastructure was slapdash, changes were pushed out without warning in the middle of the day, often bringing the site tumbling down. Infrastructure was only expanded when the outages due to overload would have the site down more often than it was up. This kind of thinking was reflected in the company office itself: disorganized, lazy, uncaring, and quite often a mess.

Then there are the companies that have a messy infrastructure, but are willing to change. I've worked at those, too, and was glad to be part of the transition from sloppy to successful. There's a great sense of satisfaction from bringing a network infrastructure out of a tangled mess of wires and hodge-podge hardware and into a streamlined, organized operation that's easier to manage and maintain.

"Investing in the future" may sound like a cliche aphorism, but when taking into consideration how large an effect it can have on a company's network infrastructure, it doesn't sound so silly at the end of the day.

Topics: Servers, Hardware

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

15 comments
Log in or register to join the discussion
  • RE: It's the Infrastructure, Stupid!

    Quote:

    `The bean counters that control the company budget give short shrift to the IT team that?s in charge of keeping your business online.`

    Tell us, in IT, something we DON'T already know!!

    Do the preaching to the `C`level ID10Ts that are more concerned with the size of their year end bonus, then keeping the lights on!
    fatman65535
  • RE: It's the Infrastructure, Stupid!

    both your article and fatman65535 make valid points. Business does undervalue IT too often and since the upper management and accounting don't understand the technology they can quick dismiss it. This sets up an unfortunate perspective that the IT personnel are just complaining and that their requests are not important. The COO (corporate operations officer) and TOO (Technical Operations Officer) are two positions that are supposed to provide the bridge between the 'bean counters - upper management group and the Technical group. Sometimes these positions are not in a business organization and that leave a gap between these groups. The facts are that IT must have a voice and an 'interpreter' to the management level to avoid these kinds of problems.
    Joe Perkins
    • RE: It's the Infrastructure, Stupid!

      @Joe Perkins <br><br>I agree joe. and since most SMB do not have a COO/TOO. I have this issue in my current position as an IT manager for a small business 38 total employees. We have no COO TOO our CFO took over the CEO position when the previous CEO was fired. I am always told their is no money i have no budget left to cut anymore.
      MLHACK
      • You are the COO/TOO

        @MLHACK
        Surprise! You just got a promotion. If those roles don't exist because of the size of the company, then you are those roles. Just add to your list of responsibilities:
        Prepare ROI presentation for Quarterly Business Review with CEO
        Create Business/IT alignment strategy
        Develop business enablement roadmap for IT
        Prepare business justification case for IT spending
        hawks5999
    • RE: It's the Infrastructure, Stupid!

      @Joe Perkins

      The flip side is, IT departments are HORRIBLE at producing the results they promise and business is tired of throwing money down a rat hole. Well that and the "latest thing" syndrom has gotten very, very old.
      No_Ax_to_Grind
      • RE: It's the Infrastructure, Stupid!

        @No_Ax_to_Grind <br><br>Are you kidding? So you prefer to continue to run old systems and old software, and spend countless IT man hours putting scotch tape on problems? IT departments don't produce results because management tends to have their heads up their asses... rather bend over a nickel to pick up a penny.
        ]AR[*tREMor
      • RE: It's the Infrastructure, Stupid!

        ]AR[*tREMor

        Are YOU kidding? No_Ax is dead on, especially in the SMB space.
        Scotch tape is pretty cheap, even if it takes a bunch of IT hours to apply it compared to:
        Software licensing for upgrades
        Implementation costs
        Data migration costs
        Integration costs
        **Employee Retraining costs**
        **Lost productivity during employee training and acclimation**
        Those **'d items typically don't register with IT folks.
        IT people are a pretty fixed cost. You cost $x/hr or $x/yr. If you have to work y more hours it's a simple x*y equation within a pretty standard range. And you can easily be told "Go home and we'll pick this up tomorrow".
        If an upgrade to the latest and greatest doesn't produce a guaranteed z% of either productivity improvement (cost cutting) or revenue improvement then why take the risk of breaking the system that currently works even with some extra x*y IT dollars.
        As No_Ax says IT doesn't have a great track record for delivering on promised improvements. cf % of IT projects that fail. cf Google in Los Angeles. cf Deloitte/SAP in Marin County
        hawks5999
      • RE: It's the Infrastructure, Stupid!

        @No_Ax_to_Grind
        Management has got to stop thinking that IT infrastructure is an art form, in spite of what the cowboys in development tell them, and start thinking of it as engineering. Better yet, plumbing. You get it right and nobody notices, you get it wrong and it stinks. There are well documented ways to do it right. Hire somebody that knows and do what they say.
        lars626
  • RE: It's the Infrastructure, Stupid!

    Man I sure am Hungry this morning. Want to see if my ration of water and Soylent Green are available this morning
    X41
  • Silly statement of the day...

    This bears repeating: DO NOT TAKE YOUR SERVERS DOWN DURING BUSINESS HOURS.

    Silliest statement of the day. I guess the author doesn't realize there are 24 time zones around the world and the world dosen't just surf their own time zones.
    No_Ax_to_Grind
    • RE: It's the Infrastructure, Stupid!

      Any company with decent staff shouldn't have to lose any face time at all.. ever... hasn't anyone ever heard of mirroring, or using production servers and test servers... there is no reason to ever go offline for a real upgrade or migration... when someone DOES go offline, because of a hack or a malfunction.. the report it as "An Upgrade" so as to not lose customer faith.. seriously.. do you want to go to a site and read.. "Our system is temporarily offline because we were hacked and/or our software or hardware failed".
      ]AR[*tREMor
    • What's a Business Hour?

      @No_Ax_to_Grind
      I had to stop reading the article when I scanned and caught the Bold admonition about "Business Hours".
      When you start off talking about Facebook and Foursquare and talk about web software startups, there is no such thing as Business Hours.
      When does Facebook not have business hours? When the USA sleeps (whenever that may be)? when China goes to work? when Eurasia is taking a nap?
      What should Foursquare call non-business hours? 9a-9p when retail is the heaviest? 5P-4a when the bars and nightclubs are checking in? 4a-Noon when breakfast and lunch diners are giving mayors free coffee? And forget accounting for EST, PDT or GMT.
      When I was running IT for a software startup we had onshore resource working 12-16 hours/day 7 days/week who would still be working when the offshore resources came on for their 12-16 hour shifts. They would still be working when the onshore resources came on. And running on 2 week sprints with committed features tied to revenue linked deadlines meant no slack for downtime.
      So I don't know what this mythical Business Hour is you speak of.
      hawks5999
      • RE: It's the Infrastructure, Stupid!

        @hawks5999 Sir, I think that would really depend highly on how many employees you want to have, their location versus location of their job and of course state laws which demand certainties for weekly hourly wage law regarding total hours worked per week before overtime could be the object. Since many are unemployed The Foundation which converts a consumer's auto to electric would be the consideration since there are two facets (1) manufacturing & distribution (2) conversion so I decide that for most situations a six hour per day might be best per employee at these facets providing a 4 shifts and that makes for a 24/7 operation. Education might not be so grand of a challenge when hands on work experience is the better option. But since Obama made consumer's the lowest beneficiary on the table there is no Foundation and banks are taking the jobs with their automated banking services which entrails payment processing with accounting and which drivews up the cost of estate taxation with high priced corporate clunkers that have low range and multi-charge potentiality per day of use when electric transport is a serious computing problem as we can imagine how a Infrastructure would be assembled for every state to endorse. How's your opinion on this consumer opportunity, Obama missed the point.
        gracioustinker
    • RE: It's the Infrastructure, Stupid!

      @No_Ax_to_Grind Can't be to silly, transistors made of silicon have a serious overheating problem while the future change is transistors made of graphene (pure carbon). Even Ultracapacitors can be improved by employing graphene in the as nanotube paths for single electron paths because each time a electron bounces without known bounce stats it is chaos and that is where overheating begins in all components. Infrastructuring will begin all over from IBM research on graphene product development.
      gracioustinker
  • You forget.. accountants don't possess brains, they are incapable of logic

    since all businesses are run by accountants, and there isn't a accountant on the planet with a brain, amateur hour will continue forever.
    Reality Bites