Conversation with John Engates, Rackspace's CTO

Conversation with John Engates, Rackspace's CTO

Summary: Rackspace's PR firm contacted me and asked if I would be interested in speaking with Rackspace's CTO, John Engates, to get a technology consumer's view of the state of virtualization technology today.  I, of course, jumped at the chance since I usually speak with representatives of various vendors.

SHARE:
TOPICS: Data Centers
2

Rackspace's PR firm contacted me and asked if I would be interested in speaking with Rackspace's CTO, John Engates, to get a technology consumer's view of the state of virtualization technology today.  I, of course, jumped at the chance since I usually speak with representatives of various vendors.

During our brief conversation, I was impressed with the size of the IT environment that falls within John's area of responsibility at Rackspace. The company is managing roughly 30,000 servers hosted in 7 data centers around the world. The size of this environment would most certainly put management tools in general to the test and would overwhelm most management tools focused on managing virtualized environments.

John pointed out that Rackspace has evaluated most of the single-vendor and open source tools available today and found that they simply are not up to handling the scale of the Rackspace environment. Most of the management tools that are available today were designed to manage tens, hundreds, or, perhaps, thousands of servers. Tens of thousands of servers, on the other hand, would be beyond their scope.

So, John and his team have taken the approach of segmenting their environment and then using available tools to manage each segment. Rackspace developed tools are being used to orchestrate these tools. While speaking about the tools that Rackspace has built to orchestrate the tools offered by vendors, John commented how pleased with the management tools being offered by VMware. He commented that they were robust and well supported.

Although I wouldn't place myself in John's league and know that my friends will be surprised, I was happy to be able to add a few useful thoughts during the conversation. I was able to introduce John to tools offered by Cassatt, Hyperic, Novell and Virtual Iron. It appeared to me that one or more of these company's offerings might be useful in the Racksapce environment.

Although Rackspace's environment is different than that found in most large-company data centers today, it is very much like the data center these same large companies are moving towards. More and more of the time, large organizations are treating their data centers as internal "hosting companies." John welcomed a dialog with internal hosting company managers.

Thanks, John, for generously spending some time chatting with me.

How do today's management tools work in your organiztion's IT environment? Are they easy enough for your team to use effectively? Do they scale well enough for your organization? Are they reliable enough?

Topic: Data Centers

About

Daniel Kusnetzky, a reformed software engineer and product manager, founded Kusnetzky Group LLC in 2006. He's literally written the book on virtualization and often comments on cloud computing, mobility and systems software. In his spare time, he's also the managing partner of Lux Sonus LLC, an investment firm.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

2 comments
Log in or register to join the discussion
  • Bad Call

    John,

    This is what I sent Lanham with no replay,

    So munch has happened to RS and you have no details.

    Please call my cell so I can give you the whole story.

    Thanks,
    Kelly Galbraith
    Data Center Engineer
    626-676-1287 Mobile


    Lanham,

    This is truly a bad situation, I have been trying to
    contact you by phone with no luck. I need to send you
    this email so you can make your judgment call knowing
    all the details.

    I feel that the last two outages never needed to
    happen do to the fact that I have been trying to get a
    SWITCHGEAR maintenance contract signed with Shermco
    for the past year!.

    I have worked at Countrywide & EarthLink main data
    centers in California for the past nine years prior
    to coming to Rackspace and know how important it is to
    have such contracts. This contract gives all data
    centers around the world a piece of mind knowing that
    their SWITCHGEAR is working (calibrated Section 7.10.)
    correctly on a year to year basses with a
    certification sticker on every breaker. Do you know
    the last time this was calibrated? I do, it?s on all
    of your SWITCHREAR dating for renewal in 2002 telling
    me that we have gone SEVEN YEARS!! With no testing. I
    as well as Shermco (Mike Moor our rep) have made
    countless attempts to get this contract signed with no
    support from Neal Parker or Bob Mielke telling me it?s
    too disruptive and costly for us right now, the PDU?s
    failed because the main breakers feeding them were at
    half of their available set point to handle the load.
    When Neal had J&N install the PDU?s the breakers could
    not handle the additional load which was brought to
    his attrition prior to the install by Mike Moor
    (Shermco). Please see attachment above.

    As for the chillers, again I have made many attempts
    to run the test needed to find out why the chillers
    will not run on generators. I have had two set up
    maintenance windows shot down at the last minute by
    Neal Parker and Bob Mielke . If you need to verify any
    of this info please do so with my vendors Shermco and
    or my engineering crew. We have many issues at DFW and
    I would like to be a part of resolving them I just
    need to be herd.

    Please call me for more details,

    Thank You,

    Kelly Galbraith
    DFW Data Center Engineer
    Rackspace Managed Hosting
    801 Industrial Blvd. Ste 200
    Grapevine, Texas 76051
    (888) 850-0211 ext. 6154 Office
    (817) 919-4910 Mobile
    Kelly.Galbraith@Rackspace.com



    From: Lanham Napier [mailto:lnapier@rackspace.com]
    Sent: Wednesday, December 05, 2007 10:42 PM
    To: allstaff@lists.rackspace.com
    Subject: [ALL STAFF] Tough Times Shape Fanatics

    Hi Rackers,

    We are facing some big challenges lately. Over the
    past couple of weeks our critical infrastructure in
    the DFW data center has let us and our customers down.
    This is frustrating stuff. It drives us all nuts and
    creates a ton of pressure on Fanatical Support. The
    bad news is we are now in a position where we have to
    execute perfectly in order to protect our customers
    and business. We have worked together for years to
    build a reputation as an incredible service company.
    Unfortunately the events of the past couple of weeks
    have put this reputation in jeopardy. We have to
    execute at a higher level than we ever have before.

    For three years the DFW facility has performed
    exceptionally well for us. In the last couple of
    weeks we have had difficulty starting our chillers
    from generator power. It worked for three years, but
    the last two weeks we have struggled to get the
    chillers going from generator power. These struggles
    have led to the heat induced outages we have
    experienced. Obviously something changed on us and we
    have to figure out what it is.

    Our engineering teams are working hard to get DFW back
    in fighting shape. Here is what we?re doing:

    1. We are making changes to the organizational
    leadership at the facility. We are literally doubling
    the engineering capability at the facility.
    2. We have installed a new chiller plant. This plant
    runs independently from our existing chiller plant and
    provides us with an additional layer of redundancy at
    DFW.
    3. With this plant in place we will be able to do some
    rigorous testing to diagnose the problem to the
    systems. We are going to go through everything in
    great detail to be sure we turn over every rock. We
    have to do this perfectly and get it right
    4. Today we had Intel Data Center Services on site.
    Tomorrow Goetting & Associates will be on site. We
    are going to tap these and other experts to help us
    out with the project. At this point everything we do
    needs to be triple checked. We cannot afford a
    miscalculation on any of our operations, so we are
    going to have to go through every detail multiple
    times before we do anything.
    5. The experts are going to lead us through an audit
    of the facility. We are going to review all of the
    systems at the facility to get everything right. We
    will come out of this with a hardened asset to support
    our growth for a long time and we will get the DC
    certified so we can share the results with customers.
    6. We are also going to spend lots of time getting our
    process and procedures properly documented and in
    place at all of our DC?s. A world class experience is
    created through the addition of a world class facility
    and world class Rackers. We are working on the
    facility and we?ll use this opportunity to provide
    additional training and capability to our teams in all
    of our DC?s.

    Rackers we are in a critical phase of our company?s
    development. Our competitors are launching missiles
    at us every moment. They are trying to capitalize on
    our recent operational setbacks. We will hold our
    ground by taking care of our customers and supporting
    our teams through the upcoming tests and changes. I
    have spent lots of time visiting with our customers
    over the past couple of weeks. The vast majority of
    them want to stay with us, but we have to earn it. We
    have to execute perfectly. Please be mindful of this
    in everything you do. Let?s focus on quality. Let?s
    take the extra step to get everything perfect ? in our
    tickets, our conversations and our presentations.
    Let?s stand up and show how that when the going gets
    tough we get Fanatical. We can do it Rackers. Let me
    know how I can support you. I feel good about our
    plans for DFW. We are going to get it right.

    LN
    Setup
  • RE: Conversation with John Engates, Rackspace's CTO

    To whom it may concern:
    The reason for this written information is quite simple. Rack Space needs to realize the kind of company?s they have working and bidding their work. The company J & N Services and now J & N Electric Inc, has done a lot of work for your company. How ever J & N has not done the work. J & N sub contracts all the work to other persons and yet takes credit for all the good work performed and credits bad work to the contractors that does his work. J & N has also bided work for Rack Space and uses material that has already been purchased by Rack Space. In other words J & N sells you material you have already purchased. This was more visible at the Grapevine location.
    All the contracts performed at Grapevine, were suppose to include all material and labor. Well I hate to say it but guys got ripped off. First, J & N re used all the wire mold 600 (throft) and covers that were already installed under raised floor. But, his contracts included him purchasing this material !!!!!! Did you guys get a discount? Did you guys get any monetary compensation for him using and selling your material back to you? I would look deep into this issue. Also, the wire mold 600 wire way that belonged to Grapevine was also re sold to you guys in Austin and San Antonio!!!!!! Receptacles, covers, #10 AWG and category 5e cable already sold to you were re sold to you guys for use in Austin and San Antonio. When questioned about it, he simply said ?THIS IS HOW I STAY PROFITABLE?. I feel that this person is nothing but, ethics???? What do you think?
    Another issue that comes to mind, is ability to do work ? in a neat and workman like manner?. Let me explain. J & N ran a lot of category 5E cable to cross connect cabinets. This work does not look neat or clean, you guys can judge that for your self. Another issue that comes to mind is the fact that all the cable ran in your locations are non plenum rated cables. The fact of the matter is that plenum rated cable is not needed for all the over head work in the data center. How ever, a closer look at National Electrical Code reveals that all the cable ran under the raised floor, and above the dropped ceiling, (since it is considered a return air ceiling) is not plenum rated. The fire marshal would have a field day writing up citations!!!!! This issue is evident in the cables ran from the relay hub center to the D-mark room. This, D-mark extensions were done by J&N and US NET. At least US NET put the cable in conduit in the hall way before you get to the D-Mark room. All the cable ran along the hallway (which happens to be the same brand J&N used for the rest of the work) is thrown above the dropped ceiling. Non plenum rated cable used in a return air ceiling???? Now, that is a Master Electrician (as he calls him self) really knowing the electrical code and low voltage section of the National Electrical Code.
    Another issue that comes to mind is grounding. According to the N.E.C all metal cabinets and junction boxes shall be grounded and bonded. In this case most of the electrical wire-ways are not properly grounded and bonded. None of the cabinets are grounded. None of the relay racks are grounded or bonded. None of the above ladder rack are grounded or bonded. This is a big issue!!! For you guys. Before, 2006 all the electrical was ran in flexible conduit. Green field conduit if you will. Well, this is ok as long as you have a ground running inside the flexible conduit. Well J & N did not run a ground inside a lot of the flexible conduit they ran!!! When he was confronted with this question, his reply was ?they have a ground running through out the whole floor and I rather use that ground than provide my own?.
    This is really a violation of the National Electrical Code. The proper way of running those circuits was running them in flexible conduit with a ground jumper. Grounding the jumper at the electrical panel and grounding the electrical wire-way. Then ,running a jumper, from the electrical wire-way to the grounding wire that grounds the raised floor. This would provide a proper ground. The second thing in grounding that was never done, the cabinets, relay racks, and ladder rack were never grounded.
    J & N lack of professionalism is evident pretty much every were he does work. In the Grapevine electrical room this is very evident. When the FORESEER system was installed, no new conduit was ran. Low voltage cable (signal cable) was run in the same race way as electrical wires. If you look at the electrical gutters close to the ceiling you will see the lack of professionalism. On a lot of locations, he ran power into the gutter with out using a knock out, he removed the cover of the gutter to get his power. This is clearly boot legging!!!!!! When confronted with it, he said ?IM TRYING TO STAY PROFITABLE?. Another way he stayed profitable, was to use Rack Space employee and material to run cables in San Antonio (AT THE TWO LOCATIONS IN SAN ANTONIO).
    I honestly believe that the engineers in charge (Bob Mielke and Neal Parker) looked the other way when this work was done because; he had something to gain from him reselling your own material to you. The engineers were present when material was taken. Also, J&N knew for a fact that he was going to keep working at Rack Space. When Mr. Duhan asked him to lower his bids on several phases, J&N refused. J&N mentioned ?as long as I keep him happy, (Neal Parker) we will always be here?. J&N would not say how he kept the engineer (Neal Parker) happy. But, the issue that arose in 2007 gives every body a pretty good answer. This would also explain the reason why J&N laughed at Mr. Duhan when asked to lower his bid. All this from a person who claims to be a good, God fearing Christian?.
    These are only a few examples of the ?dirty way? of doing business. The sad, part, is most of the Rackspace employees (M. Duhan, K. Galbraith and Clark) tried to stop it but Neal Parker always had the last say. It wasn?t until one of your employees found out that Jessie with J&N was dealing drugs through Rackspace to Neal Parker that something had to happen.?. Makes people wonder what kind of favor he was providing for the Rackspace Management????
    Setup