My Sunday Afternoon: Fun with VMWare ESX 3i

My Sunday Afternoon: Fun with VMWare ESX 3i

Summary: As soon as I heard the announcement from  bright and bushy-tailed VMWare CEO Paul Maritz that the company was shortly going to be releasing their ESX 3i enterprise-class embedded hypervisor for free, I was absolutely thrilled. Given the pace of technology reporting, I knew that I had to get it running on my test equipment right away.

SHARE:
TOPICS: VMware
24
As soon as I heard the announcement from  bright and bushy-tailed VMWare CEO Paul Maritz that the company was shortly going to be releasing their ESX 3i enterprise-class embedded hypervisor for free, I was absolutely thrilled. Given the pace of technology reporting, I knew that I had to get it running on my test equipment right away. Click on the "Read the rest of this entry" link below for more. What I didn't realize, however, what a pain in the neck it was going to be getting it to run on commodity PC and Server hardware. Oh, I knew from my previous experiences designing enterprise VMWare clusters for corporate clients that you needed enterprise-class hardware that was at least certified for use with ESX, but I decided to throw caution to the wind and try to get it running on the stuff I had lying around. I eventually did get it to work, and now that it's working, it works great, but not without more than a few headaches. Currently, the biggest limitation with ESX Server 3i and its big brother, ESX 3.5 is that due to VMWare's "closed kimono" stance on software development, the storage drivers are way behind in terms of hardware support when compared with competitive solutions from Microsoft, Citrix and the various Linux distributions/vendors. There isn't a single system, whether it be a PC or an entry-level server which use embedded SATA2 RAID chipsets that will work with ESX 3i or ESX 3.5 out of the box. If you boot ESX 3i on any of these systems, you'll be presented with a  message that no suitable storage device has been found. Doh! Game Over, man. EDIT 7-29-08: It appears there is a small list of of SATA-based servers and mainboards that will "work" but are not on VMWare's ESX 3i HCLs. Your Mileage May Vary. Typically, ESX 3i and ESX 3.5 are used on enterprise-class systems which support boot-from-SAN host bus adapters (HBA) , UltraSCSI and SAS drives. But if you want to piece together an ESX 3i system on a budget, you'll need to do a deep dive on VMWare's I/O compatibility  and systems compatibility list. However, I'm going to save you some time and trouble right off the bat -- if you're doing it on a budget, you'll want to make sure you have a PC or Server that can accommodate a supported PCI-Express SAS controller. A good candidate for this is the Adaptec 2405 series, which streets for about $200-$250. The base-level Adaptec 2405 can handle up to 4 SAS or SATA2 disks and up to 128 drives with SAS expanders, so it's not a huge investment if your mainboard can already handle PCI-Express and you have some cheap SATA2 storage lying around. I also tried AMCC/3Ware's new 9690SA, another PCI-X SAS/SATA RAID controller designed for higher performance applications, which streets for about $300 and is listed as supported for the more expensive ESX 3.5 product, but it doesn't yet work on ESX 3i as of the July 25 ISO currently available for download from VMWare's site. Given 3ware's excellent multi-platform OS support, we expect this to change shortly.  Be it as it may, the AMCC 3Ware does work phenomenally well on Linux-based hypervisor solutions such as ProxMox, Red Hat oVirt and Xen, and absolutely swimmingly on Windows Server 2008 with Hyper-V, so don't discount this excellent SAS/SATA controller value if you're not going ESX 3i. For network interfaces, you'll want to make sure you are using Broadcom, Intel or nVidia nForce chipsets. There are a number of other supported NIC chipsets, but you'll be safe with any of these, and are quite common. esx-bootscreen1.png Once you have your hardware ducks in a row, getting ESXi installed is a breeze. The software is distributed on a 250MB ISO file, which can be burned to a CD or bootable USB key. The actual installation is totally automated using a text-based interface (no mouse needed) and takes under five minutes -- once ESX 3i loads its drivers, it attempts to connect to your network via DHCP, auto configures an IP address on the first detected interface, and displays the address on the screen for you to connect to. Once ESX 3i is running, you then test its connectivity using a web browser on a client Windows-based machine (XP, Vista or Server) and you can download the VMWare Infrastructure Client 2.5 over the web. Unlike VMWare's other free virtualization solution, VMWare Server 2.0 ESX 3i doesn't currently support VMWare Virtual Infrastructure Web Access, so management is limited to Windows systems only. vmware1c.jpg

Click Here for a video tour of VMWare Virtual Infrastructure 2.5 Client managing ESX 3i.

For those of you already familiar with version 2.5 of the Virtual Infrastructure client for Windows, there's nothing new here. It's the same, polished, sophisticated interface that you get with the high-end VMWare products. There is a "gotcha" here, however. Without purchasing a license to VirtualCenter, VMotion and HA/DRS (in which case you are getting into the thousands of dollars) you won't be able to use any of the advanced features such as live migration, clustering and high availability. However, if you are standing up ESX 3i in a lab or testing environment, you will be able to prototype on the same exact platform that you will run in a VMWare enterprise production environment -- no more testing on VMWare Workstation or VMWare Server, and porting the images over. While not Open Source, this is a big step forward for VMWare as a company and its worthy of my applause. Still, even with the free release of ESX 3i, VMWare is going to face many challenges. Hyper-V and KVM will be significant players in the future, and are already poised to be the higher performance alternatives to ESX 3i. Are you going to be standing up an ESX 3i system on commodity hardware in your test lab? Talk Back and let me know. The postings and opinions on this blog are my own and don’t necessarily represent IBM’s positions, strategies or opinions.

Topic: VMware

About

Jason Perlow, Sr. Technology Editor at ZDNet, is a technologist with over two decades of experience integrating large heterogeneous multi-vendor computing environments in Fortune 500 companies. Jason is currently a Partner Technology Strategist with Microsoft Corp. His expressed views do not necessarily represent those of his employer.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

24 comments
Log in or register to join the discussion
  • I dove deep dude ...

    ... and in Table 11 came to 'iSCSI with Broadcom GB NIC'. Might give that a try before buying a RAID card.

    How about an expo from you/Robin H. on iSCSI? Some free offerings out there for iSCSI targets these days and even VISTA comes with an iSCSI initiator.
    jacksonjohn
    • Can't use iSCSi to boot with.

      You can use iSCSI as secondary storage once the system comes up, as well as NFS for VM datastores, but you can't boot with it.

      As to an iSCSI story... I'm workin' on it. :)
      jperlow
      • re: Can't use iSCSi to boot with.

        That's not entirely true. If you use an iSCSI HBA, you can boot from iSCSI. What you can't do is boot from iSCSI with a regular NIC since the drivers for the NIC are loaded by the operating system.
        brian@...
        • re: Can't use iSCSi to boot with.

          i agree with brian. and to add on to that, you can also use a ESXi usb boot image to boot from a thumbdrive.

          That will elimiate any need for a compatible storage device.
          ernestelf@...
  • Great article - Good Info!

    Similarly, as soon as I read VMware was offering their hypervisor for free, I thought about trying it. I recently installed the RTM Hyper-V from MS, and am quite impressed with its simple installation and ease of use. And, I was able to install Hyper-V on a "desktop" computer with standard hardware. Your info on the limitations of ESX 3i and what is needed will save me a lot of time if/when I get the needed hardware. Thanks for the great info!
    jpr75_z
  • Adding my $0.02 ...

    I installed VMware ESX 3.5 with no problems onto my homebrew server that has a Tyan S5396A motherboard using onboard SATA.
    Adrian Kingsley-Hughes
    • Does that MB double as a SAS controller?

      Is that a high-end server motherboard?
      jperlow
      • It's a high-end MB that uses SAS RAID, Adrian

        That's a $400 high-end server motherboard which includes a SAS controller. You've got your SATA drives hooked up to SAS using SAS/SATA translation, just like I did.

        http://www.tyan.com/product_board_detail.aspx?pid=562

        Thats an LSI 1068E SAS controller. Thats why it worked.
        jperlow
        • Yeah ...

          Having looked at the PDF on VMware's site that you linked to I'm amazed that it worked as easily for me as it did. An example of how spending money on a good board saves money down the line.

          A free tip for other readers ... you can install ESX Server into a VMware Workstation VM but you must choose SCSI drives and not IDE.
          Adrian Kingsley-Hughes
          • Yeah, you can do this for testing

            But obviously, you can't actually virtualize inside a virtualized instance. Or if you do, it will be ugly.
            jperlow
  • Don't knock the drivers

    While VMware might be a little behind on the drivers, keep in mind that it is 3rd party drivers that cause a lot of problems in Windows (BSOD anyone?) Its great that there are more available, but they can also create nagging problems. I applaud VMware for keeping stability a top issue and NOT allowing 3rd party drivers to destablize an otherwise bullet-proof product.

    When I first started reading the article, I was really going to flame for bashing ESX not working on non-enterprise hardware, but then I saw you backed off and just provided a solution. Nice job.

    Remember too, that you can always use the free VMware Server which can run on top of windows and with 2.0 (RC just released) can use the infrastruture client as well as its web interface. Its really matured into a great product for those on a budget and non-production environments.
    MicroNix
    • Amen ...

      ESX is rock-solid reliable, and I wouldn't trade that for anything.
      Adrian Kingsley-Hughes
  • RE: My Sunday Afternoon: Fun with VMWare ESX 3i

    I love how he says "Live Migration" instead of
    VMotion...Hmmmm...where has he been trained and coached?
    ???
    oljaitu
    • Training and Coaching

      That would be official VMWare coursework at a VMWare authorized training center. VMoition is a live migration technology. To call it anything else is vendor newspeak.
      jperlow
      • There is a slight difference

        With VMotion, it only moves the memory and CPU contents. The last time I tested RHEL/Xen's Live Migration, it moved the disk file along with memory and CPU-not a good thing unless you want to move to a different storage array.

        And when I tried to do a live migration with Hyper-V...never mind.
        markdean
  • Storage options

    Two things regarding storage:

    1. The ESX/VC installation manual mentions these as supported on-board SATA controllers:
    Supported on-board SATA include:
    Intel ICH9
    Nvidia MPC55 (this includes Crush8-04)
    ServerWorks HT1000
    2. Don't forget about NFS--you can set up any NFS share to act as VMFS storage; granted this storage is generally slower than local or SAN storage, but it gets the job done for testing out ESXi.
    kahharian
  • SATA support

    You don't need an enterprise class storage subsystem to run ESXi. As explained on http://www.virtlab.biz/vmware/8 you can run ESXi on the Dell PowerEdge R200, one of the cheapest rack servers available in Norway. The SATA chipset is Intel ICH9. Of course, you won't get any RAID support since the software based RAID solutions bundled with the cheap SATA cards are really, really bad, but that's a minor problem if you ask me. For the full compatibility list over cheap whiteboxes supported by ESX(i), please take a look at http://www.vm-help.com/esx/esx3.5/Whiteboxes_SATA_Controllers_for_ESX_3.5_3i.htm . So, the SATA2 cards are supported, just don't use them as "RAID" controllers :-)
    sfrode
    • Improving with time

      See also http://ultimatewhitebox.com/ for another whitebox HCL.

      Speaking of improvement, just this last weekend I changed my post about a Dell Latitude D630 from requiring the SATA controller in legacy mode to full AHCI mode with 3.5.0u4.
      CummG
  • Adaptec 2405 configuration

    I've had a few false starts with ESX Server recently and I'd be interested in a few more details about the configuration you used for the Adaptec 2405. I noticed that you have to buy a fan-out cable for the card. Did you get the SATA cable and attach SATA drives or did you have to get the SAS cable and SAS drives?

    I've currently got a box with the Adaptec 44300 SAS/SATA Raid controller but the VMWare support said it would only work with SAS drives.
    Robert Crocker
    • 2405

      The 4-port 2405 which Adaptec sent to me for eval comes with the fan-out cable. It's pre-rigged for SATA or SAS.

      You can also use amphenol cables on it as well. There are a few different models to choose from, what they sent me was the entry level unit.
      jperlow