From Chapter Two: The appliance computing culture

Each major section in BIT ends with a case study - this is the Cutter Mills case from 2001 illustrating the governance issues associated with appliance computing.

This is the 24th excerpt from the second book in the Defen series: BIT: Business Information Technology: Foundations, Infrastructure, and Culture

Note that the case study used here was orginally written for Note too that there's no way to preview a post entered here - so I don't know how the table at the end will come out until it gets published tomorrow morning.

Case Study: Appliance computing at Cutter Mills

Making the case for Unix: Systems Decisions at Cutter Mills

The "I" in this story belongs to a systems consultant brought in to advise the executive committee on the choices they face with respect to Systems operations and an internally generated systems change proposal.

Martin Cutter Mills Inc. of St. Paul, Minnesota, its staff, structure, and plans, are fabrications but the situation presented, and the remedies offered, reflect the author's experience with real-world organizations facing broadly similar situations. Cutter Mills is imaginary, but the conditions, decisions, and outcomes described are broadly based on real events.

This story is primarily about governance; about what happens when senior management abdicates its responsibility for information systems to "technical people." Remember high school? the pretty party people acted as if nerds belonged to a lower social class - and carried a contagious disease that lowered the social standing of those seen with them. When senior management acts like that with respect to Systems, they put the company at risk and make attitudinal change a much bigger issue for an incoming CIO than anything directly related to systems or technology.

Case Background

Martin Cutter Mills has about 1,800 employees and makes and packages milled grain products, predominantly breakfast foods, for both relabeled and own label distribution. Cutter's 5.5% market share makes them the fifth largest producer in a highly concentrated market in which the four biggest companies control 84% of North American production. Plant gate margins are good at about 65%, but sales and administrative expenses eat this up to leave them with only about a 2.7% return on sales of nearly $800 million in fiscal 2000.

Systems use at Cutter started in the seventies with a warehouse management suite built on the IBM System 36 with core manufacturing and order management capabilities added in the OS/400 environment during the late eighties and early nineties. From about 1995 through to mid 1999 substantial effort had been made to rebuild this functionality in a Windows environment but production processing remained on the AS/400. In late 1999 an audit letter led to the decision to stop this development effort and seek a commercial package solution instead.

The project manager for the development effort was given the job of organizing the software selection and subsequent implementation. In that role he reports directly to the systems steering committee and has now asked them to approve an "action plan". I have been brought in to review this plan and make any appropriate recommendations for change.

3. The Project Manager's Plan

When I look at the plan, four things are immediately apparent:

  1. the plan is, by weight, roughly 80% about hardware with lots of loving detail about the SAN, network switches, and new rack servers but very little information about the software, transition plans, user needs to be met, or benefits to be obtained;
  2. there's lots of verbiage about "moving forward", about business process re-engineering and the need to "retain our competitive edge in a changing world," but no real detail about actual change being contemplated or where in the organization change will be focussed;
  3. the only place where a full list of the modules to be implemented can be found is in the label column for a Gannt chart masquerading as a project plan. Implementation times look surprisingly short and, subject to my somewhat sketchy knowledge of what specific module names translate to in terms of functionality for this vendor, the list doesn't look like it adds up to an enterprise system; and,
  4. a few pages later, in the manpower and cost allocation table, some of the missing pieces turn up as interfacing projects. But that raises more questions because, according to a systems audit report from last year, the code they're interfacing to is on the AS/400 they want to ship out. As a result I'm left to conclude that this plan is at least in part an attempt to backdoor completion of the Windows based software development effort killed after the May, 1999 audit review.

As an outsider you can never really be sure whether or not a software vendor is financially at risk. In my opinion, however, a vendor which exhibits one or more of the following characteristics:
  1. the products of interest represent a shrinking percentage of the vendor's business;
  2. the vendor acquired essentially all of its products through mergers or acquisitions;
  3. the vendor does not have a core technical or industry focus defining its products or direction;
  4. the vendor refuses to reveal what proportion of annual revenue for the products of interest is attributable to support fees;
  5. the vendor's management team has no technical representation and they have a reputation for laying off the technical staff brought in through mergers or acquisitions;
  6. there are one or more lawsuits pending or under appeal in which reputable, experienced, clients claim that their implementation failures resulted from false claims made by the vendor;
  7. the vendor's financial results are superficially in line with industry averages but are being sustained by non operating items; and/or,
  8. the products of interest are constructed with, and fully dependent on, a third party toolset that is clearly at, or near, the end of its product lifecycle,
is a vendor I want to avoid.

After a day or two of wandering around talking to people it seems that:

  1. Cutter Mills has always been fairly heirarchical with promotion based more on going along with management views than on the ability to innovate successfully, but this culture has taken a deeper hold on the older managers as generational conflicts have sharpened. Specifically with respect to Systems there's an entrenched "hear no evil, see no evil" policy among employees that reflects their fear of retribution should they be seen as stepping out of line with the opinions of the vice president for Finance - to whom Systems still reports;
  2. nobody on, or off, the selection committee was really consulted on the decision. Instead committee members were led to believe that the project manager's decision had high level support and so endorsed it in public while opposing it in private;
  3. neither customers nor suppliers were formally consulted;
  4. since Systems reports to Finance the proposal is widely seen, and resented, as yet another attempt by the bean counters to impose more controls on the people who make the money; but, in reality, the Finance VP has had no more directional influence than anyone else.
    In conversation he makes it clear, in fact, that he has no idea what changes are being proposed when he talks about current system response problems being resolved by the new gear. GL batch entries, he tells me, that now take 25-30 minutes to process will go in 1 or 2 minutes.

    The problem with this is that the new software doesn't support batches as he understands them. Changes are made in near real time as data is entered, controls are no longer based on balancing batch totals but on system internals and things like access permissions for operator groups. This stuff is not in his vocabulary; the new system, he's quite sure, will be a faster version of the old one with no implications for change in his operations.

  5. Hierarchical communication is strictly enforced within Systems.
    Even a simple help desk call gets logged for approval if any kind of action is needed. Any substantial action goes up the corporate ladder to the steering committee for approval and then down to Systems management - but everything put forward by Systems management gets approved by the steering committee;

  6. IT staff are not allowed to speak directly to users on business issues and have to wear large, highly visible, name tags when they step outside the data center. Since no-one else wears these, I can only assume that the tags are meant to mark IT people as untouchables.

I get no actual answers from either the project manager or the nominal Systems Director, but the sales and production people turn over their files and so I get the answers they got. From these it seems that:

  1. the decision to go with Win2k Server instead of the AS/400 was made purely on the software vendor's recommendation;
  2. IBM, Oracle, and Peoplesoft were not short listed because their prices put them out of the ballpark; and,
  3. the project manager expects to make up for time lost "at the steering committee level" by bringing in people from a consulting firm whose tee-shirts some of his staff wear at work.

All of these are true lies. This vendor doesn't offer an AS/400 product - and may well have been picked for that reason; the other companies seem to have offered very different product packages - and may have been misled on what to propose; the steering committee delays were foreseeable - and are probably being cited to offload the project manager's responsibility for the cost consequences of his earlier decisions.

You would expect the current plan to fit into a wider corporate IT vision; but it does not; there simply is no global plan or vision to be found. Instead, each new action is treated as a project undertaken for, and funded by, a divisional or other operational group; creating a system which is locally satisficing but globally irresponsible. There are many examples of this syndrome's effects, but one, their existing supply chain solution, is particularly illustrative of the dismal consequences this produces.

A supply chain system links a company's ERP system with those maintained by its suppliers and so enables it to cost and schedule orders for which the required resources must be sourced from those suppliers.

Cutter has customers who have supply chain software and so insist that their suppliers, including Cutter, must accommodate it. Cutter, however, does not have a functioning ERP system for them to connect to and so the first customer to make supply chain functionality a requirement for Cutter to continue doing business with them caused a major kerfuffle within Cutter's management and IT ranks.

The solution they arrived at perfectly illustrates IT management's abdication of their corporate responsibilities in favor of local solutions. Since this was seen as a marketing problem they set up dedicated PCs in the marketing offices - one for each customer requesting supply chain access. Each PC was the equipped with either a traditional telecom or network connection and the appropriate EDI software for that customer. These machines automatically issue acceptance confirmations on all volume and ship date requests received and then print the orders.

When marketing gets the paper order, their people work the phones to ensure that it gets produced and shipped on time.

Marketing is quite proud of this solution because it has allowed them to present Cutter Mills as a flexible out-source supplier to various national brand marketing organizations. To Marketing, this looks like an efficient, low cost, solution; but the actual cost is orders of magnitude greater than what they see being spent on a few PCs and daily phone calls for production liaison. Those costs hardly amount to nickels and dimes; the thousand dollar bills are in manufacturing accommodations to cope with the resulting unpredictability in both product volume and product mix.

By simply accepting all such orders without back-checking availability-to-promise against production resources, the company forces itself to keep a larger finished goods inventory as a kind of surge protector and, more subtly, to make its raw products -the ready to package stuff coming off the conveyor belts- as interchangeable as possible.

The retail product name is on the boxes the stuff coming off the conveyor belt goes into. Since you can't store breakfast flakes in bulk (the edges crumble to produce a fine powder no one will buy) the answer is to put the same stuff into all the boxes so that production planning is unaffected by variations in the quantities boxed under each brand name produced. After all, as one sales guy expressed it to me, "corn flakes, is corn flakes"

Unfortunately this isn't true, and attempts to make it so are expensive. For example, the Winnipeg plant makes a multi-grain and raisin mix that's sold under various labels including a house brand for a Canadian chain of grocery super stores and that of a nationally advertised brand in the US. The American company, which presents the product as a premium brand, is mainly concerned about "mouthfeel" and requires all of its suppliers to meet various technical standards designed to ensure that the product meets consumer expectations.

During packaging the product is placed in sealed plastic bags that are then put in pre-printed pressed paper boxes. For the Canadian super stores these are shipped directly to the stores and have an average shelf life there of about three days with most customers buying for weekly replenishment to produce an average box life in the 6 to 8 day range. The American brand, in contrast, is shipped to a wide variety of retailers including many smaller stores that extend the period between production and final consumption to forty days or more.

As soon as the plastic bag is sealed in the packaging process water starts to migrate from the raisins (15.3% water initially) to some of the grain flakes (which average 3.9% water initially) and this, of course, affects crunchiness or "mouthfeel" - thereby limiting shelf life. As a result Cutter employs a variety of production and storage techniques -including using additional humectants (which osmotically transfer water out of, and sugars into, grain flakes), allowing for longer pre-production drying times, and refrigerating finished product storage- to slow this process enough to meet the US customer's standards.

Those actions raise fully allocated production costs by about 4% relative to the minimum needed to meet the Superstore's standards. Since the US product commands a premium this wouldn't be an issue except that the need to position Cutter as an instant response outsourcer forces the plant to make all of the product to the same standard -and so to incur these costs on shipments to the superstores. Since those shipments account for about 75% of the demand for this product, the net result is an average 3% in unnecessary and uncompensated production cost.

Across the company both the numbers and the accommodations made vary. In one area, product wastage is double what it should be, in another a customer responsible for about 30% of production requires that a relatively minor input be purchased from a company it owns - at well above market prices. Storage cost for both raw and finished goods exceed industry averages for almost everything Cutter touches - and turns are several points worse than industry averages too. Guessing the average cost of these adaptations at about 4% of revenue suggests that this this marketing kludge is probably costing the company about $32 million a year -a third more than the bottom line.

The Report

In the meeting called to discuss findings they don't want to hear about governance or controls. Instead they signal their eagerness not to face up to their responsibilities by talking about "where we are at this point in time" and about "moving forward from here." One even talks about "progressing the plan."

I was hired to provide a review of the current plan, not a new IT strategy; but two of the more aggressive vice presidents have talked to me before hand so I'm ready to look surprised - and sketch out their more obvious options. I tell them we'll spend a few minutes looking at where they are and then talk about the options for the future.

Where they are is easy:

  1. the selected software is not a good fit for the business; it's built using an obsolete and unsustainable toolset - meaning that even a successful implementation will require them to do it all over again in a few years - and the selected software vendor is probably headed for, if not business failure, almost certainly a kind of ghost-like existence in which customers get milked for support fees but nothing gets put back into product development;
  2. they have a history of slow and cancelled implementations for new software with the business reliant on the stability of the core As/400 environment. Things work largely because people can trust those applications, and the data behind them, to be there. It is the stability of these core applications, however limited they are, that allows systems management to spend so much time and money working with things that are fundamentally unrelated to the business -but this plan does not include contingencies for the continuation of those services past the machine's lease expiry. It relies, instead, on fast implementation of new Windows 2000 based software, including in-house developed stuff that this committee canned in June of 1999.

The bottom line on this project is that pursuing it will place the company at serious risk of short term operational failure.

As a group, they don't want to talk about reporting and control relationships and most assuredly don't want to know that the problems are natural and predictable responses to their organizational negligence. What they want is a set of action recommendations that get them off the hook; it's not that they didn't already know that this project needs to be cancelled, it's that the executive is deeply fractured along generational lines and they've all been trying to wield Systems as a weapon against the other guy but now push has come to shove and they need some way to bail out of the mess they've made.

5. Action Recommendations

What's needed here, I tell them, is exactly what the plan promises but has no obvious intention of delivering: an across the board Systems re-engineering effort aimed at aligning systems operations with business needs. It takes time and effort to develop a real plan to do this, I certainly can't do it based on a few interviews focussed on evaluating the existing project plan. What I can do, however, is talk about what the choices are and what to look for in evaluating them.

To re-engineer Systems they must first decide what kind of systems culture they want. To discuss that issue in any meaningful way we have to compare two alternatives. Since, I tell them, their project manager has picked Windows and I'm a Unix bigot, those are the two cultures we'll compare, but they are not the only choices.

Staying with the AS/400 would provide a nice middle-of-the-road choice from a cost and security perspective. Many of the facilities, particularly the use of smart displays, network inter-operability, and scalability within the limits of this business, are available in both environments; the big differences are in management approach and Systems culture.

In the AS/400 world control has to be heirachial with the relationships between systems people and users mediated by one or more layers of resource managers who control what gets done, when it gets done, and by whom it gets done. As in Windows, the default decision in terms of adding or changing a service is "no". In the Unix world control is distributed, users interact directly with systems staff, decisions to do work are mostly made by the people who do the work, and the default decision in terms of adding or changing a service is "yes".

This company has always been intensely heirachial and the only things wrong with its present AS/400 set-up are that management is distracted, the software is hopelessly obsolete, the current machine is underpowered, and costs are out of control because the people in charge of IT starved the AS/400 operation for resources while focusing on growing the Windows side of their business. But, get a good systems management team in here with the right experience and attitudes; pick something like the J.D. Edwards OneWorld package to run on it, and they should be able to transform the existing systems organization into something that delivers first class results while fitting well with the current corporate culture.

The other choice is to remake Systems into something that drives corporate innovation by lowering costs and dispersing decision making to those most qualified to make them. I believe that this is a better long run bet than extending the AS/400 investment because it will give them a serious competitive advantage relative to their four biggest competitors - one of which has OneWorld on a new iSeries, one of which is in the late stages of being SAP'd, and two of which are said to have MVS based data processing operations that I know nothing about.

One of the things they have to think about in this context, I tell them, is that you don't get competitive advantage or even competitive parity by adopting your competitor's software; you always end up at a relative disadvantage because his people are ahead of yours on the learning and acceptance curve.

"So you're saying", one of them asks, "that we should not go with Edwards?" but I'm not saying that at all. With proper management Edwards would be a lot better than what they have, or the stuff this project manager wants to buy or build. I tell them again that I'm not actually recommending a specific strategy here; I'm pointing out differences so they can see what decisions have to be made and how those decisions fit together.

We'll start, I tell them, by specifying what Cutter needs to achieve and then look at how to get there using each of the two options. After that, we'll look at what each one means in terms of things like cost, productivity, and available software.

Getting people to articulate systems goals is difficult because people tend not to look beyond work processes to the purposes of those processes. In talking about goals people tend to talk about what they do, not why it's done. Here, people mentioned work processes like matching accounts receivable against orders, monitoring grain inventories at plant sites, and confirming the customer's payment record for order acceptance. All of these are important but none are relevant. The key to business process analysis, and so to strategic goal setting, is to focus on what has to be done, not on the processes implemented to do those things.

In the meeting I use a whiteboard to put up the key points; here I'll be replicating that using these stand-out boxes. This is the first one and reflects our discussion of what systems is supposed to do for this company.

    Systems Goals

  1. allocate resources to orders
  2. optimize resource allocation to maximize returns
  3. handle financial info on transactions
  4. track orders to delivery
  5. build supplier loyalty
  6. get quality control under control

Finally they grudgingly agree: a great systems solution would do two things:

  1. provide all the communications and record keeping needed to take an order from quotation through production to delivery; and,

  2. use the mix of incoming orders, both current and pending, to allocate work, across both time and production centers, to maximize returns on infrastructure.

A few people in the room think that this requires supply chain software - but this is a serious mistake. The company functions as part of the supply chain for its bigger customers; these are the guys who want automated ordering and immediate access to availability and delivered cost information. To them this is immensely valuable because Cutter Mills is one of several intermediate producers from which they can source a range of products. It is the supply flexibility they get from this that lets them maximize product turns while minimizing transaction costs and out-of-stock risk.

The right answer for Cutter is to adopt an integrated ERP/Financials system that lets Cutter participate fully in their customer's supply chains without racking up extra production or inventory expenses to do it. With an ERP system in place a customer's order would become part of a production plan before being approved for acceptance. From Cutter's perspective a fully functioning system that meshed with customer supply chain software would cut inventory costs, cut finished goods production costs, and improve revenue predictability.

The abstraction that makes this possible is the treatment of each order as a job-lot to be independently allocated among competing production facilities. This makes it possible to maximize use of the most efficient resources while minimizing overall idle time and avoiding wasteful production or inventory decisions. This would work for Cutter because its size, diversity, and customer relationships allow it to assume that there will always be orders in the pipeline and that these orders, although highly variable individually, will exhibit statistical stability.

This abstraction does not apply to the vast majority, whether measured by dollars or delivery criticality, of Cutter's suppliers. On the contrary, Cutter deals mainly with primary producers who incur high costs when faced with short term change in the type, quantity, or quality of deliveries requested. Since these guys don't have the size and diversity needed for averaging effects across many orders to compensate for change in specific orders, any gains made by Cutter through use of automated supply chain software would come directly at their expense.

That would create a trading imbalance, inviting producers to counter with things like hedging and marketing co-operatives; thereby raising Cutter's input costs. Since these counter-measures have overheads paid to third parties, everybody concerned -except the third parties- would be worse off if Cutter tried to enforce use of automated supply chain logic.

The bottom line on this is that Cutter works at the end of the retailer's supply chain and should not try to add another link. Basically, ERP doesn't make sense at the producer level and so supply chain software to link to it doesn't make sense for Cutter. What Cutter needs is the classic fully integrated ERP stuff and nothing more.

Governance and Control
  1. Matching Systems to the business
  2. Reporting relationships and decision making
  3. Service level agreement
  4. Risks and assurances

All of the major vendors, I tell them, including SAP, PeopleSoft, and Oracle offer exactly what they need in both Windows based and Unix based versions. These are rapid implementation packages, in the $400-800K range for software and services, that offer little up front customization but deliver company wide working systems quickly and cheaply.

At this point the Finance VP, who has been getting increasingly worried about my presentation, makes the strategic mistake that ties him so firmly to the existing Systems group that he loses any chance of prevailing in next week's executive committee meeting. "We rejected bids from those guys," he says, with both anger and triumph obvious in his voice, "because they were priced way out of our league. Millions, not hundreds of thousands."

He hoped to discredit the entire exercise, but he had been setup by the project manager. What I think happened was that this guy briefed the vendors he didn't want on the importance of the customization work required to fit their products to the stuff he wants to carry forward from his own development effort. I don't know this for sure, vendor complaints don't amount to evidence and proposals not short listed were ; but, the proposal he selected doesn't meet the detailed terms of reference set forth in the RFP and there seem to be no available records on what was said at various vendor briefings.

To answer the VP's charge, I point out that the customization requested in the RFP is not in the winning bid - having been moved off-budget, and in-house. To support that I get them to compare several pages from the RFP to the modules and work listed in the plan. Heads nod, Finance is isolated and knows it; I can see that the Marketing and Production VPs are already planning the motion they'll be putting to the executive. Good, to the extent that I can take sides, I'm with them on this.

Once they agree that they need ERP with integrated financials and HR elements we move to the discussion about getting that in place.

Governance issues top the list, but governance is a topic they don't want to discuss. Going through the CoBiT framework would, I'm sure, make them all extremely uncomfortable to the point of rebellion, but we have to discuss some high level management issues whether they want to or not.

So I give them my Guide to Defenestration book and tell them that Systems isn't any different from any other part of the organization. It has its internal incentives to growth and it has a job to do. The structure provided for it influences how the people hired to work in it behave in pursuit of those goals and it also influences how they are seen by people outside Systems. Right now, Systems reports to Finance and one reason Cutter has had so much trouble implementing even simple new applications is that they're seen as part of Finance; as overhead; as something to be worked around -a necessary but resented imposition that gets in the way of getting the job done.

The new CIO has to report to the CEO and the executive committee - not because the person holding the Finance role can't handle the job, but because perception gets in the way of reality. They need production applications that are enthusiastically accepted by production users. Having Systems report to Finance creates an enormous, and unnecessary, implementation barrier that can be easily demolished, right now; before they try to hire a new CIO. Don't even bother, I tell them, to interview someone who doesn't make that a condition for considering the job.

This gets a hostile, silent, reaction so I move quickly to the most basic meat and potatoes issue: organizational posture. Do they want a rigidly controlled, hierarchical, organization that focuses on managing the resource? or a leadership oriented organization that delegates decision making to the lowest level possible?

This decision is critical to the choice of technology. It is possible, I tell them, to manage Unix in the traditional heirarchical way but the results will include user dissatisfaction and high cost, inefficient operation. That happens, I tell them, because Unix embeds very basic concepts about how systems work and provides for user freedom to do whatever they want to with the system. Using this technology in a tightly controlled way is unfortunately common but really means paying for a lot of people whose job is to stomp out the use of the capabilities that come with the gear.

What Unix calls for is leadership - the process of getting more brains focussed on a job that changes as you do it - rather than management - the process of getting more hands on a well defined job. Posture the systems organization as intensely heirarchical and you will get Windows like systems services whether you use Unix or not. You can, I tell them, drive a 2 inch nail with a sledgehammer if you want to - but it doesn't work the other way. Try to apply a leadership based approach in an all Windows environment and you'll find yourself doing the equivalent of trying to drive railway spikes with an 8 ounce hammer.

It's a difficult concept and acceptance isn't helped by the fact that most places they've seen Unix used have been disasters - usually precisely because management tried to treat it as just another constrained resource to be managed. They don't understand, of course, but I ask them to read Chapter Three of the Guide and a few promise to do it, so I let it go.

That brings us to infrastructure deployment options and costs. For this, we're going to compare the effect a Windows client-server approach with normal heirarchical systems management would have on the company to the effect of going with Unix, smart displays, and a leadership based approach to management.

Somewhere in Paul Strassman's book: The Politics of information Management (New Information Economics Press, New Canaan, Conn, 1995) he makes a throwaway comment to the effect that IBM lost the competition with Microsoft in part because they believed that the client-server revolution would require massive servers and so ramped up mainframe production instead of focusing on PCs and developing OS/2.

IBM was right of course --and they would have known, having abandoned client-server as unworkable with the Future Systems project of 1967-72.

Despite the Windows myth of personal desktop device control, Cutter will have to implement very tight, centralized, control of desktop PCs to make this work - thus leaving users with responsibility for, but no control over, desktop gear.

They know what PCs are and have some idea that the PC client-server architecture involves PC desktops and racks of PC servers but they don't know what the Unix architecture is. So I tell them there are three big differences:

  • instead of using PCs that use a forty million line OS to deliver applications running on Windows servers, the Unix architecture uses smart displays with 10,000 line operating systems to deliver applications that run on any server - Windows, Unix, OS/400, MVS, or anything else. Smart displays, I tell them, are like dumb terminals, only with lots of built in smarts and brighter, clearer, graphics than most PCs. There are no moving parts, no user accessible software, and no noisy fans. Applications run on servers in one or more GUI type environments - you can, for example, run both Windows and Unix applications at the same time on the same screen and even cut and paste between them. Unlike PCs which have high failure rates and require frequent reboots and/or replacement, smart displays offer 300,000 hour mean time to failure rates, usually last five to eight years, and don't change no matter what happens to the applications or servers.
  • in the Windows architecture networking and security are complicated issues. Systems people have to work very hard at keeping things running because your PC may have to use three or more networking technologies to talk to its local Windows server, an application in the data center, and something outside the local area network. By default PCs have access to nothing, all access has to be granted and controlled. Users, for the same reason, may need to remember one password to get their PC to complete booting, another one to access an application, and yet more passwords and IDs to do something as simple as access the internet.
    In the Unix architecture all this goes away. You don't boot a smart display, you just turn it on and wait a few seconds for the login screen to come up. You login once, after that you only need to authenticate again if you access a restricted application or data set and stuff that's hard with Windows, like inter-office access, is trivially easy because the security, although tight, is fully automated and invisible to the user.

  • the combination of high reliability with scalability means that the Unix server grows with the business. Most places will have two machines for redundancy but piling on more workload generally just means getting more bang for the buck. It's quite common, as a result, to see four or five major applications using completely different database backends running on the same server with a dozen minor applications.
    In contrast, the Windows approach requires that either one machine, or a whole cluster of machines, be devoted to each process. it might be possible to run several small applications on one machine, but few try it because the high failure rate makes it almost impossible to debug a system and, of course, a failure in one application shuts down people using other applications. It's quite common, as a result, to see Windows data centers supporting hundreds of rack mounted servers.

With this background we start by looking at what the Windows decision would probably mean, including:

  1. another eight Windows servers dedicated to the ERP system;

    These would be Compaq Proliants with four of the the newly re-released 900MHz Xeons and dual 505GB (net 244GB) external data stores on fibre channel controllers. The database servers would use clustering for both performance and reliability reasons with all gear at headquarters.

  2. about 560 PC clients;

    These would be Dell Optiplex 150s with Windows XP, 128MB, and 17 inch screens. There's an odd, but common, reaction here. "Dell," "Windows XP", "17 inch", these are words they recognize; they can't tell a Compaq Proliant from a compact car, but Finance, in particular, warms appreciably at the sound of words he recognizes.

  3. current printing and networking technologies would be continued and all existing NT based servers (42 of them) upgraded to Windows 2000 and applicable applications;

  4. the existing 520 or so PCs, all of them mid range Compaq gear unsuited to XP, would be auctioned off or trashed;

  5. Direct IT staffing would increase from the 32 now employed to perhaps 40.

The Unix solution, in contrast, would have:

  1. 120 NCD 21" smart displays for use by office staff;

  2. 440 SunRay 17" smart displays for use by production and logistics staff;

  3. 8 Sun Ultra 10 workstations with 21" screens for use in Systems;

  4. two Sun V880 machines, each with 8 x 900MHz CPUs, 32GB of RAM, and 800GB of internal disk to run the Sunrays and all non ERP/Financial/HR applications;

  5. two Sun 4800 machines, each with 8 x 900MHz CPUs, 16GB, and directly connected 1.3TB T3 data stores to run all ERP and related applications;

  6. the existing networks, local and wide area, along with all the Cisco, 3com, and PC gear running it, would be entirely replaced with a flat, VPN based, solution;

  7. IT staff would give a series of introductory Linux courses including loading and using both Linux and as well as basic communications software for users whose PCs get replaced by smart displays. Users who participate would then be given the newly Linux-ed PCs for home use;

    As explained in an earlier article this step is tremendously important because it leads to creation of a pool of knowledgeable Unix users as a counterbalance to the centralizing tendency of the Unix/Smart display architecture.

  8. all existing non PostScript printers would be gradually phased out and replaced by Minolta-QMS postscript compatible gear; and,

  9. total direct Systems staffing would fall to about 22 people - 18 in operations and four in the office of the CIO.

The biggest operational differences are in how people work with the systems, not the systems themselves. Both are based on centralized processing, but with Windows you also centralize control and responsibility, thus creating both opportunities and incentives for people to behave in counter productive ways.

One of the issues here is depersonalization. You hear a lot about people not wanting to work with applications they don't perceive as theirs, and it's true; this does have an enormous negative impact. Underlying it, however, is something more basic yet: by tying user perception of the application to a group, whether that's Systems or Finance, you depersonalize it. That makes it easier for people to rationalize actions that sabotage acceptance.

In a well run Unix environment users work directly with systems staff to make real decisions. Not only does this give users an ownership stake in the application, but it ties their perception of Systems to particular individuals. When you personalize the relationship in this way, you make it harder for people to rationalize actions that sabotage acceptance.

With Unix you centralize processing but try to decentralize control - giving users what they otherwise try to take by stealth. This is another hard issue for them to get their heads around because they think of the PC as distributed - after all the computer is on the user's desktop- and Unix as centralized because the computer is in the data center. In reality this confuses presence with control. It is an objective of Unix administration to decentralize control - to create knowledgeable users who directly control their relationships with Systems. That's one reason I usually recommend that PCs replaced by smart displays be given to their former users as take-home Linux gear -provided they attend a Linux or BSD course first. (The other reason is that having IT people give those introductory Unix courses strengthens both their relationships with users and their knowledge of Unix).

A big part of Windows systems management, I tell them, is the exact opposite. You have to work hard at ensuring that you have control over those desktops and what people do with them. Otherwise the whole network becomes utterly unmanageable; but the result, of course, is that people will fight you for that control and so part of the company's energy gets wasted in the conflict.

Cutter provides a perfect example. You think, I tell them, that you currently have 32 IT positions filled out of 35 FTEs approved - but this doesn't reflect your real staffing costs at all. Richard, who works for Finance in Accounts Payable, is really a full time Windows support person and the only one there who can debug the completely audit-proof spreadsheets he wrote and got other people to use. Brenda, in HR, is another full time Windows support person who has developed some reporting scripts for Microsoft Access that no-one else understands or is allowed near. There's a quality control lab in every plant -and they all have IT people who don't report to Systems. Telecom isn't handled by Systems - and there are part time people looking after that all over the company. All in, I'm guessing Cutter pays for upwards of 60 IT people - almost all of whom spend a big part of their time just fighting each other.

Go all Windows and that situation gets worse. That AS/400 they've got is a solid piece of gear; replace it with applications that run on Windows servers and they can expect another half dozen PC babysitters to emerge, -but it gets worse than that too; people like Brenda will be unable to maintain the support level they provide now because they'll have a complicated PC client to deal with instead of the dumb but simple TN5250 emulation used now. That will inevitably mean that more people will stop doing their real jobs to provide stealthy Windows support instead.

Go Unix and things get a lot better. Richard and Brenda will have to leave or go back to their real jobs -there are no support requirements with smart displays -none, zero, nothing- the new software will obsolete the stuff they're so committed to, and, most importantly, the devolution of control to users will cut the ground out from under them.

Rough Comparisons: Unix vs. Windows at Cutter Mills
  1. CIO reports to CEO not CFO
  2. Systems mandate is corporate revenue growth, not cost savings in Finance
  3. Operational decisions made by those most concerned; not systems management
  4. Applications supported by highly motivated, knowledgeable, users; not Windows help desk people
  5. Applications always available - 100% systems redundancy in place
  6. Server count drops from 42 now to 4 instead of rising to about 50;
  7. Systems staffing falls from 35 to 22 instead of rising to about 40;
  8. Non Capital IT cost plan for 2002 cut from 2.7 million to 1.8 million

Projected Costs with ERP
With Windows With Unix
Impact on Users Average desktop application crashes per week1 120 0
Average server failures per week2 4 0
Performance (Load 1000 email messages) >2 minutes 1-3 seconds
Security (e.g. for employees authorized to access/change quality and health related records) Login and Password within Advanced Directory; second login/password at client start-up Smart Card plus login/Password at database level with full audit trail
Security (from email and related attacks) Entirely reactive; every attack costs something Entirely pro-active; most attacks irrelevant
Employee Support Payroll based PC purchase program; remote access is by dial-in with PC-Anywhere Older PCs given to those who take Linux training provided by IT staff; access is via dedicated VPN
Impact on Costs (in thousands) Staffing Costs (fully burdened)3 2,200 1,430
Approximate cost of ERP/Financials/HR software & uncustomized implementation 600 600
Systems hardware and software costs 1,3804 1,8214
Month 36 HW/SW replacement 1,3805 2005
Total Five Year Cost 13,760 9,171
1Failure rates for 560 PCs running 8 x 5 are based on the average time between failures (188 hours) for all Microsoft Windows branded operating systems as computed from numbers reported by
2Failure rates for 65 servers running 24 x 7 are based on the average time between failures (2983 hours) claimed by Microsoft for Windows 2000 Server. Bugtoaster appears to focus on desktops, the comparable number for Windows 2000 there is about 240 hours or about 45 failures per week.
3Unix salaries estimated at 65K fully burdened; Windows salaries at 55K.
4Cost information is from the Sun, Worldcom, Microsoft, Dell, Compaq, and Lucent websites as of Nov 18/01.
5Estimates are based on a single HW/SW refresh cycle for the Windows option. Actuals are likely to be higher. See the TCO article earlier in this series for a discussion of this issue.

The big issue here, I tell them, pointing at the bottom lines in both columns, isn't this cost difference. The cost difference is important and typically increases as a percentage of total cost as the overall system increases in size, but cost is not the critical issue. Cutter doesn't make great returns, but a few million more or less on systems isn't going to make or break the company.

Embarking on a suicidal attempt to implement supply chain without considering the nature of your suppliers, and without having an ERP system in place - that can kill your company. The alternatives I've suggested look and sound like they're technology based - but they're really not. They're based on first choosing how you want to posture systems in your company and then picking the technology to make that work.

Choose to position Systems as an asset and you'll need Unix technology and leadership oriented control. Choose to position Systems as an infrastructure item, something you treat as a cost of doing business, and you'll need traditional systems management and software like the Edwards OneWorld package on an Iseries. Don't make a decision and you'll get more of what you've got; uncontrolled cost growth, escalating failures, and the dominance of personal agendas over corporate responsibility.

Some notes:

  1. These excerpts don't (usually) include footnotes and most illustrations have been dropped as simply too hard to insert correctly. (The wordpress html "editor" as used here enables a limited html subset and is implemented to force frustrations like the CPM line delimiters from MS-DOS).

  2. The feedback I'm looking for is what you guys do best: call me on mistakes, add thoughts/corrections on stuff I've missed or gotten wrong, and generally help make the thing better.

    Notice that getting the facts right is particularly important for BIT - and that the length of the thing plus the complexity of the terminology and ideas introduced suggest that any explanatory anecdotes anyone may want to contribute could be valuable.

  3. When I make changes suggested in the comments, I make those changes only in the original, not in the excerpts reproduced here.

Show Comments