X
Tech

EvriChart: A Linux Success Story

Tony Maro, CIO of EvriChart, a hospital records management and archiving business, successfully migrated his company's Windows-based line of business document management extranet application and his employees' 40-odd Windows-based desktops to a 100 percent Linux-based server and desktop infrastructure.Jason Perlow interviews Tony Maro, CIO of EvriChart, Inc.
Written by Jason Perlow, Senior Contributing Writer

Play audio version

tonymaro.jpg

Tony Maro, CIO of EvriChart, a hospital records managementand archiving business, successfully migrated his company's Windows-based line of business document management extranet application and his employees' 40-odd Windows-based desktops to a 100 percent Linux-based server and desktop infrastructure.

Jason Perlow interviews Tony Maro, CIO of EvriChart, Inc.

I had the pleasure of speaking to Tony Maro, who is CIO and an owning partner in EvriChart, a medical records management company based out of White Sulphur Springs, West Virginia.

EvriChart had some unique requirements which included replacing a Windows-based line-of-business application for document management that would not scale with the continuing growth of their business. By migrating to a Linux/Open Source-based Web application, it paved the way for full desktop Linux adoption at his company. I asked Tony to summarize his experience so that I could share it with you in the hopes that you might gain some valuable insight from the process he had to go through.

Next -->

What EveriChart Does

Our company, EvriChart, Inc. manages medical records for hospitals all across the country.  We only do healthcare, and to my knowledge we are the only document management company that has specialized in just healthcare.  We got started by doing archive scanning of records for hospitals.  Now we do a lot more, including:

Document scanning

Paper records storage

A custom clearing and indexing for destruction process we call ChartClear which is very IT-centric and integrates with our online record request system

Online hosting of scanned documents for hospitals

Custom data conversion services - for instance reading old DAT tapes and converting into a more modern format

Master Patient Index error analysis - for instance identifying where one person has multiple charts at a hospital.

If the other document storage or management vendors were smart, they'd drop healthcare clients in a heartbeat.  We've positioned ourselves from day one to exceed the requirements related to healthcare information, but most vendors treat health information the same as the local accountant's or restaurant's documents.  I've already seen vendors agree to sign business associate agreements with hospitals saying they were fully compliant with the new rules, who I know for a fact are not compliant because I've been in their facility since they signed it.  What they don't realize is that it's not the hospital that will be enforcing that agreement, it's the Federal Government, and there's stiff penalties associated with it.

Why EvriChart migrated to Linux

Initially we installed a Windows based document imaging system to handle the digital conversion of records, along with the indexing and production of the actual data.  Everything happened through Windows.  As we grew, we found that the design of the software didn't scale well with our business, even though they brag that NASA and other large organizations use it daily, as do many scanning service bureaus.

We found that once you reach more than a few million pages of scanned images nearly every document imaging package would bog down.  Considering we were doing that on a monthly basis, and wanted to keep things online for an additional 30 to 60 days, we were exceeding those limits regularly with only a few users.  Our only other option was to deploy a second or third Windows 2003 server with another complete installation of the Windows based document imaging platform to run in parallel with the one we already had.

Repeatedly we would visit hospitals that already had an in-house document imaging system who would complain that they could find a CD copy of the scan, load software on the CD and find the record before it could be pulled from their "online" document imaging system.  This led us to believe that keeping with the current paradigm wasn't the answer.

These were some of the challenges that drove us to Linux and to developing our own in-house document imaging and archival system, in addition to the basic issues of document imaging software license costs, Microsoft license costs, viruses and malware.

Nearly from the beginning we used the Asterisk Vo/IP open-source phone system, and still do today.  We started our migration in slow steps, beginning with implementing our own workflow and inventory management application in PHP running on an Ubuntu server.  As we started to run into performance problems with our Windows document imaging system - somewhere around 500,000 pages per month, I started looking for ways to wedge Linux into the process.  We found that the Windows app was literally sending the same data across the network to and from the same server a total of 4 times.

We immediately set about wedging a Linux server in between.  We eliminated half of the traffic to the Windows server immediately by doing the initial scan storage and preprocessing of images on a Linux server with a web browser interface for the clients.   Immediately we saw relief - not only did we need half as many user licenses for our Windows scanning software, but we were able to take the most time sensitive parts off of Windows entirely so it felt significantly faster to users and allowed us to increase our throughput another 5 times.

Next -->

How EvriChart stores Terabytes of data and gets at it quickly

To solve the mass storage problems, I studied the likes of Google, Slashdot, Flickr and Reddit.  I even drove out to meet with Steve Huffman and Alexis Ohanian, the founders of Reddit, to talk to them about how they deployed their servers for high traffic.  It took two years, but we developed a document imaging repository structure that would scale endlessly as bandwidth allows, returns results significantly faster than anything else on the market and was considerably easier to use.

We call it "DocStore" and it's running on Linux servers using LAMP with DRBD for high availability.  We developed it with a REST interface in mind so we could plug it in wherever we needed it as we continue to grow.  We're currently storing just over 3 TB of scanned images online, where we could never reach 800 MB on our Windows software running with MS SQL Server without having severe performance issues.  Where others developing software with Windows in mind would develop a monolithic server, we developed a distributed network of servers to handle whatever load we needed.  Our structure also allows us to securely replicate over the Internet to secondary datacenters for redundancy.

The final conversion

In the past few months we were able to roll out our final server-side conversion - we utilized our DocStore backend system with an AJAX enabled frontend to allow indexing and quality control of scanned documents.  Now, every document we scan is completely managed through a web browser, which eliminated the need for any Windows workstations at all.  That's when we started converting the rest of our Windows desktops to Ubuntu.  To help that process I rolled out a Puppet configuration with a PXE unattended installation setup.  You simply boot a workstation, pick if you need a 32 or 64 bit OS, set the machine name and walk away.  Forty-five minutes later you have a freshly installed Linux desktop that is completely updated and configured for the network.  It even sets the user's homepage in Firefox to our intranet site.

How EvriChart delivers encrypted data

Part of this final conversion is that we developed a way of delivering scanned data to customers that has a viewer on the DVD for searching scanned images.  It stores all the scans encrypted with SHA-256 so that the data exceeds the new healthcare HITECH "data at rest" requirements. We're in the process of switching our customers to the new application at which point I won't need Windows in-house anymore - our app will work in both Linux and Windows.  We were also able to automate much more of our workflow, for instance any data we are shipping to customers rather than hosting is automatically published as soon as it's ready to go using a special tower we built with an array of DVD writers in it.

Next -->

The software EvriChart uses

We currently use the following open-source software (among others):

Ubuntu Linux (server and desktop)

Tripwire

Apache

MySQL

PHP

Pound reverse proxy

Memcached

Ganglia network monitor

Nagios

DRBD

OpenLDAP

OpenVPN

OpenOffice

Firefox

Thunderbird

E-Groupware (which we sync OTA to our BlackBerry's)

Asterisk

Eclipse IDE

Puppet network management

On dealing with pushback from employees

As far as the migration goes, I've received zero pushback from employees with two exceptions.  The office manager and the Chief Operating Officer are both hooked on Outlook on their laptops and don't want to give it up.  For the most part the employee just sits down at the new computer, gets a 5 minute demo on how to do the basics and is cut loose.  My initial test of how that would go was when we appointed my assistant.  Without asking her I had her computer prepared as a Linux desktop.  A 5 minute tutorial on how to log in, get to OpenOffice and print and she was good to go.  I found it much easier to find Linux capable IT staff and software developers than I originally thought it would be.

I'm a firm believer that the reason people first used Windows in the 1980's was because they sat down at work one day and a Windows based computer was there.  They bought what they were familiar with for home and over time Windows became the standard.  They initially either learned through trial-and-error or through special training classes.

Years ago I sold computers at Radio Shack.  I'll never forget the day a couple came in to buy a computer so they could use a spreadsheet to help with their farm expenses.  The wife sat down at the computer, picked the mouse up and touched it to the screen.  You can imagine how much tact it took to explain the proper use of the mouse without embarrassing her.  Today you can safely assume that everyone knows how to use a mouse, how to click a menu item, and how to browse the web.

Even if they've never used Linux, those same basic skills transfer over.  That's why I've focused on the 5 minute demo for new Linux users in most cases, and for those who are just going to use a web browser we don't even do that.  The computer will already be configured with the Firefox icon on the desktop and the homepage set to our workflow app.  We simply say "here's your computer - it's waiting for you to log in" and we can walk away.  They intuitively know how to open Firefox, do their jobs, and print documents when needed.  Other users get a 5 to 10 minute demo on where the common applications are located and a quick description of how the "Start" menu is at the top of the screen now.  We also describe the basics of the Thunderbird email software for them and preconfigure their signature file.

Since we started the major desktop migration, our production manager actually asked me to install Ubuntu on both his desktop and laptop, and to provide him a disc so he could install it at home.  Our Controller actually asked me to investigate a couple of Linux based accounting applications she had found saying she "wouldn't mind running Linux."

What's important is that for the most part these people are not computer people.  They are average Joes.

How EvriChart addresses security

Another big aspect this has been very helpful with is security.  Since we only do healthcare we've always run our business like a medical records department where most vendors didn't take that kind of care with security.  This put us in a unique position with the Obama administration came out and said that "Business Associates" of hospitals would be held to the same rigorous standards as hospitals, and by the way those standards just got a lot tougher.

With Linux I was able to easily take a few extra precautions like forcing a screensaver to come on and require a password after 5 minutes, and disabling USB storage devices without disabling USB keyboards and mice.  Our Linux firewall doesn't allow Internet access from any workstation that might contain medical information, except to get security updates.  Our Linux laptops all use whole disc encryption with AES-256 and strong passwords.  Our accounting office in Virginia is linked to us through an always-on VPN link using OpenVPN which was worlds easier to configure than the IPSec hardware we initially tried.

How Linux gave us cost savings

We build all of our own boxes.  For servers this normally means a barebones 1U server that we customize with RAM, HD and CPU depending upon what tasks it will perform.  For desktops we build a typical low-end desktop with a reasonable video card to handle the newer 22" widescreen LCD's and a server-grade hard drive and power supply.  The two points most likely to fail in a desktop are hard drive and power supply, and by placing better grade equipment there I'm able to still save money on RAM, CPU and mobo while getting a machine that's going to be maintenance free for many years.

Our average server costs between $1500 and $2000, total, just slightly less than what the Microsoft license fees would be for our number of users.  Desktops range from $400 to $600.  Over the past 3 years I would say our migration has saved us at least $75,000 in just server software and licenses, not counting desktop licenses.  The IT staff certainly has cost me no more than if I would have stuck with Windows, but it does mean I hired different people than I would have otherwise considered.  With the Linux hardware detection at the level it is today, it doesn't matter for the most part what hardware we use and we still don't have to do any special driver setup after installation.

The coolness factor

The last thing is just the "coolness factor" of what we're able to do.  We've made our inventory / workflow system able to control our phone system provisioning right alongside user account management.  We use OpenLDAP as a single login for multiple systems.

We can track aspects of our business that previously weren't accessible to us, like how many boxes of documents a given person produced during a certain period, per hour actually worked.  With the traditional (*)nix design most open-source software can communicate directly with other software without the need for an API, so we're able to take disparate applications and use them as building blocks to produce our desired results.  We can sit and have discussions over what would help us drive the business in the right direction and I can say with confidence that our team can integrate disparate features in a short period of time without vendor interaction.

Is your company also a Linux Success Story? Talk Back and Let Me Know.

Editorial standards