The Challenge of Vendors'' Products
Twenty to 30 years ago, computer systems that captured clinical data and automated
clinical processes existed almost exclusively in academic medical centers where
their use was largely experimental. Today, health care providers with fairly homogeneous
IT needs form a large market for clinical solutions. Their need to find extra
cash, control costs, and respond to both government and public demands creates
and governs this growing and highly specialized market.
All health care organizations have somewhat different processes. However, very few have the time or financial resources to develop their own custom clinical applications. Clinical solutions vendors have the resources to develop and adapt clinical information software for a wide range of provider organizations.
Since health care organizations will have a long, intimate partnership with their clinical solutions vendors, they need to understand the challenges that they and their vendors face. Exploring these challenges is the subject of this paper.
What Is a Vendor?
When health care providers acquire the right to use a computer solution, they buy a license from a "software or solutions vendor." Usually, the vendor selling the license also offers some level of implementation support. Over time, the vendor evolves the solution and, for a fee, offers both support and incremental improvements to the product for a yearly fee. Sometimes, a legal designee of the original license-holder performs these support services. When the purchase is concluded, the health care provider and vendor enter into a contract specifying the products and services the vendor will supply to the provider.
Vendor Products: Choices and Tradeoffs
When providers acquire or alter an information automation solution, they have two basic choices. They can either license a solution that has been developed to meet the needs of more than one potential customer, or custom build a solution tailored to their own specific requirements. Both approaches have advantages and drawbacks. However, as suggested above, the license option is by far the most attractive for a large majority of health care providers.
The build option offers the potential advantage of providing an exact match between software capabilities and the user's needs. All things being equal, however, the build option costs more because one buyer is taking on all the costs of the solution (design, build, testing, implementation, and support) and cannot leverage these costs over multiple organizations.
When a provider licenses a solution, the vendor assumes the costs of development and support. The vendor creates the solution so that it can be implemented and supported at multiple customer sites. The solution is configurable to a variety of situations. A solution's configurability may extend just to simple changes (such as tables to hold user names, roles, locations, and code sets for orderable items) or include sophisticated customization of screens, data fields and relationships, screen and data ordering, subsequent triggered events, and interfaces.
The more configurable a clinical solution, the more appealing it is to potential customers — especially to those who need the configurable capabilities. However, configurability costs money. Since highly configurable solutions cost more to design, build, and support, vendors pass these extra costs on to buyers in a pro-rated fashion. These costs are rolled into the price, and all buyers pay for them equally. Other costs are unique to, and vary with, specific implementations, and they are priced and billed separately from the base solution.
The Solution Lifecycle
A clinical solution is composed of many pieces of complex software and associated hardware. Buying a solution generally entails buying new hardware. The licenses give the provider the legal right to use the software, and a copy of the software itself. The software is then implemented at the provider organization, and the software's lifecycle begins.
A software solution's lifecycle starts with the product's design, building, testing, delivery, and support. When a new version of the solution is developed, this sequence of events is repeated. New versions correct deficiencies in current versions and usually add new functions to the product. The vendor provides the new versions, along with other support services, for the provider organization for a monthly maintenance fee (usually 1 to 2 percent of the solution's license fee per month). Some new versions may require new hardware, new implementation services (performed by the vendor, provider organization, or a third party), the conversion of existing data, a period of service disruption for users, and training on the updated system for users.
No solution is updated indefinitely. The license agreement may specify the vendor's responsibilities in this regard. However, vendors (or their successors) will eventually notify customers that they intend to cease supporting the solution — even if a customer holds a "perpetual license" entitling them to continue using the software without future support. Many times, this notification will arrive with an announcement of a new solution that typically requires a new license and possibly new computer hardware. While a vendor may decide that it is a good business practice to offer a significant discount to their customers to upgrade to the new solution, most licenses do not require the vendor to do so.
The Software: Who Makes It, Who Controls It
Most vendors today rely on third-party companies to provide fundamental software components that, together with the hardware, form the platform for delivering their solution. Vendors in health care and other industries use components that include items such as operating systems, database systems, communications software packages, and security subsystems.
For the actual complex clinical solutions — such as order entry, result reporting, and pharmacy — vendors develop, enhance, and support these solutions themselves. Sometimes, a customer is directed to acquire components such as hardware and operating systems themselves according to the vendor's specifications. The customer will then own, and have to support, the components separately from the vendor's support for the rest of the solution.
In many cases, third-party software development and run-time tools are embedded within what is otherwise the vendor's software. Professional IT departments in large provider organizations usually want to understand and, when possible, control the mix of hardware and software undergirding their solutions. Understanding the underlying architecture can give IT departments more control over the quality and cost of the IT services they deliver.
But even when IT departments understand, they cannot always control, their organization's clinical systems. The end users — clinicians — often exert extraordinary power over the solution selections, which then dictate the underlying architectures, even though they are not responsible for IT delivery or maintenance. The trouble is, clinicians and IT professionals often have very different priorities for the overall system.
IT departments, working with limited resources, usually seek to minimize the number of different platform combinations that have to be maintained. Fewer architectures require fewer distinct skill sets and competencies among the maintenance staff and lead to better quality support. They may seek to standardize development and delivery architectures and minimize the risk of incompatibilities among components and services common to workstations, browsers, Web servers, networks, and so on.
Clinicians, on the other hand, focus on a solution's clinical functionality. Standardizing and minimizing are not important goals because (to most clinicians) they do not appear to bear on a solution's clinical performance. In fact, certain departments — such as obstetrics or oncology — may push hard for highly specialized solutions that give them the precise information they feel will best support patient treatment.
Both sides make good arguments and need to listen to each other. IT departments need to understand that their goals, while worthwhile, cannot be made a requirement. A personal anecdote explains why. I was once told by a focus group made up entirely of IT people that MUMPS was an unacceptable application technology for their hospital. After listing the vendors and products that their users would probably want to consider, I pointed out that by disallowing MUMPS most of these products would be off limits. I then asked them if they were prepared to deliver the bad news to their users. Realizing that their decision would encounter stiff opposition from the user community, they quickly reconsidered.
Support for Customer-Defined Processes
Most vendor products can, within limits, be configured to support data and processes defined by user organizations. In fact, some vendors distinguish themselves competitively by promoting their solutions' flexibility and/or the ease of implementing this flexibility.
To evaluate these claims, it is worth knowing how customer-defined processes are typically implemented. Armed with a good understanding of a solution's flexibility, the implementation team either selects leading industry best practices or defines the organization's own desired processes. The team then identifies the data, screens, position of data on the screens, screen flow, and processes to support the custom design.
Message-based interfaces must also be considered. Data originating from other vendors' solutions and data originating from the solution that will support other systems and processes are also identified. Sometimes (hopefully not often), data must be exchanged among different solutions in real-time to complete a process, and these more complex arrangements must also be identified. For example, a registration process may need to verify a patient's identity with an external enterprise master person index in order to initiate the registration process.
Vendors use a variety of different mechanisms to customize a solution. Screen design tools and tables that specify data and process steps are common. All vendors provide mechanisms for defining interfaces, and health care providers frequently maintain their own external interface engines to support the customization of interfacing mapping between different vendor solutions.
Provider organizations should gain some understanding of these mechanisms before purchasing a solution. Not all vendors use the same mechanisms, and some are difficult to use and require long implementation times. Some even allow one to take an existing customization and manually re-enter it to an upgraded version of the software. If a solution is difficult and labor-intensive to maintain, its long-term costs will rise, and the IT department may ultimately be unable to support the solution users envisioned.
Other industries use workflow engine technology to map a solution's process flows to an organization's process flows. This technology is now starting to find its way onto the drawing boards of health care solutions. It is still too early, however, to know whether workflow engines will enhance the capabilities of, and help manage maintenance and upgrades for, health care solutions. The volume and complexity of the workflows found in large health care organizations have foiled previous attempts. Nevertheless, implementation, flexibility, and maintainability could all be improved if this technology were applied successfully.
Supporting organization-defined processes costs a considerable amount. Implementation and maintenance costs are visible to users; other, less-obvious, costs include product design, coding, testing, and performance. Customizability complicates the solution considerably and thus makes it more expensive. When customized, an entire "layer" of a solution's design is devoted to the exact data and screen flow that the user has specified.
Customization cuts down performance as well. As the solution is running, software decision logic is constantly following instructions to create the data and screen flows that the user sees. In programmer terminology, the vendor has created an interpreter that reads and follows instructions laid down during implementation. The system resources allocated to reading and following these instructions each and every time the solution is used are significant. If, by contrast, the vendor's solution had been hard coded for only one (or a very few) possible processes, the coding techniques would have produced a far more efficient product. This is why — all things being equal — less flexible products can do the same work as more flexible products at significantly lower costs.
Release Management
All vendors release periodic updates to their solutions. At a minimum, releases will correct deficiencies identified since the previous release. However, updates are not always thoroughly tested by vendors and may create new problems as they resolve old ones. In truth, releasing "100 percent tested" updates is no more possible than completely testing the original software.
Customers implement solutions in a wide variety of ways with different data, screens, process flows, and interfaces. Vendors simply cannot replicate and test their solutions with all the possible configurations customers might implement. Some vendors do a better job than others, but "100 percent tested" is not an achievable standard.
If a customer encounters a particularly critical flaw, the vendor may create a quick solution and send it to just that one customer. As a rule, these quick problem fixes should be incorporated into the next general product release. Sometimes they are and sometimes they aren't. Only good management in a vendor's development and support department will ensure that each release incorporates all the quick fixes issued since the previous release.
Receiving too many product updates can be as much of a problem as receiving too few. If a product has a lot of problems, then of course customer and vendor alike will want to see a remedy as soon as possible. On the other hand, a new release every month or so can be too much for customers to implement. Health care organizations must create dedicated hardware configurations for testing new releases in their environment before they are given to users. The hardware and management resources required for this testing are significant.
A software update to a complex clinical solution is never simple or cheap. Even after testing it, the IT department must plan the distribution of the updated software and any possible data conversions that must take place simultaneously.
Thick client/server solutions (which still dominate the industry) are particularly tricky to upgrade, as health care vendors still do not support a feature known as "version skew." Version skew would allow, for example, upgraded server software to simultaneously support workstations running either the new or the old version of the workstation software. Version skew support makes it possible to upgrade workstations slowly, in a controlled manner. Without version skew, all workstations and servers must be upgraded at the same time. This problem is typically resolved with technologies such as Citrix®, whose Metaframe® servers actually run workstation software on the Citrix server in the computer room. The workstations then simply run a browser that views and controls the software running on the Citrix server. This work-around adds considerable hardware and software licensing costs to any large implementation.
Data conversion requirements for updates — entailing extended downtime for the system — can create huge challenges and costs for a health care organization. Ironically, the reduction or elimination of paper in the clinical setting — one of the major advantages of automation — only intensifies these challenges. Hospitals are open 24 hours a day every day. An IT department cannot simply announce to the organization that their clinical information system will be unavailable for hours or days while a long data conversion takes place. But they do because vendors sometimes require it. The unintended consequence of downtime is greater resistance from clinicians to use and rely on the system as a whole.
Given their mission, health care organizations cannot tolerate a lack of access to critical clinical information, even for essential IT functions like data conversion. If health care organizations are ever to adopt automation wholeheartedly, vendors will have to develop implementation processes that allow data conversion while a system is serving end users. Health care organizations, for their part, may need to acquire duplicate hardware configurations to run their systems during upgrades and conversions. All of this additional effort will create extra costs for which provider organizations must be prepared.
Typical Shortfalls
Health care organizations contemplating the acquisition of a complex clinical system must understand the typical shortfalls in current vendor technology. It is not practical to wait until these problems are all resolved. Rather, buyers must understand which shortfalls apply to their solution and manage them. The most frequently found solution shortfalls are:
- Insufficient solution flexibility to support users' needs.
- Complex and difficult installation requirements.
- Difficult and expensive maintenance requirements.
- Burdensome hardware requirements for proper configuration.
- An inability to produce data for outbound interfaces.
- An inability to accept data from non-vendor systems through inbound interfaces.
- A lack of speed that irritates users and obscures the solution's value.
Summary
Health care buyers of clinical information solutions need to understand how vendors develop, implement, maintain, and update their software in order to make intelligent buying decisions. Often they are confronted with less-than-ideal alternatives, and they need to know which problems can be avoided, which ones can be mitigated through proper planning, and which ones need to be accepted and managed. The good news is that current solutions, if properly understood, can do a lot and will only get better with time.
