Fault Tolerance & Redundancy

Risk Management is now Disaster Preparedness

 

Risk management has been around for a long time although many didn’t realize it until after the September 11 tragedy. Now referred to as disaster preparedness by most, risk management is important for any business. Of course, law firms face many different aspects of it, some of which are not found in other small businesses. While law office risk management issues cover a broad spectrum, I wish to focus on the technological aspects in this article.

 

Why is it important to my law firm?

 

            Disaster preparedness in the technology arena includes telephone systems, fax machines, copiers, and, of course, computer systems. It involves assessment of the potential disruptions that failure of critical systems will impose if they occur. It also concerns the costs necessary to avoid these unwanted negative consequences. While it is possible to create fail-safes for most situations, the costs can be unacceptable. Consequently, the risks must be weighed and decisions made. These decisions are the essence of disaster preparedness.

            The best way to look at fault tolerance is to view it as another form of insurance. There are few true productivity benefits involved. Of course, if disaster should strike, then the benefits are tangible and far exceed the costs. However, peace of mind is the true benefit of fault tolerance.

            Lets look at why disaster preparedness should be important to you. Imagine what would happen to your law firm if the telephone system stopped operating? It would paralyze your office, wouldn’t it? How much is it worth to ensure that it doesn’t happen? Quite a bit, I would wager.

Fortunately, telephone systems are highly dependable. This makes failure unlikely. Couple that with the high cost of purchasing and installing a backup system and it is clear that few law offices would choose to implement a backup telephone system.

Fax machine have the hiccups? While not nearly as bad as losing the telephone system, still a painful situation. Luckily, replacements are easy to find and not that expensive. This, combined with the relatively minor impact of failure, leads most firms to choose to forego a backup fax machine. Forward thinking firms who install a network fax software enjoy the benefit of using their standalone fax machine as a backup.

Losing a copier wouldn’t be any fun either. While generally more painful than a fax machine loss, placing a replacement in service can usually be done within hours near a metropolitan area. Despite this, many law firms, especially those in rural areas, have decided to keep their old copier as a backup. Again, forward thinking firms are implementing high speed scanning solutions for archival purposes but realize this provides copier backup via scan and print technology.

Finally, we have the computer system. Until recently, computers were used primarily as fancy word processors and accounting ledgers. As law firms move toward case management, the Internet, litigation support, and a host of other legal programs, failure of the computer system becomes more critical. Today, many firms would suffer at least as much from the failure of their computer system as from the failure of their telephone systems.

One problem is that computers are not yet as dependable as the telephone system. It is not as easy or inexpensive to replace as a copier or fax machine. Remember, replacing computers involves not only hardware costs but the associated labor costs of setup. Consequently, this is a prime area to consider implementing some type of fail-safe system.

 

What is fault tolerance and redundancy?

 

            Fault tolerance is one type of fail-safe that uses redundancy and other techniques to accomplish its purpose. The key to successful fault tolerance involves uniqueness. A fault tolerant backup to a system will not be susceptible to the same weaknesses as the main system. A good example is the belt and suspenders. They perform the same function but most of the causes of failure for one will not affect the ability of the other to continue to perform its function.

Redundancy is simply the addition of multiple devices in order to keep the system operating if one device fails. For instance, having two belts instead of one. Or a spare suit for a one day business trip.

There are a number of areas of concern when looking at fault tolerance and redundancy. Let’s take a look at the major ones.

           

Network backbone

 

The network backbone consists of all hubs, switches, routers, network cards, and wiring. Except for the network card, failure of any of these components can bring down the entire system. Failure of a network card will bring down only the computer in which it is installed.

The hub is an item that may be worth duplicating. They are inexpensive yet not easy to quickly acquire. The spare can either be installed in the system or kept on a shelf.

Switches, on the other hand, are more expensive. It would not generally be cost-effective to keep a spare switch on hand. It would, however, be cost effective to keep a spare hub on hand that could temporarily be swapped out for the switch until another can be acquired.

Routers are not yet common in most law firms. This will gradually change as more firms establish full time Internet connections. Routers are the best way to protect your internal network from hackers and other outside threats. Due to price drops, it could be cost-effective to keep multiple router units on hand in a rural area.

The best way to ensure redundancy for wiring is to install multiple (at least two) connections in each office space. If one goes out you simply plug the computer into the other one. Also, make sure that spare patch cables are kept on hand.

Network cards are the most common area of failure. The only computer which needs dual network cards is the Server. A Server loss shuts down the entire network whereas a lost workstation is significant but not disastrous.

If a decision is made to use dual network cards, then use two separate cards instead of one card with dual circuitry. Some cards have two connections but don’t duplicate all of the circuitry so a failure will knock out both ports.

A fault tolerant and redundant solution for wiring and network cards is the concurrent installation of a wireless network for the LAN. They can both run simultaneously and if the wiring infrastructure goes out the wireless will take over.

 

Server power supplies

 

Computers run on Direct Current (DC). The power supplied via your wall outlet is Alternating Current (AC). In order for your computer to operate, a device called a power supply is used to transform AC to DC. If this power supply fails then the computer stops operating.

This is another situation where the Server is the primary concern. Many Servers are ordered with dual power supplies. Typically, the cost is $500 or less for adding this option. It only takes about half an hour to swap out a power supply but acquiring another one can take 24 to 48 hours.

 

System Data

 

This is the most important of the various areas of concern. Most of the concern lies with the Server hard drive data so lets start there. Among the options to consider are how many hard drives, what type of RAID to implement and whether hot-swappable capability is desired. Other options include backup systems discussed in a previous article of the CLO a few months ago.

RAID stands for Redundant Array of Independent (or Inexpensive) Disks. RAID can be either hardware or software driven. Windows NT has the software capability built into it but this utilizes a great deal of system resources and slows down data transfer. For this reason it is better to use a hardware solution. Fortunately, the cost of these systems has dropped considerably in the last year.

There are many different RAID levels available but most law firms need only consider two of these. RAID level 5 is the most common but requires three hard drives. If three 9 GB drives are used, then RAID 5 will provide 18 GB of usable storage space.

RAID level 0 is basically just a mirroring technology but requires only two hard drives. If two 9 GB drives are used then 9 GB of usable storage space is available. This wastes more space than RAID 5 so if you can afford three hard drives then stick with RAID 5.

The wonderful thing about RAID is, even if one of the hard drives fails, none of the data is lost. Simply replace the failed drive and the other ones will write the information back to the new drive. To make this even better, hot swapping capability can be added.

Hot swappable hard drives allow the hard drives to be replaced without even shutting down the Server. A message will inform the administrator that one of the drives has failed and the drive can be exchanged. Once this is done the data is copied back to the new drive and business can continue as usual.

The use of RAID should be combined with a tape back-up system to provide comprehensive protection for any eventuality.

 

 

BDC – Backup Domain Controller

 

            Another technique is to use a somewhat older computer as a BDC. This is often a replaced workstation which acts as a backup to the server. It mirrors all of the system settings, passwords, configuration, data, etc. which is located on the PDC – Primary Domain Controller server. In the event the PDC goes down, the BDC promotes itself automatically to a PDC and the system continues to operate. Due to the low cost of modern computers this is now a feasible option.

 

Backups

 

            Backups are essential but are often complex and beyond the capability of the personnel in a small law office. Unfortunately, many firms find, to their dismay, that backups are not yet at the ‘set and forget’ stage of development. You must monitor the logs every day to ensure the system is operating. You must perform test restore procedures to verify the tapes are good. If all of this double checking of the backup seems overly time consuming and complex then consider another option.

            Due to the proliferation of high speed Internet access, the online backup model is making a big comeback. It was first attempted using dial up Internet access and was a big flop as connections were slow and unstable. With xDSL or Cable it is emerging as a cost effective and reliable solution.

 

 

Who needs this type of technology?

 

This is NOT technology reserved for big law firms. Even a well computerized solo practitioner will suffer substantial costs if the network fails for even a few days. As the dependence on the computer system increases, so does the cost and the risk.

Generally, if your law firm has a computer network and relies on programs other than word processing, then you need it. The specific level of need is determined by your particular situation. An expert needs to evaluate your situation and recommend a solution that meets this need.

 

Where should I use fault tolerance in my law office?

 

The best place to start is with the Server. Hopefully, you have a network in place and store all of your critical information on the Server. There should not be any critical data stored on the workstation itself. Consequently, RAID can be reserved for Servers thus sparing us that expense for workstations.

Fortunately, most of the big players in the direct mail order PC business have added RAID to their Server lineup. Dell, Compaq, Gateway, Micron, etc., all offer hardware RAID machines along with all of the other protections discussed above.

The additional cost of RAID and the other techniques above is around $2000-$3000. Compared to the $10,000.00 we were willing to spend ten years ago on previous editions of the 80x86 technology, this really amounts to a fantastic opportunity to provide substantial protection for your computer system at a bargain price. Additional options like dual network cards or power supplies will add a few hundred dollars but are still very affordable.

A big benefit to ordering all of these items together is compatibility and technical support. Trying to add some of these components to existing hardware can be difficult because of unknown incompatibilities. Even excellent products do not always work well together. Let the manufacturer bear the cost of compatibility testing.

Obtaining vendor tech support, i.e. the manufacturer support, not your local support, can quickly degenerate into something similar to the “Empty Chair” defense common in third party litigation. The problem always seems to be the other vendor’s hardware or software. Of course, you don’t really care whose problem it is, you just want it fixed. Avoid this situation by buying all hardware from one source.

 

How do I start?

 

            The best way to begin is to assess your current situation. What type of fail-safes are built in to your current system? What fail-safes should there be? This may require outside help to evaluate the computer system and establish your current needs.

Next, create a long term technology plan and budget. Again, this may require outside assistance. The technology plan should be an integral part of the firms overall disaster preparedness plan.

The last, but most difficult, step is to purchase and install the new hardware. This will definitely require outside help for the vast majority of law firms.

 

When should I consider moving to better fault tolerance?

 

As with all of your purchasing considerations, you must prioritize. The time frame you choose should be based upon available financing, the extent that technology is currently employed by the firm, and the value of the peace of mind that this technology provides. How much is it worth to rest easy knowing that your law firm is protected? Only you can answer that.

 

A short quiz


Let’s see how prepared you really are by answering a few easy questions.

 

1)      If all of the computers in your office were stolen or destroyed over the weekend:

a)      do you have all the serial numbers of the equipment,

b)      the original cost of the equipment,

c)      the value of the equipment,

d)      could you still prepare work product with any kind of efficiency, and

e)      how long would it take you to:

i)        contact all of your clients,

ii)       contact your insurance company,

iii)     contact opposing counsel,

iv)     determine your calendar,

v)      place billing and accounting back on-line and

vi)     generally get back to business?

2)      To what extent do the answers to d and e above revolve around the time and ability to recreate all your computer data?

3)      If you, or your staff person, had a major illness or accident:

a)      are your client data files organized so that someone could find them without major difficulty?

b)      have you documented your cases in such a way that someone could handle them for you on short notice?

4)      To what extent does the answer to 3 above revolve around the information located on your computer system?

 

Conclusion

 

Fault tolerance  and redundancy are important parts of the overall disaster preparedness plan. The specifics will vary according to the particulars of your law firm. The common thread is that every law firm needs to take the time to plan in order to be protected from disaster, or at the very least, to be prepared if, or when, disaster strikes.