London Ambulance Service Failure

The London Ambulance fiasco ? The London Ambulance Service (LAS) Computer Aided Despatch (CAD) system failed dramatically on October 26th 1992 shortly after it was introduced: †¢ †¢ †¢ The system could not cope with the load placed on it by normal use; The response to emergency calls was several hours; Ambulance communications failed and ambulances were lost from the system. ? A series of errors were made in the procurement, design, implementation, and introduction of the system.  ©Ian Sommerville 2004 Software Engineering Case Studies Slide 1 London Ambulance Service ? ? Managed by South West Thames Regional Health Authority. Largest ambulance service in the world (LAS inquiry report) †¢ †¢ †¢ †¢ Covers geographical area of over 600 square miles Resident population of 6. 8 million people (greater during daytime, especially central London); Carries over 5,000 patients every day; 2,000-2,500 calls received daily, of which 1,3001,600 are emergency calls.  ©Ian Sommerville 2004 Software Engineering Case Studies Slide 2 Computer-aided despatch systems ? Provide one or more of the following: †¢ †¢ †¢ Call taking; Resource identification; Resource mobilisation; Ambulance resource management. CAD software & hardware; Gazetteer and mapping software; Communications interface (RIFS). Radio system; Mobile data terminals (MDTs); Automatic vehicle location system (AVLS). ? Consist of: †¢ †¢ †¢ †¢ †¢ †¢  ©Ian Sommerville 2004 Software Engineering Case Studies Slide 3 T he manual system to be replaced ? Call taking †¢ Recorded on form; location identified in map book; forms sent to central collection point on conveyor belt; Form collected; passed onto resource allocator depending on region; duplicates identified. Resource allocator decides on which resource to be mobilised; recorded on form and passed to dispatcher; Dispatcher telephones relevant ambulance station, or passes mobilisation instructions to radio operator if ambulance already on road; ? Resource identification †¢ ? Resource mobilisation †¢ ? Whole process meant to take < 3 minutes. Software Engineering Case Studies Slide 4  ©Ian Sommerville 2004 Concept/design of the CAD system ? Existing systems dismissed as inadequate and impossible to modify to meet LAS’s needs †¢ Intended functionality â€Å"greater than available from any existing system†. To consist of Computer Aided Dispatch; Computer map display; Automatic Vehicle Location System (AVLS); Must integrate with existing MDTs and RIFS (Radio Interface System). Near 100% accuracy and reliability of technology; Absolute cooperation from all parties including CAC staff and ambulance crews. ? Desired system: †¢ †¢ ? Success dependent upon: †¢ †¢  ©Ian Sommerville 2004 Software Engineering Case Studies Slide 5 Problems: Procurement (i) ? Contract had to be put out to open tender †¢ Regulations emphasis is on best price; 35 companies expressed interest in providing all or part of the system †¢ Most raised concerns over the proposed timetable of less than 1 year until full implementation. ? Previous Arthur Andersen report largely ignored †¢ †¢ Recommended budget of ? 1. 5M and 19 month timetable for packaged solution. Both estimates to be significantly increased if packaged solution not available; Report never shown to new Director of S upport Services. ? Only 1 out of 17 proposals met all of the project team’s requirements, including budget of ? 1. 5M. Software Engineering Case Studies Slide 6 Ian Sommerville 2004 Problems: Procurement (ii) ? Successful consortium †¢ †¢ †¢ Apricot, Systems Options (SO), Datatrak; bid at ? 937k was ? 700k cheaper than the nearest bid; SO’s quote for the CAD development was only ? 35k †¢ Their previous development experience for the emergency services was only for administrative systems. Ambiguity over lead contractor. Systems manager: Career ambulance man, not an IT professional, already told that he was to make way for a properly qualified systems manager; Analyst: Contractor with 5 years experience working with LAS. ? 2 key members of evaluation team: †¢ †¢  ©Ian Sommerville 2004 Software Engineering Case Studies Slide 7 Problems: Project management ? Lead contractor responsible †¢ †¢ Meant to be SO, but they quickly became snowed under, so LAS became more responsible by default; No relevant experience at LAS or SO. ? ? Concerns raised at project meeting not followed-up. SO regularly late in delivering software †¢ Often also of suspect quality, with software changes put through ‘on the fly’. ? ? Formal, independent QA did not exist at any stage throughout the CAD system development. Meanwhile, various technical components of the system are failing regularly, and deadlines missed. Software Engineering Case Studies Slide 8  ©Ian Sommerville 2004 Problems: Human resources & training (i) ? ? ? ? ? Generally positive attitude to the introduction of new technology. Ambiguity over consultation of ambulance crews for development of original requirements. Circumstantial evidence of resistance by crews to Datatrak equipment, and deliberate misleading of the system. Large gap between when crews and CAC staff were trained and implementation of the system. Inability of the CAC and ambulance staff to appreciate each others’ role †¢ Exacerbated by separate training sessions. Software Engineering Case Studies Slide 9  ©Ian Sommerville 2004 Problems: Human resources & training (ii) ? ? ? ? ? ? ? Poor industrial relations. Management ‘fear of failure’. CAD system seen as solution to management’s desire to reduce ‘outdated’ working practices. System allocated nearest resource, regardless of originating station. System removed flexibility in resource allocation. Lack of voice contact exacerbated â€Å"them and us†. Technical problems reduced confidence in the system for ambulance crews and CAC staff.  ©Ian Sommerville 2004 Software Engineering Case Studies Slide 10 System problems Need for near perfect information †¢ Without accurate knowledge of vehicle locations and status, the system could not allocate optimum resources. There were numerous possible reasons for incorrect information being passed back to the system. Numerous technical problems with the system, including: †¢ Failure to identify all duplic ated calls; †¢ Lack of prioritisation of exception messages; †¢ Exception messages and awaiting attention queues scroll off top of screen. ? Poor interface between crews, MDTs & the system †¢ ? Unreliability, slowness and operator interface †¢  ©Ian Sommerville 2004 Software Engineering Case Studies Slide 11 Configuration changes ? Implementation of the system on 26 October involved a number of significant changes to CAC operation, in particular: †¢ †¢ †¢ †¢ †¢ †¢ †¢ †¢ Re-configuring the control room; Installing more CAD terminals and RIFS screens; No paper backup system; Physically separating resource allocators from radio operators and exception rectifiers; Going ‘pan London’ rather than operating in 3 divisions; Using only the system proposed resource allocations; Allowing some call takers to allocate resources; Separate allocators for different call sources. Ian Sommerville 2004 Software Engineering Case Studies Slide 12 So, what happened? ? ? Changes to CAC operation made it extremely difficult for staff to intervene and correct the system. As a consequence, the system rapidly knew the correct location and status of fewer and fewer vehicles, leading to: †¢ †¢ †¢ †¢ Po or, duplicated and delayed allocations; A build up of exception messages and the awaiting attention list; A slow up of the system as the messages and lists built up; An increased number of call backs and hence delays in telephone answering.  ©Ian Sommerville 2004 Software Engineering Case Studies Slide 13 Why did it fail? ? Technically, the system did not fail on October 26th †¢ †¢ Response times did become unacceptable, but overall the system did what it had been designed to do! Failed 3 weeks later due to a program error – this was a memory leak where allocated memory was not completely released. Management; Union; System manager; Government. ? It depends who you ask! †¢ †¢ †¢ †¢  ©Ian Sommerville 2004 Software Engineering Case Studies Slide 14 Lessons learned ? Inquiry report makes detailed recommendations for future development of the LAS CAD system, including: †¢ †¢ †¢ †¢ †¢ †¢ Focus on repairing reputation of CAD within the service; Increasing sense of ‘ownership’ for all stakeholders; They still believe that a technological solution is required; Development process must allow fully for consultation, quality assurance, testing, training; Management and staff must have total, demonstrable, confidence in the reliability of the system; Any new system should be introduced in a stepwise approach.  ©Ian Sommerville 2004 Software Engineering Case Studies Slide 15