In the Techniques section:
Techniques
eHealth Project Risk Assessment: Design-Reality Gap Technique
"Is my e-health project likely to fail?"
This page offers one technique for answering this question by assessing project risk.
A gap exists for all e-health projects between the design assumptions/requirements and the reality of the client health organisation. The larger this gap between design and reality, the greater the risk that the project will fail. The risk assessment technique presented here asks you to rate the size of a set of design-reality gaps. Follow this link for further explanation about design-reality gaps (and some related case examples).
Assessing the Success/Failure Factors
Assessment consists of questions relating to a series of seven 'ITPOSMO' dimensions - information, technology, processes, objectives & values, staffing & skills, management systems and structures, and other resources - with attached rating numbers.
1. Using each of the seven ITPOSMO dimensions in turn, analyse two things. First, the organisational reality relating to that dimension that exists right now at the time of analysis. Second, the conceptions/requirements within the design of the e-health application.
2. For each one of the dimensions, give a numerical rating to indicate the size of the design-reality gap on that dimension. The rating for each dimension's gap can be anywhere on a scale from zero to ten. As a guide, illustrations are just given here for gaps corresponding to ratings of zero, five and ten, but all numbers in the range are possible. Illustrative ratings:
- 0 rating would indicate 'no change between the design proposal and current reality';
- 5 rating would indicate 'some degree of change between the design proposal and current reality';
- 10 rating would indicate 'complete and radical change between the design proposal and current reality'
3. Thus, for example, taking the first dimension - information - 0 would indicate that the information used in the e-health application was exactly the same as the information currently really being used in the healthcare organisation. 5 would indicate that the information used in the e-health application was somewhat different from the information currently really being used. 10 would indicate that the information used in the e-health application was completely and radically different from the information currently really being used.
The other six dimensions to be rated from zero to ten are:
- the technology used in the healthcare organisation (comparing the requirements contained within the design of the e-health application vs. the real situation now);
- the work processes undertaken in the healthcare organisation (comparing the processes needed for successful implementation of the e-health application vs. the real situation now);
- the objectives and values that key stakeholders need for successful implementation of the e-health application vs. their current real objectives and values;
- the staffing numbers and skill levels/types required in/by the healthcare organisation (comparing the requirements for successful implementation of the e-health application vs. the real situation now);
- the management systems and structures required in the healthcare organisation (comparing the requirements for successful implementation of the e-health application vs. the real situation now);
- the time and money required to successfully implement and operate the new application compared with the time and money really available now.
Presenting, Analysing and Using the Results
Overall Rating Analysis
The simplest and crudest thing you can do is add up the rating numbers for all seven ITPOSMO dimensions and interpret them according to the following table.
Overall Rating |
Likely Outcome |
57 - 70 |
Your e-health project will almost certainly fail unless action is taken to close design-reality gaps. |
43 - 56 |
Your e-health project may well fail unless action is taken to close design-reality gaps. |
29 - 42 |
Your e-health might fail totally, or might well be a partial failure unless action is taken to close design-reality gaps. |
15 - 28 |
Your e-health project might be a partial failure unless action is taken to close design-reality gaps. |
0 - 14 |
Your e-health project may well succeed. |
Individual Dimension Analysis and Action
The scores for each individual dimension can be presented using a table or a diagram arranged to show the gaps in size order from largest to smallest. Both are illustrated in the worked example below. The dimensions with the largest gaps are those that should be prioritised for action if risks of failure need to be addressed. Follow this link for further details about actions to take, plus links to real-world examples.
A slight variation is to use the weighting system described below, with the width of the arrow line related to the weight.
Variations on the Basic Technique
1. Who does it
These seven rating scales can be used by a single individual, such as a project consultant or project manager, to help them with their own understanding and recommendations. Alternatively, a more participative approach can be used. The seven scales can be presented to a group of key project stakeholders in a facilitated workshop. The stakeholders discuss and rate each dimension. The main problematic design-reality gaps are identified. The workshop would then move on to work out how best to close those gaps.
2. Weighted dimensions
The basic technique makes a questionable assumption - that all dimensions/gaps are equally important to the success and failure of the e-health project. A more complex variation would involve two rounds. In the first round, the risk assessment team would assign a weight to each of the dimensions. An 'ordinary' dimension might be given a weight of 1; a dimension that was considered 'important' in the particular e-health project could be given a weight of 2; and a dimension that was considered 'very important' in the particular project could be given a weight of 3. The weighting score would be multiplied by the rating to give an overall set of weighted ratings. For example, if 'objectives and values' were felt to be very important for this specific project, that dimension could be given a weight of 3. If the design-reality gap on that dimension to be only moderate, it could be given a rating of 5. The overall weighted rating for that dimension would be 3 x 5 = 15.
From experience, the objectives and values dimension should be given a higher weighting than other dimensions because it incorporates key elements such as politics, culture, self-interest, motivation, and the aspirations that a whole variety of different stakeholder groups seek to achieve from the new e-health system.
3. More complex dimensions
The use of just seven rating scales is very much a 'blunt instrument'. A more sophisticated - also more time-consuming - approach is to break each main dimension down into a series of sub-dimensions. Each sub-dimension is then allocated its own rating scale. For instance:
- The 'technology' dimension could be broken down into three sub-dimensions: software, hardware and networks.
- The 'staffing and skills' dimension could be broken down into one sub-dimension for each significant staff grouping involved and/or one sub-dimension for each of the six key e-health competencies (strategic, change/project management, information systems development and management, hands-on, interpersonal, 'intelligent customer' (contracts, suppliers, procurement)).
Such sub-dimensions can either be pre-set or they can be determined within a facilitated workshop. In the latter case, sub-dimensions can be attuned to particular organisational context.
4. Creating your own dimensions
This 'attuning' just mentioned can go further: stakeholders can use the seven suggested ITPOSMO dimensions merely as a starting point for discussion, and can then develop their own particular dimensions and sub-dimensions that are seen to be relevant to the specific context. Design-reality gaps can then be assessed for each one of those dimensions/sub-dimensions.
5. Incorporating drivers
Design-reality gaps can be thought of as constraints or risks to implementation of an e-health project: they give a sense of what may make the project fail. They may not give a good sense of what may make the project succeed: the drivers. The drivers can be analysed as well, and illustrated alongside the gaps/constraints/risks using a force-field diagram with drivers on one side and constraints on the other.
6. Process as the focus
In the basic and variant approaches described above, two things can be borne in mind:
- The outcome: a sense of gap between design and reality.
- The process: the deeper understanding - of reality, of design, of other stakeholders - that gap analysis creates.
In some situations, the process may be more valuable than the outcome. It may then be appropriate to take a more iterative, learning approach to gap analysis. Here, stakeholder groups revisit gap analysis at regular intervals during the project cycle. They reflect on the dimensions selected, the ratings and the closure techniques. They also reflect on what has been learned about the project and the e-health implementation process.
Pros and Cons of this Technique
This technique is relatively simple and quick to understand and put into practice. One key advantage is that it matches the unique situation of each individual e-health project, rather than imposing a "one size fits all" concept. On the downside, it tries to cram a lot of issues into each single dimension (particularly into 'objectives and values' and 'staffing and skills'), and it will not work well if there are competing designs or competing ideas about what counts as 'reality'.
Worked Example
A new Web-based procurement system is being implemented by the Memorial Hospital in Gedactia, to support the purchase of medical supplies. Introduction of the system is being promoted and partly-funded by an external donor, which has put in place many of the formal skills and technology required, but the project has relatively little internal support. Is this e-health project likely to succeed or fail? An assessment and answer are given below.
Questions, Answers & Ratings
Information
Question: What is the gap between the information assumptions/requirements of the new e-procurement system design, and the information currently in use in reality in the Hospital?
Answer: The project consultants have made use of a fairly 'generic' design for the e-procurement system. In reality, this matches some core elements of information currently in use in Gedactian medical supplies procurement. However, the Hospital currently makes use of slightly different information to this 'one size fits all' assumption. In reality also, there are shortcomings in availability of information that the design assumes will be present - a list of all healthcare suppliers, accurate pricing information, and a clear set of guidelines on procurement. Thus there is a fair-sized gap between the information assumptions of the design and current realities.
Gap rating: 6.5
Technology
Question: What is the gap between the technology assumptions/requirements of the new e-procurement system design, and the technology currently in use in reality in the Hospital?
Answer: The e-procurement system design assumes the presence of a set of robust Internet connections, Web servers, and procurement software within the Hospital; it also assumes the presence of Internet-connected systems in a broad range of suppliers. In reality, Memorial Hospital currently makes fairly limited use of ICTs, the telecommunications infrastructure in the country is somewhat limited, and some healthcare suppliers lack access to ICTs.
Gap rating: 7
Processes
Question: What is the gap between the work processes required for successful implementation of the new e-procurement system design, and the work processes currently in use in reality in the Hospital?
Answer: The e-procurement system design requires a set of formal, rational work processes that deal efficiently with procurement. These proposed work processes under the new system design follow roughly the same lines as the current system, and that current system does function, but with a number of 'hiccups' and inefficiencies in the way that work is carried out.
Gap rating: 2.5
Objectives and Values
Question: What is the gap between the objectives and values that key stakeholders require for successful implementation of the new e-procurement system design, and their current, real objectives and values?
Answer: The e-procurement system design assumes a procurement system that values rational functioning within hospitals, such as freedom of procurement from political or personal interventions. The design assumes objectives of greater efficiency (whatever the impact on jobs), and of the spread of e-health systems. The reality is somewhat different, though it varies from stakeholder to stakeholder. The donors - who are driving the project - largely share these objectives and values; as do the project consultants and IT suppliers working for the donors. Many senior Hospital officials do not share them: they are either happy with the status quo or they have other priority objectives than e-procurement, they support a politicised/personalised rather than rational culture within the Hospital, and they are not particularly keen on the spread of ICTs. Many clerical staff working in Memorial Hospital similarly do not share the design objectives and values: they fear the new system and they cannot see its value.
Gap rating: 7.5
Staffing and Skills
Question: What is the gap between the staffing numbers and skills levels/types required for successful implementation of the new e-procurement system design, and current, real staffing and skills?
Answer: The e-procurement system design assumes the presence of a whole range of competencies for both its implementation and its ongoing operation. For example, it assumes a reasonable-sized team with good experience of designing and implementing e-procurement systems; it assumes good knowledge within that team of healthcare and Hospital specificities; it assumes some capacities within the Hospital to manage the implementation contract and the procurement system; it assumes a set of hands-on IT skills among clerical staff in the Hospital. In reality, some of these competencies are present and some are not. The project team has good experience, but knows little about healthcare/Hospital specifics; the Hospital has a limited set of management experitise; and clerical staff have a few basic IT skills but not the higher-level skills that operation of the Web-based system will require.
Gap rating: 6
Management Systems and Structures
Question: What is the gap between the management systems and structures required for successful implementation of the new e-procurement system design, and current, real management systems and structures?
Answer: The e-procurement system design assumes some limited changes to management systems compared with current reality, with the introduction of some IT management of the Web systems, and some changes to oversight mechanisms for procurement. The design assumes no significant changes to Hospital structures.
Gap rating: 2.5
Other Resources
Question: What is the gap between the other resources (money, time, other) required for successful implementation of the new e-procurement system design, and current, real availability of those resources?
Answer: The e-procurement system design assumes two sets of financing to be available. First, a larger sum for introduction of the system; second, a smaller ongoing sum for system operation and maintenance. In reality, the donor is making available the money for the first set, and for the second for the first two years. The design also assumes a relatively gentle timescale, using an incremental approach in roll-out of the system. This seems to match fairly well with the amount of time that staff have available (and that political timescales impose).
Gap rating: 2.5
Table of Results
Dimension |
Rating |
Information |
6.5 |
Technology |
7 |
Processes |
2.5 |
Objectives & Values |
7.5 |
Staffing & Skills |
6 |
Management Systems & Structures |
2.5 |
Other Resources |
2.5 |
Overall Total |
34.5 |
Diagram of Gaps
Conclusions and Action
Given the overall rating of 34.5, the project is at some risk of failure unless action is taken. The main actions needed are to reduce the largest design-reality gaps (i.e. starting at the top of the gap diagram and working downwards, trying to identify ways in each case to make design more like reality and/or to make reality more like design). Follow this link for further details about actions to take, plus links to real-world examples.