P. TIMMER and J LONG

Ergonomics & HCI Unit, University College London, 26 Bedford Way, London, WC1H 0AP

Expression de l’efficacité des horizons de planification

RÉSUMÉ

L’évolution technologique est souvent motivée par des ‘problèmes’. Pourtant, l’expression de ces problèmes en termes de performance des systèmes de travail n’est souvent qu’anecdotique ou implicite. Cette recherche propose une méthode explicite pour exprimer l’efficacité d’un système de travail. La méthode est illustrée sur un système de travail de gestion du transport. L’intérêt particulier de ce domaine concerne la façon dont l’interaction opérateur-technologie soutient efficacement la planification à l’avance (sous la forme d’un horizon de planification). La méthode est composée de quatre étapes. Premièrement, le comportement de planification à l’avance est conceptualisé. Un aspect critique de la méthode est la ‘Théorie des Horizons de Planification de l’Opérateur’ ainsi que ‘l’extension’ et ‘l’opportunité’ d’horizons de planification particuliers. Deuxièmement,  le domaine de travail est modélisé, afin d’établir la qualité du travail effectué par le système de travail. Troisièmement, les comportements qui soutiennent la planification efficace sont modélisés. Finalement, une comparaison est faite entre la qualité réelle du travail effectué et la qualité désirée. Si la performance tombe en-dessous d’un niveau désiré, les comportements du système de travail contribuant à l’inefficacité sont analysés. Si une planification inefficace est identifiée (c’est-à-dire un problème), la méthode soutient la recherche des origines du problème, ainsi que la construction d’une théorie causale. Bien que l’illustration ne porte que sur la planification d’un système de travail de gestion du transport, les étapes de la méthode sont proposées pour soutenir plus généralement l’expression de l’efficacité des systèmes ou autres.

 

Mots-clés:            Horizon de Planification; Efficacité d’un système de travail;  Contrôle du trafic aérien.

 

 

 

 

 

 

 

I.         INTRODUCTION

 

I. 1.            MOTIVATION AND OVERVIEW

This paper describes a method that enables the expression of: a) the plans of a process operator, and how far into the future those plans extend; and b) an assessment of how adequate those plans are, for ensuring that work goals are attained. The method is illustrated using an Air Traffic Management microworld. The need to express operator plans, and their extension, arises when technologies are being developed to support the planning of interventions with a dynamic evolving process; domains where process evolution needs to be anticipated, and process interventions need to be planned to address anticipated process states. By associating each individual plan with an assessment of adequacy, design problems may be characterised, problems that may be alleviated by technological support. Where the expression of plans shows those plans to be inadequate for attaining management goals, then new technologies can be proposed that may result in more effective operator planning behaviour.

 

In general, the evolution of a human-technology worksystem may either be problem-driven or technology-driven (Woods & Roth, 1988). Problem-driven evolution arises when a specific problem is attributed to the design of a technology, and the technology is then redesigned (or replaced) to remedy that specific problem. Such problems are frequently expressed by operators in the form of a subjective, experience-based report or anecdote, which may or may not lead to some in-depth analysis of the problem. Alternatively, technology-driven evolution may arise when a technology is redeveloped (or replaced) simply because redevelopment (or replacement) is possible, that is to say, not in the light of a specific ‘problem’ (such as inadequate planning). As the aim of the method proposed here is to relate operator planning (and plans) to effective or ineffective performance (intervention outcomes), and given that ineffective performance is considered a problem in need of a (technological) solution, the method may be understood as a system development tool, to support problem-driven evolution at the early stage of ‘problem formulation’ (Rasmussen, 1992; Woods & Roth, 1988). The method therefore assists in the process of progressing from operator anecdotes about problems, to a structured and more formal analysis and expression of those problems. The need to express planning problems, and thereby evolve technological solutions, arises in traditional process control domains such as Air Traffic Management (ATM), Railway Signal Management (RSM), nuclear power generation, and so forth. Throughout the paper, method application will be illustrated with reference to an ATM-like microworld of sufficient complexity to demonstrate the phenomena of concern to the method.

 

Before continuing, a definition of the term ‘design problem’ is offered.  A design problem is considered to exist, and therefore acts as a motivator for problem-driven evolution, when a desired level of performance is not being achieved by a human-technology worksystem. That worksystem may then be termed ‘ineffective’, as desired performance is somehow compromised. Where a design problem is believed to exist, a method is needed for expressing that problem (its severity, frequency, behavioural causes, etc.) in a manner that will contribute towards its solution. Outlining such a method is the aim of this paper. Being able to express worksystem ineffectiveness is particularly valuable when problem-driven evolution occurs within safety critical domains. For domains where the consequences of ineffective human-technology interaction have serious potential outcomes for human life, being able to express whether or not particular interactions are effective, supports reasoning about the adequacy of the technology in question. Within the context of Cognitive Engineering, such expression and reasoning is termed ‘diagnosis’ (Dowell & Long, 1998; Rasmussen, 1986) and, as shall be illustrated in the paper, can support formulation of the design problem that a re-designed technology should solve (Dowell, 1998).

 

The emphasis on ‘design’ and ‘design problems (and solutions)’ characterises the present approach as one of engineering, that is, contributing to the design of effective worksystems (Amalberti & Deblon, 1992; Dowell & Long, 1998; Flach, 1998; Hollnagel, 1998; Rasmussen, 1986; Reason, 1998; Vincente, 1998; Woods, 1998), rather than as one of science, that is, understanding the phenomena associated with worksystems and their behaviours (Barnard, 1991; Meyer & Kieras, 1999). Within the design approach, the present can be more precisely characterised as ‘design for effectiveness’, seeking to use the design primitives of ‘work’, ‘worksystem’, and ‘performance’ to motivate the acquisition and validation of design knowledge, to diagnose and solve design problems (in contrast, for example, to ‘human performance’, expressed as some form of speed and errors (Reason, 1998)).

 

Problems of interaction arise with many different types of technology. Here, particular consideration is given to problems arising with technologies that support planning ahead. In presenting the method, at the first stage, a theory is presented, for use when modelling how far ahead an operator plans. In contrast to other work on planning (Amalberti & Deblon, 1992; Boudes & Cellier, 1997; O’Hara & Payne, 1999), this theory makes reference to a plan’s ‘extension’, and its ‘adequacy’ (how well the plan ensures work goals are achieved). A method for expressing the effectiveness (or otherwise) of a plan necessarily requires consideration of the plan’s extension (how far into the future the plan accounts for an intervention), and whether or not the plan is adequate to ensure goals are attained.

 

The aim of this paper is therefore to propose a method for expressing the adequacy of operator planning, with special interest in capturing instances of ineffective planning (diagnosing design problems). The presented method comprises four stages. In the first stage, planning behaviours of interest are conceptualised by a domain-independent Theory of the Operator Planning Horizon, and requirements for modelling an operator’s planning horizon are generated from that theory. During the second stage, the work carried out by the human-technology (planning) worksystem is considered, and captured by a model of the domain. Here, the ATM-like microworld is described. In the third stage, models of operator planning horizons are constructed. Finally, these two types of model, of the domain (work) and of planning horizons (plans and planning behaviour), are considered alongside each another, and diagnostic assessments are made as to whether or not the plans formed were effective (i.e., whether or not there is a design problem concerning planning). When a problem is identified, a causal theory is constructed to relate how operator-technology interactions contributed to the occurrence of that problem. The four stages together constitute a method that helps to establish an explicit link between planning behaviours and the quality (effectiveness) of work executed by the worksystem, a relation frequently addressed in only an implicit fashion (Boudes & Cellier, 1997). The method therefore addresses the construction and use of models during the design process, a tradition well established within HCI research (Blandford & Young (1993); Card, Moran & Newell (1983)). In the analysis of operator-technology interactions, ‘effectiveness’ is considered a primitive (ontological) entity, alongside: human (planning) behaviour; technological behaviour (and how it supports planning); and details of the work performed (Dowell & Long, 1998). The paper is structured to reflect the stages of the method outlined above, and concludes with a discussion of the method.

 

I.2            ATM-LIKE MICROWORLD

For the purposes of method illustration, a laboratory-based ATM-like microworld was used. The microworld was constructed on the basis of an observational field study at Ringway Control Centre in Manchester, and possesses selected characteristics of the operational system that make it suitable for illustrating the method (Dowell, 1998; Long & Timmer, 2001). The domain is dynamic and imposes a significant planning burden upon the operator, who must anticipate the future state of air traffic, establish goals, collect and integrate data from different sources, plan tactical interventions to two aircraft variables (altitude and speed), and finally intervene with aircraft. Aircraft response to worksystem intervention is time-lagged, and the quality of aircraft management, in terms of aircraft safety and expedition (fuel use, progress to plan, minimum interventions), is calculated by the simulation software, following an interactive scenario.

 

The managed domain, containing aircraft, beacons, airways and so forth (Dowell, 1998), is generated by the simulation software and displayed on a computer-based radar. Paper-based flight progress strips are used, as observed in the operational worksystem, and document aircraft entry and exit states, aircraft identity and route. The strips are annotated after each intervention with details of aircraft state changes. Interventions to aircraft altitude and speed can be made via menu selection on the radar, rather than by ground-to-air radio. Through interaction with these technologies, the operator’s task is to ensure the safe and expeditious management of aircraft across the sector. Aircraft traverse a sector along airways, moving from an entry beacon to an exit beacon, via an intermediate beacon (see Figure 1). Between eight and ten aircraft are managed during a scenario. Operators are naïve subjects, trained in the management procedures necessary to ensure aircraft safety and expedition, and with some practice in the management task.

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 1. Topographic representation of the simulated sector

Figure 1 Représentation topographique du secteur simulé

 

 

While the simulation constitutes a considerable simplification of operational ATM, it is sufficient for method illustration. Planning can be observed, of interventions to (process) object variables, and is available for analysis. Likewise, desired levels of worksystem performance can be specified, actual levels of performance measured, and effectiveness or ineffectiveness thereby expressed. Providing these criteria are met, it is proposed that the method may be applied to other domains (operational or microworld). Having considered the microworld of interest, the method is presented in four stages. First, consideration is given to conceptualising the operator planning horizon. Second, the work carried out by the microworld operator is considered (Stage 2), after which models of planning horizons are constructed (Stage 3). Finally, ineffectiveness is diagnosed (Stage 4).

 

II.         THEORY OF THE PLANNING HORIZON

 

II.1            PLANNING SCOPE

The term ‘planning’, within the Cognitive Ergonomics / Science / Psychology literature is used to refer to a wide range of human operator behaviours: problem-solving (O’Hara & Payne, 1999); task scheduling (Hayes-Roth & Hayes-Roth, 1979); decision-making (Pietras & Coury, 1994); or anticipation (Boudes & Cellier, 1997, 1998). Additionally, planning can refer to an immediate action (Shallice, 1982), or an action after some temporal delay (Amalberti & Deblon, 1992). In consequence, some clarification of the scope of the term, as used here, is required. Planning refers to the formation of plans for actions/interventions with aircraft at some point in the future. An important distinction here, therefore, is between pre-planned interventions and unplanned interventions (referred to in this paper as management by ‘instant execution’). Instant execution occurs where an intervention is specified, and then immediately executed by the operator. As such, the term refers to the formation of real-time control decisions, with no forward-looking time delay between the specification of an intervention and its execution. In contrast, planning is always for some future intervention (ahead), that is, not the next action undertaken by the operator, after specifying the intervention. The planning ahead of interventions is commonly: (i) viewed as more efficient (i.e., involving fewer interventions) than instant execution, when implementation costs are high (O’Hara & Payne, 1999); (ii) associated with expertise and strategic thinking (de Groot, 1978); and therefore (iii) considered a form of best practice. For example, de Groot showed that grandmasters plan ahead to a depth of six or seven chess interventions, in contrast to novices, who may only consider one or two interventions ahead. The benefits of such ability are clearly demonstrated by game outcome; the grandmaster is a ‘master’ for good reason.

 

The development of technology, for the management of dynamic processes, reflects the general desirability of planning interventions further into the future. The ATM-like microworld, used in this paper, serves only to illustrate the method. However, within operational Air Traffic Management in recent years, the concept of ‘gate-to-gate’ aircraft route planning has emerged. This concept requires that the planning of aircraft interventions no longer be devolved to the level of sector management, but rather be considered for all sectors traversed by aircraft during flight, by a separate team of specialist ‘multi-sector planners’. Again, this strategy suggests that planning further ahead is considered generally to be a desirable feature of dynamic process management, a feature for which technological support is sought (David, 1997; Miaillier, 1998).

 

II.2            PLANNING THEORIES

Models of human and machine planning have a long history (Hoc, 1988; Miller, Galanter, & Pribram, 1960; Sacerdoti, 1977), such models drawing to a greater or lesser extent, on planning theory  (Suchman, 1993, 1987; Vera & Simon, 1993). For the purpose of expressing planning effectiveness, at Stage Three of the method, a model is constructed that represents an operator’s plans for future interventions with aircraft. Model construction is informed by a Theory of the Operator Planning Horizon. The theory is largely a synthesis of parts of other planning theories, with some additions to make it suitable for expressing planning effectiveness. Such theory-driven modelling is possible because the simulation presents the operator with a well-defined problem space, and the task is completed by operator-technology interaction. There are no human-to-human collaborative processes influencing the planning task (Hughes, Somerville, Bentley, & Randall, 1993). As such, the method addresses planning as work at the level of micro-level mental processes, rather than as a macro-level community activity (Engeström, 2000). There is only one agent of management, who plans, anticipates, intervenes, and so forth.

 

From planning theory, a number of stable phenomena have been documented, of both planning behaviour, and of plans themselves, that may be assumed when modelling operator planning for dynamic process management.  For example, planning is most frequently characterised as being a form of goal-oriented behaviour (Newell & Simon, 1972). Having expressed the goals of management within the microworld as the maintenance of aircraft safety and expedition, it is anticipated that individual plans for interventions (to speed and altitude of individual aircraft) can be associated with these management goals.

 

During the planning process, a number of mental representations (knowledge sources) are known to be utilised, mental representations of (1) the domain state (Moray, 1992; Rasmussen, 1986), and of (2) how to interact with technologies/devices (Young, 1983) to bring about transformation of that domain (Payne, Squibb, & Howes, 1990). A mental representation of the state of the domain corresponds to what the operator knows, from moment to moment, about the state of the managed traffic – the work being undertaken (Dowell & Long, 1998). This representation is frequently referred to as the operator’s ‘picture’ (Cox, 1992; Whitfield & Jackson, 1982), and it is used to predict the future states of aircraft on the sector, and thereby plan.

 

Different types of plan are known to exist (Hoc, 1993), two possible types being plans for: (1) high-level process management; and (2) particular interventions to particular process objects. Here, the plans of concern are scoped to the latter class, plans for interventions with simulated aircraft (worksystem actions that transform the domain). In addition to mental representations of the domain (used in planning), the operator requires representations concerning how to interact with worksystem devices (Young, 1983), so that the detailed specification of actions, which bring about interventions, can be constructed.

 

Hayes-Roth and Hayes-Roth (1979) have shown that human planning is opportunistic, in that individuals plan when opportunity and necessity demand. Operators may not maintain a complete, coherent and well integrated set of plans for all aircraft managed within the sector. Rather, plans may exist for some aircraft, known to be particularly problematic (top-down planning), but as domain events arise (or are predicted to arise) that affect management goals, plans may be constructed in a reactive manner (bottom-up). It is therefore not the case that what is planned (at ‘planning time’) is always executed (at specified ‘execution time’). Rather a range of outcomes may be observed as plans may be discarded or repaired (Hoc 1988; Woods, 1988), due to failures in information acquisition and subsequent anticipation, or decay (i.e., be forgotten (Timmer & Long, 2000)). Having considered some of the relevant planning literature, for the management of dynamic processes, the Theory of the Operator Planning Horizon can now be presented.

 

II.3            THEORY OF THE OPERATOR PLANNING HORIZON

The first stage of the method, for expressing the effectiveness of a planning horizon, involves conceptualising the planning phenomena of interest. For this microworld, conceptualisation involves scoping operator planning behaviour, observed in the simulation, with respect to the planning literature, and synthesising existing theory with concepts necessary to express the effectiveness of a planning horizon. The outcome here is the Theory of the Operator Planning Horizon (TOPH), which makes explicit the planning phenomena of concern to the research. The TOPH may be stated, in domain-independent terms, as follows:

• An interactive (planning) worksystem formulates plans for interventions to a domain that are intended to attain management goals.

• Plans are formed by planning behaviour that requires mental representations of: i) the domain; and ii) the devices (and how to interact with those devices, to bring about interventions).

• Plans specify: i) interventions to domain object attribute values; and ii) a ‘triggering condition’ for plan execution.

• One or more plans, that refer to the same domain object, constitute the planning horizon for that object.

• A planning horizon may be described in terms of its ‘extension’.

• A planning horizon’s extension is expressed in terms of the future state of the domain object in question, if all planned interventions are executed.

• The extension of a planning horizon may be described as being adequate or inadequate for ensuring that the goals of management are met. The adequacy of a planning horizon’s extension is determined by the individual plans it contains, that is to say, whether or not when implemented, those plans ensure management goals are attained. If an operator’s planning horizon extension is adequate, and assuming the planned interventions are executed, the horizon will support effective management.

The theory places emphasis upon the planning worksystem’s goals, mental representations, and plan details. The concept of an horizon’s ‘extension’ arises from the need to express how far into the future plans (within an horizon) account for the state of a managed object. For example, two planned interventions to an aircraft (in the microworld) may be sufficient to ensure that the aircraft leaves the sector in its planned ‘exit’ state. The horizon (comprising those two planned interventions) may then be said to ‘extend to’ that aircraft’s exit state (object goal state, at the time of planning). Alternatively, horizons for aircraft may extend to states, associated with particular beacons, or parts of airways. Expressing the planning horizon’s extension in terms of the state of an object may be contrasted with other theories, such as Boudes and Cellier’s (1997) Theory of Anticipation Range. In their theory, the extension of a set of plans is expressed in terms of time, for example, plans that extend over the next 5 minutes for a given aircraft (see also Anderson & Settle, 1996). Finally, given the present work’s focus upon expressing problems with plans, the notion of planning extension adequacy is critical and novel. The adequacy of an extension is a concept that is used to relate plans for particular interventions to management goals, and thereby enable the expression of whether or not the specified plans for an aircraft ensure safety and expedition for that aircraft, over the extension of the plans. In conclusion, while the theory embodies some familiar concepts from planning theories (e.g., use of mental representations in planning, existence of triggering conditions, and so forth), it also possesses some novel concepts that are important for the expression of planning effectiveness.

 

Use of the term ‘horizon’, to refer to the limits of a mental behaviour, is not new. Hutchins (1990) describes an ‘horizon of observation’ in ship navigation, and more recently Wong, Sallis and O’Hare (1997) have discussed the planning horizon with respect to ambulance dispatch, yet in a non-technical manner, without defining what is meant by the term. Hence the need to conceptualise the term, and thereby generate a theory. Perhaps the most complete and explicit exposition of an alternative theory to TOPH, that examines the horizon of some form of mental behaviour, is Boudes and Cellier’s (1997, 1998) Theory of Anticipation Range, which possesses two components: (1) a mechanism for anticipating future domain object states; and (2) a temporal horizon. In contrast, the TOPH possesses three components: (1) planning behaviour (including plans); (2) horizon extension (in terms of future object states, rather than time); and (3) the adequacy of horizon extension (explicitly addressed).

 

II.4            REQUIREMENTS FOR A MODEL OF THE PLANNING HORIZON

The TOPH conceptualises the planning phenomena of interest, for the modelling of an operator’s horizon. From the theory, requirements may be generated, concerning behaviours that a model of a planning horizon should capture (at Stage 3), if the model is to represent accurately an horizon. These requirements are:

• Models need to reference interventions with objects (aircraft) that can be associated with management goals (of safety and expedition).

• Models need to reference the operator’s mental representation of the state of the domain at the time of plan formation.

• Models need to specify the details of planned interventions, and the conditions for triggering such interventions.

• Planning horizon models need to be specified for each managed object, and moment-to-moment changes of horizon extension need to be expressed in terms of a change in state of the object to which reference is made – domain objects here may be abstractions, i.e. groups of functionally related objects (aircraft).

• Models need to specify all the interventions that actually take place with an object (planned and unplanned), and their impact upon management effectiveness, such that the adequacy of an horizon extension may be expressed retrospectively.

 

With respect to the microworld, it can be seen that following the TOPH, a model needs to represent: (1) operator plans for a particular aircraft (one aircraft per horizon); (2) the operator’s mental representation of the state of the domain at the time of planning; and (3) the interventions actually carried out. Such a model addresses the planning concern of this research. To examine the effectiveness of the plans formed, a model of the planning horizon needs to be considered alongside a second model, a model that captures the quality of work carried out (i.e. the extent to which the aircraft objects were actually managed in accordance with management goals). This second model of work quality is termed a domain model, and the focus for Stage 2 of the method. In the next section, a microworld-generated domain model is considered.

 

III.         DOMAIN MODEL

In the microworld, a computer-based simulation generates a radar image of the managed sector, updated as aircraft traverse the sector, or change speed/altitude in response to operator interventions. In addition to generating this image, after each operator intervention, the simulation calculates new domain model values for a hierarchy of aircraft attributes that may have been altered as a consequence of the intervention. At the lowest level of the hierarchy are what Dowell (1998) calls PASHT attributes, standing for: Position; Altitude; Speed; Heading; and Time.  From these low level attributes, intermediate attributes are calculated for aircraft: Progress (flight duration); Fuel Use; Separation and Number of Manœuvres. Finally, at the apex of the attribute hierarchy, values for aircraft safety and expedition are calculated from intermediate attribute values. Therefore, an intervention to change an aircraft altitude will be recorded in the domain model at the PASHT level, and consequences of that aircraft climbing/descending to the new altitude will be calculated in terms of progress and fuel use, and ultimately any consequences for aircraft safety and expedition at the new altitude (and during the ascent or descent) will be calculated. Following the last intervention with each aircraft, the set of attribute values calculated represent the final ‘actual’ values for: Progress, Fuel Use, and so forth; for that aircraft’s passage across the sector. In addition to calculating these values after each intervention, for each attribute the model possesses a goal value. These goal values are calculated, by the simulation software, based upon an optimal (goal) path across the sector. Therefore, given ‘actual’ attribute values (from management scenarios) and ‘goal’ values (from an optimal scenario), comparisons may be made, from intervention to intervention, between the actual and goal states of managed aircraft. Large discrepancies between actual and goal values will constitute the starting point for diagnosing worksystem design problems.

 

 

 

 

 

 

Figure 2. Part of a domain model for a single intervention with aircraft BAN

Figure 2. Partie du modèle du domaine pour une intervention unique sur l’avion BAN

 

 

Figure 2 shows part of a domain model for aircraft BAN, following an operator intervention to slow down BAN from 900Kmph to 720Kmph (the change to BAN’s PASHT attribute – Speed – is shown at A). At B and C, only Intermediate attribute values are shown, from which values for safety and expedition can be calculated. Bold values in brackets, alongside each attribute name, represent goal values for that attribute. Unbracketted values show calculated predictions for each attribute, given the aircraft’s state, following the most recent intervention. A set of such values, for all aircraft after each intervention, constitutes the domain model of interest. Therefore, as a consequence of an intervention to reduce BAN’s speed, its predicted Progress across the sector is slowed, from 1170secs to 2220secs; and Fuel Use is greatly reduced, as 720Kmph is the cruising speed for all aircraft (the speed at which fuel use is optimised). Within the model the Separation attribute shows the aircraft is safe, no separation conflicts with other aircraft have been predicted, given the new speed value. As a consequence of the intervention, the Number of Manœuvres attribute is increased by one.

 

When such data concerning the actual state of an aircraft are compared with goal values for that aircraft, an analysis of the quality of aircraft management is possible, both from moment to moment, and at the end of the management session. The data for BAN here show that prior to the speed intervention, BAN’s time to progress across the sector was 1170sec. When compared to BAN’s goal value for progress, this value indicates BAN to have been progressing too quickly across the sector, and due to exit the sector 1260secs earlier than planned (the difference between actual and goal values). The cost associated with such fast passage is clear from the fuel consumption figures, which show the aircraft, prior to intervention, consuming 328 more units of fuel than the goal value. Both before and after the intervention, BAN’s separation value is safe, and so BAN’s superordinate safety attribute can likewise be calculated as being safe – BAN is in a safe state, both before and after the intervention. Following the intervention, BAN’s progress and fuel consumption are more closely aligned with goal figures.

To establish instances of ineffective management, some criteria need to be applied to the size of any discrepancy between goal and actual attribute values. Here, it is assumed, for the purpose of expressing effectiveness, that if values for an aircraft’s progress or fuel consumption exceed 10% (either side) of the goal value, ineffective management has occurred. Likewise, if aircraft separation is violated, ineffectiveness has occurred. Finally, the Number of Manœuvres value should not exceed 3 (i.e. 50% in excess of the goal value shown). Consideration will now be given to the impact these criteria have (for intermediate attribute values), on high level attributes of safety and expedition. If separation is violated, the attribute value for safety likewise reflects this state (Safety = Unsafe). If any one of the values for: Progress; Fuel Use; or Number of Manœuvres exceed specified criteria, the aircraft may likewise be considered ‘unexpeditious’, i.e. the management goal of expedition is not being attained.

 

In conclusion, data from the microworld-generated domain model concern the quality of aircraft management, and instances of attribute values violating criteria may be associated with instances of ineffective management – unsafe aircraft, aircraft progressing too fast or slow, consuming too much or too little fuel, or undergoing intervention too often. Expressing the effectiveness of an operator’s planning horizon, therefore, involves associating domain model data with particular interventions, and establishing whether or not those interventions were planned.  In the next section, Stage 3 of the method for expressing the effectiveness of planning horizons is presented. A second model, a model of an operator’s planning horizon, is discussed. Stage 3 is followed in Stage 4, by an analysis of the effectiveness of that horizon, when the planning horizon model is considered in the light of data from the domain model (Stage 2), which contains data similar to that discussed above.

 

IV. Model of a Planning Horizon

IV.1 Data Requirements for Model Construction

From the Theory of the Operator Horizon, a number of requirements were identified for a model of the horizon. Each requirement is now considered in turn, and the means for obtaining data to address the requirement discussed.

 

Requirements

• Models need to reference interventions with objects (aircraft) that can be associated with management goals (of safety and expedition).

In the microworld, all interventions are made with the radar device, and are therefore observable. Continuous observation of operator interaction with worksystem technologies yields such data.

 

• Models  need to reference the operator’s mental representation of the state of the domain at the time of plan formation.

Operators establish the state of the domain by referencing worksystem devices. By observing head movements and pointing actions to fields on a Flight Progress Strip, it is possible to establish what data concerning the domain are being acquired by the operator. Traditionally, concurrent verbal protocols provide a rich source of data concerning what an operator knows about a problem, and how that knowledge is used in problem solving. Verbal protocol data, in addition to observation of operator head and hand movement, yields data that assist in inferring the operator’s changing mental representation of the domain. In addition, the mental representation can be inferred from interventions. For example, if an operator intervenes with an aircraft to give it a cruising speed, it is possible to infer that the operator knew that aircraft’s speed was not the desired cruising speed, prior to intervention.

 

• Models need  to specify the details of planned interventions, and the conditions for triggering such interventions.

In the absence of planning tools that support the explicit documentation of a set of plans, operator plans for interventions remain in the head of the operator. In consequence, only verbal protocol data can reveal the details of such plans, and associated triggering conditions.

 

• Planning horizon models need to be specified for each managed object, and moment-to-moment changes of horizon extension need to be expressed in terms of a change in state of the object to which reference is made.

Each model should reference an individual aircraft. As the horizon changes, the state of the aircraft in question can be established with reference to the computer-generated domain model for that scenario (see Section III).

 

• Models need to specify all interventions that actually take place with an object (planned and unplanned), and their impact upon management effectiveness, such that the adequacy of an horizon extension may be expressed retrospectively.

To satisfy the first requirement, all interventions are documented. The impact of each intervention, upon management effectiveness (how well the actual state of the aircraft matches its goal state), can be established with reference to the domain model generated for the management scenario.

 

Therefore, with regard to acquiring data for the purpose of modelling operator planning horizons, observational data, of operator hand and head movements are required (for information search and interventions), plus concurrent verbal protocol data (for plans and evidence of the content of the associated mental representation). With such data, a model of the operator’s planning horizon can be constructed, as in Figure 3, which separates each of these classes of data.

 

IV.2            MODEL OF AN OPERATOR PLANNING HORIZON

Figure 3 presents a model of an operator’s planning horizon for an aircraft LOG. The ‘Plan/Execution’ column of the model records all human-technology worksystem goals that concern interventions. Data in this column distinguish goals for immediate execution (in bold), from plans for future interventions (in italics). The ‘Intervention’ column documents those goals for interventions that were actually implemented.

 

 

 

 

Figure 3. Model of an operator’s planning horizon for aircraft LOG

Figure 3. Modèle pour un opérateur d’un horizon de planification pour l’avion LOG

 

 

 

 

Goals that concern interventions are formed, given a particular mental representation of the state of the aircraft in question, at the time of planning or instant execution. The model records such mental representations in the ‘category’ column. The set of possible categories reflect the range of possible states of the objects in the domain (Timmer & Long, 1997). Aircraft may therefore be ‘Active’, when they arrive on the sector, or ‘Incoming’, prior to arrival. Once active, they may be ‘Safe’ or ‘Unsafe’, ‘Expeditious’ or ‘Unexpeditious’ with respect to their speed (‘Unexpeditious (Speed)’) or altitude (‘Unexpeditious (Altitude)’). Once an aircraft is at its exit altitude and cruising speed, no further interventions are required as it is in its exit/goal state (‘Active Aircraft (Exit)’). The final ‘Encode’ column records device information fields that were searched by the operator as a means to forming a mental representation (category) of the state of an aircraft, for the purpose of goal/plan specification or monitoring.

 

The model for aircraft LOG is considered in detail. The model commences (Line 1) by showing that at the beginning of the management session, having encoded LOG’s Flight Progress Strip (FPS), the operator knows that LOG is an incoming aircraft. No plans are formed at this stage. LOG then enters the sector (Line 2), and by encoding LOG’s radar trace, the operator’s mental representation of the state of LOG is updated, from ‘Incoming Aircraft’ to ‘Active Aircraft’. Referencing again LOG’s FPS (Line 3), the operator establishes LOG as travelling at 900Kmph, and at an altitude of 13,000ft. Given LOG’s excessive speed, the aircraft is mentally represented as being unexpeditious with respect to its speed, and an intervention is immediately formed (Line 3) and executed (Line 4) to slow the aircraft down, and transform its state to ‘expeditious’ with respect to its speed. This intervention is an example of instant execution, and represented within the model as such. One consequence of the intervention is that the operator’s mental representation of LOG is now updated, to reflect its new expeditious state. LOG is safe and expeditious, resulting in the operator forming a default plan, to leave LOG in its current state for the foreseeable future. The model then shows that LOG is left to progress from its entry beacon to the intermediate beacon ‘Delta’, and then on to its exit beacon ‘Epsilon’, at altitude 13,000ft and speed 720Kmph. Once at its exit beacon (Line 5), the operator encodes FPSs for both LOG, and a second aircraft SAM, and realises that both aircraft occupy the same altitude, and safety conflict is possible. LOG is therefore mentally re-categorised as ‘Unsafe’, and a plan is formed to give LOG its exit altitude of 4,000ft later (in the near future). The model then shows (Line 6) that the operator’s mental representation of: (1) the state of LOG; and (2) the plan for future intervention (formed at Line 5), are both forgotten (decay) and the plan to ‘Give LOG (its exit) altitude later’ is re-formed (at Line 7). On the occasion of this re-planning, the model shows that no safety conflict with SAM was identified, and LOG was represented once more as a ‘Safe Expeditious Aircraft’. The next line of the model (Line 8) shows that the operator appears to have forgotten (again) the state of LOG, and by referencing data from FPSs (Line 9), reforms the mental representation that LOG is a safe expeditious aircraft. The plan formed at Line 7 is not assumed to have decayed on this occasion, as the model shows no further re-planning of the intervention, until it is executed at Line 10. The final line of the model shows that LOG is given its exit altitude as planned, and the default plan to leave LOG until it exits the sector is formed, as the aircraft is in its exit state, and no further interventions are necessary.

 

If the model is considered with respect to how it documents the extension of the operator’s planning horizon for LOG, the following observations can be made. At Line 4, a plan is formed to ‘Leave LOG’. Given that LOG is flying at a high altitude, and at cruising speed, and is safe (i.e. is an ‘Active Safe Expeditious Aircraft’), LOG may be left in such a state, until near its exit beacon, and then it should be given its exit altitude. The plan to ‘leave’ LOG may therefore be said to extend to ‘near LOG’s exit beacon’. At Line 5, when LOG is near its exit beacon, a further plan is formed to give LOG its exit altitude. At Line 5, with such a plan, the operator’s planning horizon may be described as extending to ‘LOG’s exit state’ (i.e. to the end of its management). At Line 7, the new duplicate plan (of the plan formed at Line 5) has an identical extension to that of the plan at Line 5. Therefore, with a model of an operator’s planning horizon, from moment to moment, it is possible to express the ‘extension’ of the plans formed, either in terms of: a) the state of the aircraft in question, for example, the plans extend to LOG’s exit state; or b) some position on the sector at which the aircraft’s state must change, for example, plans for aircraft LOG extend to near LOG’s exit beacon’. A description of the extension of a planning horizon, therefore, arises from the details of the operator’s plans for a given domain object (aircraft) at a given moment. As plan details change, so too does the horizon’s extension.

 

V.         EXPRESSING HORIZON EFFECTIVENESS

Having conceptualised the planning horizon (Stage 1), described how a domain model captures work quality (Stage 2), and illustrated how a planning horizon is modelled (Stage 3), in Stage 4 of the method, expression of the effectiveness of that horizon is undertaken. In the first instance, the model of the planning horizon is considered, line-by-line, alongside objective data from the domain model, concerning aircraft states after each intervention. The actual state of aircraft (from the domain model) can then be compared with the operator’s inferred mental representation of the state of that aircraft (‘Category’ column of a planning horizon model), and plans and interventions considered, to establish whether or not the plans/interventions were effective. Once completed, and in the event of an instance of ineffectiveness occurring, some expression of its cause can be undertaken.

 

The Theory of the Operator Planning Horizon expresses the concept of effectiveness (attaining management goals), in terms of the adequacy of an horizon’s extension. Adequacy of an horizon’s extension relates to the quality of work that will be brought about by the worksystem, if the plans that make-up the planning horizon are executed. An horizon’s extension may be considered adequate, if the plans (that make-up the horizon), when executed, lead to the attainment of worksystem goals of safety and expedition. Adequacy is, therefore, a difficult attribute of an horizon to assess. At one moment the horizon may extend to an Aircraft’s (Aircraft X) exit state, and be adequate to ensure goals are met. At another moment, a second Aircraft Y may be given the same altitude as Aircraft X, thereby rendering the plans, that make-up Aircraft X’s planning horizon extension, inadequate for ensuring the maintenance of safety. Re-planning is then necessary. Existing plans may need to be discarded, given the new unsafe state of the aircraft. For each line of the model in Figure 3 (starting at Line 4), an assessment of adequacy is made, following a short description of the plan being assessed.

 

• Line 4

Encode       Intervention          Category                                           Plan/Execution

LOG to 720

Active Safe Expeditious Aircraft

 

description

LOG is given speed 720Kmph (instant execution), and a plan is formed to ‘Leave LOG’ at speed 720Kmph, and altitude 13,000ft (i.e. as an Active Safe Expeditious Aircraft). A single plan is formed that extends to near LOG’s exit beacon.

 

adequacy

Following the intervention to LOG’s speed, at the time of plan formation, the domain model’s prediction of LOG’s state is presented in Figure 4.

 

 

 

 

 

 

 

Figure 4. Domain model performance data for LOG

Figure 4. Données du modèle de performance du domaine pour LOG

 

 

The domain model shows that LOG is:

• safe in its new state

The planning horizon at Line 4 is made-up of a single plan, to leave LOG, and given that LOG is safe in its current state, it may be said that the horizon extension, at this point, is adequate for ensuring that aircraft safety is maintained.

 

• expeditious in its new state

Projected progress is within 10% of criterion. Projected fuel use is 18.8% in excess of criterion. Given LOG’s speed and high altitude, it would seem the projection of excessive fuel use refers to fuel already consumed, before the intervention, i.e. when travelling at 900Kmph earlier in the management scenario (see Line 3, Figure 3). The plan to leave LOG near its exit beacon is therefore adequate for ensuring aircraft expedition.

 

• Line 5

Encode                  Intervention    Category                    Plan/Execution

Position = ti1Altitude = 130Exit-at = EpsilonExit Altitude = 40 SAM, Altitude = 130

Active UnsafeExpeditious Aircraft

Give Altitude 40, Later

 

 

description

LOG progresses across the sector to its exit beacon. A plan is formed to give LOG its exit altitude of 4,000ft, ‘later’. This single plan extends to LOG’s exit state. While ‘later’ is a triggering condition of minimal specification, the assumption that the horizon extends to LOG’s exit state is considered justified, because all necessary interventions have been specified to ensure LOG leaves the sector in its exit state.

 

adequacy

At planning time, and as a consequence of the adequacy of the planning horizon’s extension at Line 4, LOG is safe and expeditious. This plan is actually executed at Line 10. It is therefore possible to consult the domain model’s assessment of LOG’s safety and expeditiousness after this intervention (Figure 5), and thereby assess the adequacy of the plan extension.

 

 

 

 

 

 

Figure 5. Performance data for LOG after the last intervention

Figure 5. Données de performance de LOG après la dernière intervention

 

 

From Figure 5, it is therefore possible to say retrospectively:

• the planning horizon extension (to LOG’s exit state) will ensure its safety

• while progress and exit altitude are as planned, LOG’s fuel

consumption increases with this intervention (40% in excess of the

criterion value). It would therefore appear that this horizon extension is not adequate for ensuring that aircraft expeditiousness is maintained.

 

• Line 6

Encode           Intervention               Category                              Plan/Execution

LAPSE

LAPSE

 

 

The details of the plan to ‘Give LOG altitude 40, later’ appear to have decayed, and so the model shows the operator has no plans at this time.

 

• Line 7:

Encode                    Intervention     Category                  Plan/Execution

Position = ti1Altitude = 130Exit Altitude = 40Speed = 720 TAW, Position = ti2 TAW, Altitude = 130 TAW, Exit Altitude = 130 TAW, Speed = 720

Active Unsafe Expeditious Aircraft

Give Altitude 40, Later

 

 

description

The intervention is re-planned, as at Line 5.

 

adequacy

As at Line 5.

 

No further plans are formed. At Line 10, the plan formed at Line 7 is executed. The domain model, as discussed at Line 5, shows LOG is unexpeditious with respect to its fuel use (40% excess).

 

From this line-by-line analysis, it is clear that the effective management of aircraft LOG’s fuel consumption did not take place as desired. In consequence, the domain model shows LOG to have been unexpeditious as it exited the sector, having consumed 40% more fuel than desired (the bracketed goal value for fuel use in the domain model). Given this ineffectiveness, the following expression of the problem, with this ATM planning task, is possible:

The problem of managing aircraft expedition with respect to fuel use may originate, as in the case of LOG, with aircraft entering sectors at high speed (greater than cruising speed), thus already rapidly consuming large quantities of fuel. The timely reduction of aircraft speed can alleviate this problem. However it would seem a greater part of the problem arises from judging the appropriate moment to intervene with aircraft, to allocate low exit altitudes. Managing aircraft at cruising speed, as high as possible for as long as possible, is the best strategy for minimising fuel consumption. With a reduction of altitude comes a commensurate increase in fuel consumption. If an aircraft is to leave a sector at a low altitude (e.g., for airport approach), careful judgement is required as to when to execute such an intervention, so that the aircraft exits the sector at the exit altitude. If such an intervention is executed too early, the aircraft will fly for an extended duration at a low altitude, consuming higher quantities of fuel. In the case of LOG, this problem would appear to have been the most important. The operator’s plan to ‘Give LOG Altitude 4,000ft, later’ lacked accurate reference to a position within LOG’s final airway, when the intervention would be made, ‘later’ merely meaning ‘near LOG’s exit beacon’. It is assumed that in consequence, LOG was given its exit altitude too early, and flew too low for too long, to maintain an expeditious level of fuel consumption. While worksystem technologies supported the operator in forming a plan for the correct intervention, they offered no support for judging the most appropriate moment for plan execution.

 

This expression of the causes of ineffectiveness accounts for ineffectiveness in the management of a single aircraft. It is proposed that for each additional instance of ineffectiveness, attributed to fuel use, a similar analysis be undertaken. When considering the data over a number of instances of the same problem, a general expression of the problem is possible (Timmer, 1999). Nevertheless, from consideration of the expression above, it is proposed that problem-driven technological evolution can benefit from problem expressions, similar to that illustrated. In the case of the evolution of technologies to support better fuel use management, redesign may commence with consideration of how to support operator judgement in timing the moment of low-level aircraft descent, prior to exit.

 

 

V.         DISCUSSION

In this paper, a method has been proposed and illustrated for expressing human-technology worksystem effectiveness during a planning task. The method involves stages of: conceptualising behaviour of interest; modelling the domain to measure work quality; modelling planning behaviour; and considering work quality alongside worksystem behaviour, to enable an expression of effectiveness. Using an illustration of the poor quality management of expedition, ineffectiveness was identified, and expressed in terms of worksystem behaviour, and in a manner proposed to support problem-driven technological evolution.

 

Of the method, a number of observations may be made. The method is comprised of a set of general stages, rather than a set of detailed procedures. For example, following conceptualisation, a domain model needs to be constructed. The illustration discusses one such model (not how it was constructed), and it is accepted that there are many alternatives to the one discussed. For the method to express effectiveness successfully, it is merely a requirement that a domain model measure work quality in some way, using some performance criteria for judgement of ineffectiveness (design problems). Without such measurement, comparing worksystem behaviour with work quality during Stage 4 is not possible. One advantage of expressing the method in this manner is that it extends the method’s scope of application. An ATM-like microworld was the focus of the illustration, and planning ahead conceptualised with respect to the planning behaviour likely to be observed during management of that microworld. Using another domain (the domain of Railway Signal Management (RSM) has also been analysed), it is possible to envisage other behaviours being conceptualised, for example, the notion of planning extended to include strategic plans, (for example to maximise train throughput or even out train flow), as well as plans for discrete interventions (tactical) (for example, to particular signals). Provided such planning is modelled accurately at Stage 3, the logic of the method, and successful expression of effectiveness, should be maintained.

 

The success with which the method can be migrated from a microworld to an operational environment, is largely a function of the extent to which a) the planning horizon modelling requirements can met, and b) the target domain modelled (and performance measured). In current operational ATM, for example, the method, as it stands, will not scale-up, as the verbal protocol data (revealing planning behaviour) can not be obtained without task interference. In the future, with Datalink technology, method application to operational ATM may be more feasible. In RSM, the voice channel is largely free, except for control-to-train communication, and control room communication – here also, management is largely undertaken by a single signalman (with some communication with other managed line sections and a supervisor).

 

When this work is considered alongside the work of Boudes and Cellier (1997, 1998), concerning controller ‘anticipation range’ in ATM, some similarities and differences can be identified. Theoretically, Boudes and Cellier’s work has strong similarities, in that they too seek to establish relationships between operator mental representations of the domain and devices, and plans for interventions and flight strip management. However, the method presented here, for modelling the effectiveness of planning horizons, possesses a number of novel components. Firstly, it attempts to characterise the extension of a planning horizon in non-temporal terms, but rather in terms of the future states of intervened aircraft (whether or not aircraft will be safe or unsafe, and to what future point (on the sector) do such plans extend). Secondly, the method possesses a domain model, with which to assess the adequacy of plans for future interventions. The domain model is crucial to the successful execution of the method, as it enables an assessment to be made of how well the operator is planning, and how adequate the specified plans are for ensuring that management goals are achieved. Without a domain model, a method can only determine that an operator is planning. No assessment of the quality of those plans can be made.  As the focus of the method is to support the development of technologies that will improve operator planning abilities, the domain model is crucial for determining existing planning ineffectiveness and how such ineffectiveness can be overcome through re-design.

 

To conclude, this paper tries to make explicit a number of important relationships that need to be established before the expression of effectiveness is possible. The Theory of the Operator Planning Horizon tries to make clear the behavioural phenomena of concern, and enables the clear derivation of requirements for data that constitute a representation of such an horizon. Likewise, the domain model enables the identification of problems of worksystem performance, and subsequent construction of a worksystem model to establish the behaviour that brought about less than the desired quality of work. Establishing relationships from the data serves to augment the subjective (anecdotal) recall of operator ‘problems’ with technologies, and offers some basis for quantifying problems, and the subsequent generation of priorities for re-design. Such a method is therefore considered a useful tool, to compliment existing design practices, to support the explicit expression of the magnitude of a design problem. As such, it is considered to have advanced the ‘design for effectiveness’ approach, since without a well-specified expression of the design problem, there can be no (known) design solution, nor acquisition and validation of design knowledge supporting the transition from one to the other (Long & Timmer, 2001). Phenomena-driven and human performance-driven approaches are unable to support such a design origin and transition.

 

 

 

 

VI.         REFERENCES

Amalberti, R., & Deblon, F. (1992). Cognitive modelling of fighter aircraft process control: a step towards an intelligent on-board assistance system. International Journal of Man-Machine Studies, 36, 639-671.

Anderson, B. F., & Settle, J. W. (1996). The influence of portfolio characteristics and investment period on investment choice. Journal of Economic Psychology, 17, 343-358.

Barnard, P. (1991). Bridging between basic theories and the artifacts of human-computer interaction. In J. M. Carroll (Ed.), Designing interaction: Psychology at the Human-Computer Interface. Cambridge, UK: Cambridge University Press.

Blandford, A., & Young, R. M. (1993). Developing runnable user models: separating the problem solving techniques from the domain knowledge. In J. L. Alty, D. Diaper, & S. Guest (Eds.) People and Computers VIII, Proceedings of HCI’93. Cambridge, UK: Cambridge University Press.

Boudes, N., & Cellier, J-M. (1997). Evaluating the concept of anticipation range in air traffic control. Paper presented at the Ninth international Symposium of Aviation Psychology, Columbus, Ohio, May.

Boudes, N., & Cellier, J.-M. (1998). Étude du champ d’anticipation dans le contrôle du trafic aérien. Le Travail Humain, 6, 29-50.

Card, S. K., Moran, T. P, & Newell, A. (1983). The Psychology of Human- Computer Interaction. Hillsdale, New Jersey: Lawrence Erlbaum Associates.

Cox, M. (1992). The cognitive aspects of the air traffic control task: a literature review. IAM Report No. 718. Psychology Divison, RAF Institute of Aviation Medicine.

David, H. (1997). Radical revision of en-route air traffic control. EUROCONTROL Experimental Centre Report No. 307. http:/www.eurocontrol.fr/public/reports/eecreports/1997/rep307

de Groot, A. D. (1978). Thought and choice in chess. The Hague: Mouton.

Dowell, J., & Long, J. (1998). Conception of the cognitive engineering design problem. Ergonomics, 41, 126-139.

Dowell, J. (1998). Formulating the cognitive design problem of air traffic management. International Journal of Human-Computer Studies, 49, 743-766.

Engeström, Y. (2000). Activity theory as a framework for analyzing and redesigning work. Ergonomics, 43, 960-974.

Flach, J. M. (1998). Commentary. Cognitive Systems Engineering: putting things in context. Ergonomics, 41, 163-167.

Hayes-Roth, B., & Hayes-Roth, F. (1979). A cognitive model of planning. Cognitive Science, 3, 275-310.

Hoc, J-M. (1988). Cognitive Psychology of Planning. London: Academic Press.

Hoc, J-M. (1993). Main features of the human supervision of a long response latency process. In Proceedings of the IEEE/SMC’93 Conference on Systems, Man and Cybernetics, pp. 114-119. Piscataway, NJ: IEEE.

Hollnagel, E. (1998). Commentary. Comments on ‘Conception of the cognitive engineering design problem’ by John Dowell and John Long. Ergonomics, 41, 160-162.

Hughes, J. A., Somerville, I., Bentley, R., & Randall, D. (1993). Designing with ethnography. Making work visible. Interacting with Computers, 5, 239-253.

 

Hutchins, E. (1990). The technology of team navigation. In J. Galegher, R. E. Kraut, & C. Egido (Eds) Intelligent Teamwork: Social and Technological Foundations of Cooperative Work, pp.191-220. Hillsdale, New Jersey: Lawrence Erlbaum Associates.

Long, J., & Timmer, P. (2001). Design problems for research: what can we learn from ATM-like microworlds. Le Travail Humain, 64.

Meyer, D. E., & Kieras, D. E. (1999). A computational theory of executive cognitive processes and multiple-task performance: Part 1 Basic Mechanisms. Psychological Review, 104, 3-65.

Miaillier, B. (1998). ATM Strategy for 2000⁺: Volumes 1 and 2. EUROCONTROL Report FCO.ET1.ST07.DEL02. http://www.eurocontrol.be/ded/atmstrat

Miller, G. A., Galanter, E. & Pribram, K. H. (1960). Plans and the Structure of Behaviour. New York: Holt, Rinehart & Winston.

Moray, N. (1992). Mental models of complex dynamic systems. In P. Booth, & A. Sasse (Eds). Mental Models and Everyday Activities, Second Interdisciplinary Workshop on Mental Models, pp. 103-132. Cambridge, UK, March.

Newell, A., & Simon, H. A. (1972). Human Problem Solving.  Englewood Cliffs, NJ: Prentice-Hall.

O’Hara, K, P., & Payne, S. J. (1999). Planning and the user interface: the effects of lockout time and error recovery cost. International Journal of Human-Computer Studies, 50, 41-59.

Payne, S. J., Squibb, H. R., & Howes, A. (1990). The nature of device models: the yoked state space hypothesis and some experiments with text editors. Human-Computer Interaction, 5, 415-444.

Pietras, C. M., & Coury, B. G. (1994). The development of cognitive models of planning for use in the design of project management systems. International Journal of Human-Computer Studies, 40, 5-30.

Rasmussen, J. (1986). Information processing and human-machine interaction: An approach to cognitive engineering.  New York: North Holland.

Rasmussen, J. (1992). The ecology of work and interface design. In A. Monk, D. Diaper, & M. D. Harrison (Eds.), People and Computers VII. Proceedings of the HCI’92 Conference. Cambridge, UK: Cambridge University Press.

Reason, J. (1998). Commentary. Broadening the cognitive engineering horizons: more engineering, less cognition and no more philosophy of science, please. Ergonomics, 41, 150-152.

Sacerdoti, E. D. (1977). A Structure for Plans and Behaviour.  New York: Elsevier.

Shallice, T. (1982). Specific impairments of planning. Philisophical Transactions of the Royal Society of London, B 298, 199-209

Suchman, L. (1987). Plans and situated actions: the problem of human- machine communication. Camridge, UK: Cambridge University Press.

Suchman, L. (1993). Response to Vera and Simon’s situated action: a symbolic interpretation. Cognitive Science, 17, 71-75.

Timmer, P. (1999). Expression of operator planning horizons: a cognitive engineering approach. PhD Thesis, University of London.

Timmer, P., & Long, J. (1997). Separating user knowledge of domain and device: a framework. In H. Thimbleby, B. O’Conaill & P. Thomas (Eds.) People and Computers XII. Proceedings of HCI’97, pp379-395. London, UK: Springer.

Timmer, P., & Long, J. (2000). Plans versus outcomes: establishing the costs of planning. In E. Hollnagel, P. Wright, & S. Dekker, (Eds.) Proceedings  of the Tenth European Conference on Cognitive Ergonomics, pp158-165. Linkoping, Sweden: European Association of Cognitive Ergonomics.

Vera, A. H., & Simon, H. A. (1993). Situated Action: a symbolic interpretation. Cognitive Science, 17, 7-48.

Vincente, K. J. (1998). Commentary. An evolutionary perspective on the growth of cognitive engineering: The Risø genotype. Ergonomics, 41, 156-159.

Whitfield, D., & Jackson, A. (1982). The Air Traffic Controller’s ‘Picture’ as an Example of a Mental Model. In G. Johannsen, & J. E. Rijnsdorp, (Eds.) Analysis, Design, and Evaluation of Man-Machine Systems, pp. 45-52. New York: Pergamon.

Wong, W., Sallis, P., & O’Hare, D. (1997). Eliciting portrayal requirements: experiences with the critical decision method. In H. Thimbleby, B. O’Conaill, & P. Thomas, (Eds.) People and Computers XII. Proceedings of HCI’97, pp. 397-415.  London, UK: Springer.

Woods, D. D., & Roth, E. M. (1988). Cognitive Systems Engineering. In M. Helander (Ed.), Handbook of human-Computer Interaction, pp. 3-43. Amsterdam: North Holland.

Woods, D. D. (1988). Commentary: Cognitive Engineering in complex dynamic worlds. In E. Hollnagel, G. Mancini, & D. D. Woods (Eds.) Cognitive Engineering in Complex Dynamic Worlds. London: Academic Press.

Woods, D. D. (1998). Commentary. Designs are hypotheses about how artifacts shape cognition and collaboration. Ergonomics, 41, 168-173.

Young, R. M. (1983). Surrogates and mappings: two kinds of conceptual models for interactive devices. In D. Gentner, & A. L. Stevens (Eds.), Mental Models, Ch. 3. Hillsdale, New Jersey: Lawrence Erlbaum Associates.