In this section, we first present a theoretical analysis of the role of dual-task problem solving and cognitive fit in ill-structured problem areas. We then apply the theory to a study of problem solving on the ill-structured problem of software maintenance (Shaft and Vessey, 2006).
When the IS problem is ill-structured, the problem statement does not contain all of the information required to solve it, and the tasks of developing mental representations of the application and IS domains are both essential to solving the problem. When the knowledge required to solve each of the tasks is consistent (that is, ‘fit’ exists), then solution performance will be facilitated. When that knowledge is not consistent (that is, ‘fit’ does not exist), however, there will be a mismatch between the two types of knowledge required for problem solution. In this case, the mental representation of one of the sub-tasks must be transformed to match the other in order to facilitate problem solving. Hence the task is rendered much more complex than when cognitive fit exists.
We again draw on the theory of cognitive fit (Vessey, 1991) to provide the theoretical basis for what happens when knowledge of two tasks is essential to problem solving.
When cognitive fit exists, the problem solver’s mental representation of the application domain and their mental representation of the IS task emphasise similar types of knowledge and therefore have similar processing requirements. Hence, no transformations are required to form the mental representation for task solution, effectively reducing the cognitive resources needed for problem solving (Vessey, 1991). The problem solver can therefore more readily allocate attention between tasks. A problem solver is able to shift attention relatively easily, therefore, between the tasks in the IS and application domains with efforts in the application domain being directed toward the information most relevant to solving the IS task. As a result, problem-solving performance is likely to be more accurate and quicker than would otherwise be the case.
We state the following proposition:
Proposition IS-1: In ill-structured IS problem areas, when the mental representation of the application domain is consistent with (i.e., matches) the mental representation of the IS task, increased knowledge in the application domain is associated with higher levels of performance on the IS task (problem-solving performance).
When cognitive fit does not exist (that is, when knowledge in each of the two task areas does not mutually support problem solving), the problem solver’s mental representation of the application domain and the mental representation of the IS task emphasise different types of knowledge. Without cognitive fit, there is nothing to guide the problem solver in working towards task solution (Vessey, 1991). As a result, activities in the application domain are likely to interfere with the problem solver’s ability to complete the IS task, the problem solver may experience difficulty in allocating attention effectively between tasks in the IS and application domains that are not mutually supportive, and performance suffers.
In these circumstances, some kind of transformation must be brought about for problem solving to proceed: either the mental representation of the application domain must be transformed to emphasise the same type of knowledge as the mental representation of the IS task, or vice versa. The problem solver confronted with these challenges may take one of two approaches to resolve the situation: 1) focus further on the task in the application domain; or 2) focus further on the task in the IS domain.
In the first case, because the mental representation of the application domain tends to drive the solution process (Burkhardt et al., 2002), there is a tendency for the problem solver to heed information in the application domain that is consistent with the knowledge already emphasised in the mental representation of the application domain, rather than the knowledge consistent with their mental representation of the IS task (cf., Broadbent, 1971). Changing the mental representation of the application domain to reflect an increased understanding of knowledge that is consistent with their mental representation of the IS task does not, however, enhance the problem solver’s ability to complete the IS task. Hence, both efficiency and effectiveness are likely to be affected and it is likely that improved knowledge of the application domain will be associated with lower levels of problem-solving performance.
In the second case, problem solvers who focus more on task solution perform more effectively than those who focus on the present state (Durso et al., 1998; Hogg et al., 1995; Vessey, 1991). Such an approach would require them to switch attention to their mental representation of the application domain only when necessary to resolve an issue directly related to solving the IS task. Focusing on the IS task, however, has its own difficulties. First, because a problem solver’s mental representation of the application domain tends to be quite stable over time (Corritore and Wiedenbeck, 1999), problem solvers find it difficult to shift to a different mental representation of the application domain after they have invoked an inappropriate one (Taylor et al., 1997). Second, when a problem solver attempts to acquire knowledge consistent with that required to conduct the IS task they find difficulty in mapping that knowledge into their mismatched mental representation of the application domain. The process of building up the knowledge essential to conducting the IS task (that is, developing the mental representation for task solution) is therefore quite challenging. Hence, again, both efficiency and effectiveness are likely to be affected and it is likely that improved problem-solving performance will be associated with lower gains in application domain knowledge.
Based on these arguments, therefore, when there is a mismatch between the problem solver’s mental representation of the application domain and their mental representation of the IS task, problem-solving performance will be inversely related to improvements in knowledge of the application domain that occur during problem solving. Note that the situation we are describing is that of dual-task interference. We state the following proposition.
Proposition IS-2: In ill-structured problem areas, when the mental representation of the application domain is inconsistent with (i.e., does not match) the mental representation of the IS task, greater increases in knowledge of the application domain are associated with lower levels of performance on the IS task, and higher levels of performance on the IS task are associated with lesser increases in knowledge of the application domain.
In summary, then, when the mental representation of the application domain does not support the mental representation of the IS task to be conducted:
paying increased attention to the application domain distracts the programmer from the primary IS task of completing the modification; and
focusing on the IS task interferes with the problem solver’s understanding of the application domain.
In both cases, the relationship between knowledge of the application domain and performance on the IS task is an inverse one and this is indicative of dual-task interference.
For our application of dual-task problem solving to ill-structured problems, we draw on Shaft and Vessey (2006). In order to evaluate problem-solving performance, the authors used two different types of software maintenance tasks in both familiar and unfamiliar application domains to conduct a study in which the IS task (making a modification to software) and the application domain (represented by the software) were both matched and mismatched.
We first address the ill-structured nature of software maintenance, followed by the nature of both mental representations of the software (mental representation of the application domain) and modification tasks (mental representation of the IS task). Finally, we present details of the experiment and its findings.
Software maintenance requires knowledge of both the application area that is being addressed and how to develop as well as maintain software, both of which represent application domain knowledge. Hence application domain knowledge is essential to the solution of ill-structured problems. Allocating attention between tasks in the IS and application domains is particularly relevant to software maintenance because maintainers must comprehend the existing software in order to be able to make relevant changes. Comprehending the software, alone, is a substantive task because maintainers must divide their attention among multiple elements within the software itself (e.g., different modules), as well as understand information other than that in the software (e.g., the external software documentation, etc). Modifying software is also a substantive task, as maintainers must both understand the modification to be conducted as well as make appropriate changes to the software.
Software maintenance is therefore a task for which the initial and goal states are vaguely defined and for which there are no well-defined solution paths. Further, because the problem statement does not contain all of the information needed for solution, it is not clear what actions are required to solve them. We therefore characterise software maintenance as an ill-structured problem area.
We first examine the intrinsic characteristics of software. We then apply that knowledge to the mental representations that software maintainers form when they examine software (mental representation of the application domain), as well as to the types of modification tasks that they may be requested to accomplish (mental representation of the IS task).
Part of the essential difficulty of building and maintaining software comes from the need to represent the numerous types of information that are embedded in a piece of software (see Brooks, 1987). We address the characteristics of software and software tasks using these types of information (see, for example, Brooks, 1987; Curtis et al., 1988; Pennington, 1987a, 1987b). The most important types of information embedded in software are function, data flow, control flow, and state information (see, for example, Pennington 1987a, 1987b). Function information reflects the main goals of the program and the hierarchy of sub-goals. Data flow information reflects the series of transformations that data objects undergo. State information relates to the condition-action information embedded in a program; that is, the program actions that result when a set of conditions is true. Control flow information reflects execution sequence; that is, the order in which actions occur.
The mental representations of the software (application domain) formed from these types of information are typically characterised as domain, program, and situation models (von Mayrhauser and Vans, 1995).
First, a software maintainer’s domain model is a high-level model that focuses on software functionality (Vans et al., 1999). It is more closely aligned to the problem (that is, what the software accomplishes) than how the software accomplishes it (Vans et al., 1999; von Mayrhauser and Vans, 1996) and therefore represents the maintainer’s understanding of function information embedded in the software. Second, a software maintainer’s program model emphasises how the software accomplishes tasks (von Mayrhauser and Vans, 1995) and is quite closely aligned with the programming domain (Pennington, 1987a). It therefore represents the maintainer’s understanding of state and control flow information embedded in the software. Control flow reflects the sequencing of actions within software, while state information reflects connections between execution of an action and the state of the software when the action occurs (Pennington, 1987a). Third, a software maintainer’s situation model allows a maintainer to avoid cognitive overload by reorganising the knowledge gained through detailed study of the software into higher level chunks and reflects the maintainer’s understanding of data flow information in the software. It therefore serves as a bridge between the domain and program models (Vans at al., 1999). We can consider the domain and program models as being at the opposite ends of a continuum, with the situation model residing between them.
When engaging in software comprehension, a software maintainer invokes one of the possible mental representations of the software (domain, program, or situation model) (Vans at al., 1999), which then drives the comprehension process (Burkhardt et al., 2002). Because the domain model is linked to what the software accomplishes, software maintainers tend to invoke it when they are familiar with the application domain (von Mayrhauser and Vans, 1995). When maintainers lack application domain knowledge, they tend to invoke the program model thereby relying on their understanding of the programming language and standard programming constructs (von Mayrhauser and Vans, 1995). Because the situation model tends to develop after the program model, and only after extensive interaction with a piece of software, it is unlikely to be invoked at the outset of comprehension (von Mayrhauser et al., 1997).
Software modification tasks can be conceptualised as emphasising one of the types of information embedded within software. In this study, we investigated two types of tasks that have been examined in prior modification studies: a control flow modification task and a function modification task. A control flow modification task should be consistent with a program model because it emphasises the way in which software accomplishes tasks (von Mayrhauser and Vans, 1995), while a function modification task should be consistent with a domain model because it emphasises function information (Vans et al., 1999).
We conceptualise software maintainers as creating a mental representation of the IS task based on the requirements of the modification task, the way in which the modification task is presented (external IS problem representation), and their existing knowledge of software development (internal representation of the application domain). The external representation of the modification (external IS problem representation) is a specification of the software modification task such as a narrative or graphic. Essentially, then, the maintainer’s mental representation of the IS task will emphasise the type of knowledge that is emphasised in the task requirements (IS task). The maintainer’s mental representation of the software (application domain) and mental representation of the modification task (IS task) may or may not match in that they may or may not emphasise the same type of information.
This study operationalised cognitive fit by crossing familiarity with the application domain with the type of modification task. A maintainer’s domain model was invoked by using software from a familiar application domain (Vans et al., 1999) and their program model by using software from an unfamiliar application domain (von Mayrhauser and Vans, 1995). A maintainer’s mental representation of the IS task was invoked using a modification task that emphasised function information or one that emphasised control flow information. Hence cognitive fit exists when maintainers conduct either a function task in a familiar application domain or a control flow task in an unfamiliar application domain. Such a match facilitates problem-solving performance. Correspondingly, cognitive fit does not exist when a control flow task is conducted in a familiar application domain and a function task in an unfamiliar application domain. This mismatch results in dual-task interference, which results in lower problem-solving performance.
The theory presented suggests that the fit conditions established would moderate the effectiveness of the comprehension that occurs during problem solving, which is reflected in changes to the mental representation of the software (application domain). Hence the dependent variable in the study was change in the level of comprehension observed during the modification task. The data analysis therefore evaluated the three-way relationship between application domain familiarity, type of modification task, and changes in comprehension.
The propositions presented earlier were supported in this study; that is, the three-way interaction between application domain familiarity, type of modification task, and changes in comprehension was significant. As shown in Figure 2, when cognitive fit exists (familiar application domain/function modification and unfamiliar application domain/control flow modification), the relationships between percent change in comprehension and modification performance are positive; that is, increases in comprehension of the software are associated with higher levels of performance on the modification tasks, and Proposition IS-1 is supported. In contrast, as shown in Figure 3, when cognitive fit does not exist (familiar application domain/control flow modification and unfamiliar application domain/function modification) the relationships are negative, demonstrating dual-task interference; that is, increases in comprehension are associated with lower levels of modification performance. Proposition IS-2 is supported.