Abstract
Protocol analysis is an empirical method applied by researchers in cognitive psychology and behavioural analysis. Protocol analysis can be used to collect, document and analyse thought processes by an individual problem solver. In general, research subjects are asked to think aloud when performing a given task. Their verbal reports are transcribed and represent a sequence of their thoughts and cognitive activities. These verbal reports are analysed to identify relevant segments of cognitive behaviours by the research subjects. The analysis results may be cross-examined (or validated through retrospective interviews with the research subjects). This paper offers a critical analysis of this research method, its approaches to data collection and analysis, strengths and limitations, and discusses its use in information systems research. The aim is to explore the use of protocol analysis in studying the creative requirements engineering process.
Table of Contents
Requirements engineering (RE), an early phase in information systems (IS) development, has been commonly agreed to be one of the most crucial phases in the development process (e.g. Boehm, 1981; Loucopoulos and Karakostas, 1995; Nuseibeh and Easterbrook, 2000). RE is concerned with the elicitation, modelling and specification of user requirements for the new system to be built. Recently, creativity has been increasingly seen as playing an important role in RE (Nguyen et al., 2000; Maiden and Gizikis, 2001; Robertson, 2005; Nguyen and Swatman, 2006; Maiden and Robertson, 2005).
Creativity involves the exploration of conceptual spaces by people in order to produce an outcome that is both novel and useful for a specific context (Boden, 1991; Plucker and Beghetto, 2004; Sternberg, 2005). Based on this understanding of creativity, we see two strong supporting arguments for the role of creativity in RE: creating a vision for ICT-enabled future business practice and developing a requirements specification for an information system to enable the vision. First, creating a vision into future ICT-enabled business practice is crucial in order to develop a new system with an objective to leverage the competitiveness of the organisation and effectiveness of its business functions (Robertson, 2002; Robertson, 2005). Robertson has advocated that the requirements process should involve a creative discovery of requirements to invent business processes rather than passively eliciting requirements from business users as currently described in the RE literature because ‘we won’t make significant improvements to our software products by following a logical train of thought’ (Robertson, 2005). Second, the RE process in its own right is not a purely deterministic, systematic process; it is an exploration of conceptual spaces involving cycles of structured and opportunistic insight-driven episodes (Nguyen et al., 2000; Nguyen and Swatman, 2003). Therefore, fostering and supporting creative thinking within the requirements gathering process is a key to effectively practice requirements engineering.
We argue that one major challenge in fostering and supporting creativity in RE is caused by the difficulty in obtaining a deep understanding of the creative cognitive process involved. For example, while all the practitioners participating in a focus group agreed that creativity was an essential requirement in all of their past requirements projects, they found it difficult to describe how the creative thinking process occurred (Cybulski et al., 2003). In their studies to select and integrate creativity techniques within RE, Maiden and Robertson (2005) criticised the fact that practitioners lack creativity theories and models to guide their creative process in RE. In response to this criticism, Nguyen and Shanks (2006) explored different facets of creativity, and especially different perspectives of creative processes in the creativity literature, and related them to creativity in RE. They concluded that an in-depth understanding of the creative cognitive process is required in order to effectively prompt and support creative thinking in RE.
In this paper, we explore the potential of protocol analysis, an empirical research method in cognitive psychology and behaviour analysis, for use in studying the creative RE process. The following section briefly describes different understandings of the problem solving process in RE and related fields to set a context for the following sections. After that, we describe and critically discuss the protocol analysis research method in terms of current approaches to data collection and analysis, and their benefits and limitations. Then we present a discussion of previous applications of protocol analysis in RE for different research purposes with a view to assessing its relevance to research into the creative requirements process. Finally, we summarise the paper and outline future research directions.