Protocol analysis has been applied to the study of the cognitive behaviours of software and database designers (Guindon, 1990a; Sutcliffe and Maiden, 1992) and systems analysts (Batra and Davis, 1992; Chaiyasut and Shanks, 1994). A majority of these studies focus on categorising cognitive behaviours exhibited by systems designers or analysts and/or examining similarities and differences between novices and experts.

Guindon (1990a) discovered that the ill-structuredness of the requirements problem was an important factor inducing the opportunistic behaviours of the software designer. The opportunistic behaviours are associated with inferences related to new, emergent details associated with the incompleteness and ambiguity of the ill-structured problem. Often, upon sudden discovery of such details, a designer tends to immediately develop new partial solutions and test and modify them, rather than continuing to work on their previous planned task at a higher abstraction level. Their traversal between different abstraction levels was not systematic.

Sutcliffe and Maiden (1992) analysed verbal protocols supplemented by retrospective questionnaires from the development of a requirements specification for a delivery scheduling system. They were able to categorise and model the cognitive behaviours as consisting of complex dependencies between information-gathering, assertions, conceptual modelling, planning, recognising goals and reasoning. According to these authors, the strongest associations were between information-gathering, assertions and conceptual modelling. These associations were explained as a representation of the analytical side of understanding the problem domain.

Batra and Davis (1992) examined similarities and differences between novice and expert database designers and concluded that novices focused on structuring requirements while experts’ efforts were directed towards developing a holistic understanding of the problem, abstracting, categorising and representing. They noted cyclic movements between problem understanding and problem modelling by experts. With a focus on the data aspect, Chaiyasut and Shanks (1994) examined differences between data models produced by expert and novice data modellers. The authors categorised the cognitive process into six detailed types and noted that novices’ models were developed ‘literally’ from the problem description while experts' models were more comprehensive, complete and held a holistic view of the problem.

Other studies using protocol analysis in requirements research are not related to the creative requirements process. For instance, protocol analysis was adopted as a research method in evaluating conceptual tools in modelling composites, data and properties (Shanks et al., 2003). More recently, Owen and his colleagues (Owen et al., 2006) criticised a lack of applications of protocol analysis in software engineering research and demonstrated benefits of protocol analysis as a research method in gaining valuable insight into how human factors influenced the interpretation and use of technical documentation by systems developers. Interestingly, protocol analysis was not only seen as a research method but also suggested as a way to observe and learn about the requirements problem context through users’ work patterns and behaviours. Protocol analysis was also included as a technique in the ACRE framework, a framework to guide practitioners in requirements acquisition (Maiden and Rugg, 1996).

We have reviewed various research approaches to studying the creative RE process and behaviours of the systems analysts. The table below summarises our findings in terms of their strengths and weaknesses.

As we have argued, cognitive behaviours are important in studying creative processes in RE. In obtaining data about the cognitive behaviours, a challenge faced by researchers is that creative thinking involves both verbal and non-verbal activities. While verbal activities (meetings, conversations, requirements workshops, and group brainstorming) can be observed directly to generate data, non-verbal activities (silent cognitive behaviours that occur in the mind of the systems analyst) are much harder to access. Therefore, protocol analysis can be a key research method to gain insight into cognition and creative thinking in the requirements process. Surprisingly, protocol analysis was used to study cognitive behaviours in RE in the early 1990s although it has not been used specifically in studying the creative requirements process.

We suggest two possible applications of protocol analysis: using it to examine creative thinking and cognition in the creative requirements process; and using it to evaluate different requirements processes that utilise creativity techniques (such as that of Maiden and Robertson, 2005) and that do not utilise creativity techniques (for example, UML in Dennis et al., 2002) in relation to assessing the creative outcome produced through using these different RE processes.

Protocol analysis comes with inherent limitations: limited generalisability to real commercial projects and weak suitability to study collaborative process. To address these, a combination of research methods can be useful. In fact, a number of authors adopt a combination of different research approaches. For example, Simsion (2006) used interviews, surveys and laboratory experiments in studying creativity in data modelling, and Khushalani et al. (1994) used workshop observations and laboratory experiments to examine opportunistic behaviours by systems designers. Protocol analysis can, we argue, potentially be used with other fieldwork research approaches and to study cognitive behaviours in the creative, emergent and collaborative process in RE in particular.