Researchers in the psychology of problem solving and design studies have recognised the importance of describing and understanding the cognitive process used by the problem solver. The belief is that a good understanding of the cognitive process would be useful to support and improve the problem solving and design practice and to effectively train practitioners. Protocol analysis is an empirical research method for studying the cognitive behaviours and thought processes used by problem solvers (Ericsson and Simon, 1993).
Protocol analysis usually takes place in a controlled environment. The research subject is a problem solver who is given a specific task and works individually on that task. Protocol analysis aims to collect as much detail as possible about the problem solving process, analysing the collected data and reconstructing what happens in the mind of the problem solver. On one hand, the controlled environment reduces ‘noise’ and allows the researcher to collect rich details and relevant data about the problem solving activities and artifacts produced during the experiment. On the other hand, shortcomings of this research method include a limited time period, a small problem, and the exclusion of social processes, teamwork and communication that often take place in everyday work.
There are different approaches to conducting protocol analysis. We discuss them in terms of data collection and data analysis.
There are two approaches to data collection in protocol analysis: concurrent and retrospective (Dorst and Dijkhuis, 1995; Ericsson and Simon, 1993).
Concurrent protocols are generated when the problem solver verbalises their thoughts while working on a specific task. First, the problem solver is trained to verbalise his or her thoughts using a thinking aloud technique. Second, with a given task, the problem solver verbalises his or her thoughts while working on a given task. The process is video and/or audio taped, and transcribed. As a result, a thinking aloud concurrent protocol acts as the generator of the data source, which is then later coded and analysed.
Two assumptions underlie the validity of the verbalisation of thoughts process in concurrent protocols. The first is that the problem solving process has a conversational characteristic. Schön (1996) described design as a reflective dialogue of the problem solver with the materials of a problem situation. In developing a design rationale tool, Kaplan (1990) viewed the design process as a conversation-oriented activity, being either a monologue by one designer or conversations between different designers. The second is that the verbalisation of thoughts during the problem solving process will not affect the process. Ericsson and Simon (1993) describe three levels of verbalisation ranging from direct verbalisation without special effort to communicate thoughts, minimal intermediate processing to explicate the thought contents, and verbalisation with an explanation of thoughts, ideas and motives. Having reviewed empirical studies using these levels of verbalisation, Ericsson and Simon (1993) concluded that concurrent verbalisation does not alter the structure of thought processes. There is a disagreement about this conclusion. Lloyd et al. (1995) were concerned with the validity of concurrent protocols because thinking aloud may interfere with the problem solving process and, consequently, concurrent protocols may be incomplete and not reveal true insights into the actual problem solving process. A common view shared by the design studies research community is that concurrent protocols reveal a sequence of cognitive events and information processing stored in short-term memory (STM), thus providing rich details and opportunities for analysis to gain insight into the cognitive behaviours by the problem solver.
Retrospective protocols conduct interviews with the problem solver after the problem solving process, usually immediately. During the interview, the problem solver is asked to recall his or her activities. Interviews are audio and/or video taped and transcribed. The generated retrospective protocols serve as data for later coding and analysis to reconstruct the problem solving process and gain insight into what happened during the process.
While both concurrent protocol and retrospective protocol approaches share a common position that collected data can be used to reconstruct the problem solving process, the latter is often seen as less intrusive to the process under observation (Lloyd et al., 1995). However, Ericsson and Simon (1993) have argued that, after the experiment session is complete, information processing details are no longer accessible from STM because they have been transmitted into Long Term Memory (LTM) from which it is harder to retrieve. Consequently, the reconstructed process based on a retrospective protocol may be incomplete and inaccurate. Retrospective protocols may not show the actual sequence of cognitive events, instead they may show a rationalised or theorised story of the problem solving process. To address this in design studies, Suwa et al. (1998) suggested videotaping the design experiment session and using the videotapes to assist the retrieval of the cognitive events stored in LTM after the experiment session. In addition, the contents (sketches and diagrams) can also be collected for analysis. Guindon (1990a) supplemented her concurrent protocols with retrospective interviews to obtain additional design rationale and to gain a deep understanding of the designer’s cognitive behaviours and the design process.
Gero and Tang (2001) conducted an empirical study to examine similarities and differences between concurrent and retrospective protocols. They found that both types of protocol methods show a similar frequency of changes of design intentions and consistent structures of the design process. They also found that the number of segments in a retrospective protocol is larger than the number of segments in a concurrent one. They explain that, through a revision of sketches and rehearsed memory after the thinking aloud session, the retrospective protocol produced more details than the concurrent protocol (Gero and Tang, 2001). The authors concluded that concurrent and retrospective protocols lead to similar results and that the concurrent protocol is an efficient and applicable method in understanding the design process.
Kuusela and Pallab (2000) conducted a similar comparative study using an experiment set in a context of customer decision making. Although the problem solving contexts and coding methods in studies by Gero and Tang (2001) and Kuusela and Pallab (2000) are different, a common conclusion was reached, namely that both concurrent and retrospective protocols lead to consistent understandings of the problem solving process. In addition, Kuusela and Pallab (2000) suggest that concurrent protocols are more suitable for examining the process while retrospective protocols are more suitable for examining the outcome. Their conclusions support the potential use of protocol analysis to gain insight into the problem solving process in RE.
There are nevertheless two weaknesses with both concurrent and retrospective protocols. One of these is the well-known Hawthorne effect since both of these data collection approaches involve observation of a research subject who knows they are being watched. Other research approaches such as, for example, case study, action research and ethnography, also share this limitation (Neuman, 2003). Another weakness of protocol analysis is the difficulty in recruiting and training participants who are willing, capable and motivated to provide meaningful protocols. Previous successful applications of protocol analysis in design research have addressed this issue by explaining to participants the significance of the research and providing training that facilitates thinking aloud and articulating ‘on the fly’ thoughts.
Data generated using either concurrent or retrospective protocols are coded (segmented) for the analysis and identification of cognitive patterns. First, the data is coded into segments. Often a change in the problem solver’s intention, or the contents of their thoughts, signals a new segment. Second, the problem solving process is reconstructed as a sequence of coded segments. Finally, correlations between segments are identified. Based on the two views of the design process, rational problem solving and constructivist, there are two approaches to segmenting data: process-oriented and content-oriented (Dorst and Dijkhuis, 1995; Gero and Neill, 1998).
The process-oriented segmentation approach aims at describing the design process as a sequence of problem solving activities, using a problem solving taxonomy such as, for example, problem recognition, goal setting, solution proposing, solution analysing, or top down vs. bottom up strategies. In this approach, the protocol transcriptions are often coded into segments by syntactic markers, such as pauses, intensity, intonations, phrases and sentences that then aggregate into cognitive units called design intentions or design moves, for analysis (Ericsson and Simon, 1993). Alternatively, protocols can be directly segmented by design intentions based on the problem solving taxonomy — for example, problem domain including abstraction levels, functions, behaviours, structures; and micro and macro design activities such as proposing solutions, analysing solutions, explicit strategies, top down, bottom up, opportunistic (Gero and Neill, 1998). The categorisation of design intentions is often determined before the segmentation of the protocol. Gero and Neil (1998) also suggest open segmentation of protocols to allow new categories to emerge during the segmentation process. The segments generated from the protocol are often quantitatively analysed to identify time spent on different types of design intentions, and to reconstruct a sequence of, and correlations between, them.
Benefits of process-oriented segmentation include: a design process described in the form of a sequence of design intentions and an understanding of correlations between design intentions, often presented in a graph form. Dorst and Cross’s (2001) protocol analysis, involving an evaluation of nine creative designs in industrial design experiment, offered a refined model of a co-evolution of both the problem space and solution space. Their study supported Schön’s (1983) argument that insight-driven problem (re)framing is crucial to the creative design process. Another example is a study by Guindon (1990a) involving eight designers in a lift control software design experiment. This study is often cited in the RE literature. Using a process-oriented segmentation method to examine concurrent protocols produced in this study, Guindon (1990a) observed significant deviations from a systematic structured process. She was amongst the first authors to propound opportunistic cognitive behaviours in high-level software design. Opportunistic behaviours and deviation from a structured process were also observed and reported in requirements engineering by Khushalani et al. (1994) and Nguyen et al. (2000).
Dorst and Dijkhuis (1995) have criticised the process-oriented approach on the basis that it fails to examine what designers see and think and what knowledge they exploit. This weakness can be addressed using the content-oriented segmentation approach.
The content-oriented approach to protocol segmentation focuses on the cognition of the problem solver; that is, what he or she sees and thinks and what knowledge he or she uses (Suwa and Tversky, 1997; Suwa et al., 1998). There are two types of cognitive contents: visual contents (depicted elements and their spatial relations as drawn in the artifacts, and movements such as eye movement, moving pencils, etc) and non-visual contents (including thoughts and knowledge). A well defined classification of content-oriented segments (Tang and Gero, 2000) includes:
Physical — depiction, looking, motion;
Perceptual — perceiving depicted elements and their relations;
Functional — assigning meaning to depictions/perception; and
Conceptual — goal setting and decision making.
To study discontinuity and unexpected discoveries in the design process, design segments are indexed as being new, continual or revisited.
The content-oriented segmentation approach has been found to be useful in examining cognitive interactions between designer and artifacts. Using a content-oriented segmentation classification scheme, Suwa et al. (1998) found that sketches can seen as an external memory useful for subsequent inspections, visual cues for functional actions, and a physical setting for functional thoughts to be constructed on the fly in the emergent problem situation. The use of sketches was also investigated in a recent study (Bilda et al., 2006) using a revised content-oriented segmentation scheme. This study found that sketching or externalising may be useful but not necessary to design in terms of developing a network of ideas, pursuing cognitive activities and obtaining a satisfactory outcome. As systems analysts often use requirements models to represent and communicate requirements with each other and with other stakeholders, interactions between systems analysts and requirements models can be examined using content-oriented protocol analysis.
According to Tang and Gero (2000), there are two types of content-oriented segments and both are essential in the design process. Goal-driven segments reflect the rational problem solving process (Newell and Simon, 1972) and sensor-driven segments reflect the constructivist and reflection-in-action process (Schön, 1983). To us, this observation can be related to the description of catastrophe cycles in the requirements gathering process (Nguyen et al., 2000; Nguyen and Swatman, 2003).
In summary, the content-oriented and process-oriented segmentation approaches can both be beneficial. In RE, the invention or discovery of requirements and changes to requirements models should be studied in relation to associated cognitive behaviours to evaluate the creative requirements process and their impact on the creative outcome. There are, though, two common weaknesses from the point of view of RE in current segmentation classification schemes. First, both process-oriented and content-oriented segmentation approaches need to be adjusted to the RE knowledge domain, tailored, for example, to a particular requirements method and process. Second, segment classification should be linked to different types of creativity and creative thinking styles such as, for example, exploratory, combinatory, analogy, transformation, structured and unstructured (Boden, 1991; Ward and Finke, 1999; Sternberg, 2005).
Protocol analysis is widely used in problem solving research, especially in design studies. As the debate about the strengths and weaknesses of protocol analysis continues, this research method evolves. In terms of data collection protocols, comparative studies tend to confirm that concurrent and retrospective protocols produce similar results (Tang and Gero, 2000; Kussela and Pallab, 2000). In terms of data segmentation and coding, segmentation schemes are developed to enable researchers to gain in-depth understandings of the process as well as the interaction between the designer and artifacts (Gero and Neill, 1998; Tang and Gero, 2000; Bilda et al., 2006).
Protocol analysis has also been adopted and adapted to studying thinking processes in teams. For example, Stempfle and Badke-Schaub (2002) recorded team concurrent communication and analysed the generated protocol sentence by sentence. They developed a new coding scheme to examine collective design actions. Amongst others, important findings concerned the structuring of group process and, a continual ‘interweaving of content-oriented and process-oriented sequences’, and a tendency to immediately evaluate new ideas by team members (Stempfle and Badke-Schaub, 2002). In our view, since the pseudo-concurrent protocol did not capture verbalised thoughts, the retrospective protocol may be complementary: a combination of intermediate artifacts, video tapes and retrospective interviews can be useful in reconstructing multiple cognitive processes and teamwork dynamics. Distributed cognition theories can be also adopted to investigate creative team processes.
It is interesting to observe that, between the 1990s and early 2000s, design studies and RE researchers have come up with similar observations about the creative, emergent problem solving process and the co-evolution of the problem space and the solution space. But researchers in design studies have used protocol analysis, and proactively invented new segmentation schemes to examine the creative design process while RE researchers have used other research approaches, as will be discussed below.