Our brief was to examine the circumstances and settings in which ICT is implemented by Australian organisations in a number of industry contexts. In particular, DCITA was interested to explore the relationships between ICT, the environment, the industry, and organisation and management, and the associated contribution of ICT to business value.

As we had only a few months to collect and analyse considerable amounts of data, we needed to use an efficient method. Simultaneously, our methodology needed to fulfil DCITA’s dual requirements for rigour and relevance and the study had to withstand academic scrutiny as well as provide meaningful lessons and outcomes for the Australian ICT industry.

To compete for the grant, we adopted an integrated multi-paradigm research strategy that involved different types of explorations, building on each other and on previous empirical research. By systematically adding layers of knowledge to our understanding of the problem domain, we could aim to go as far as possible given the time constraints and yet produce rigorous and useful research. To fulfil the requirements of the request for tender, we believed it was critical to follow an efficient theory construction approach in which research actions produced slices of knowledge that can be aggregated and explored in more detail by successive research actions.

Our basic objective was to use statistical methods to discover patterns (also called data mining) as well as interviews and focus groups to obtain rich data upon which we could ground our enquiry before the survey and then further develop and explain the patterns emerging from the survey. The design and execution of these activities took advantage of the diverse research skills within the team. We had six people working in the project: three IS experts (one skilled in the use of grounded theory), two statisticians and one project manager/industry consultant.

The study was designed to have three phases, each including data collection and analysis: Phase One in which the survey instrument was developed, Phase Two in which the survey was conducted and analysed, and Phase Three, which involved the conduct and analysis of interviews and the final integration of conceptualisations, as discussed in the following sections.

Critical to our objectives was the development of a survey instrument based on empirical theory from prior research work, and grounded in the current context of the Australian industry. This grounding activity was essential to ensure a sound basis for identification and specification of constructs, rigour in the project methodology and to give legitimacy and credibility to the results. The flow of main tasks and outputs of this phase are represented in Figure 1, which is explained in more detail below.

We investigated the literature across empirical studies conducted worldwide. The aim was to find well-tested survey models, although as a starting point only since uncritically relying on those models could ignore peculiarities of the Australian context. In parallel with the literature review, we convened a series of focus groups with business executives and senior ICT managers.

The focus groups were central to the initial development of the survey instrument. A total of 27 organisations took part in four focus groups held in Melbourne, Sydney and Canberra. The results were used to ensure that the survey instrument was grounded in, and relevant to, the current circumstances and settings of Australian organisations across a range of industries, organisation sizes, and geographical locations.

The focus groups used the Nominal Group Technique (Delbecq et al., 1986) to develop and prioritise a list of issues associated with the successful use of ICT. This technique allows the participants to discuss and prioritise their own issues rather than forcing a prioritisation based on extant literature. The experience of the focus group participants regarding the benefits arising from ICT aligned well with previous empirical research that categorised benefits as being strategic, informational and transactional. Factors identified as important to successful ICT use included: strategic planning, education and training, cost, support for ICT, effective client-supplier relationships, business case understanding, and effective management decision making. These key observations from the focus groups were used as inputs to the development of a draft survey instrument.

To help reduce the ambiguity often present in questions, the draft survey instrument was then refined through seven face-to-face cognitive interviews conducted in Sydney and Canberra. Participants were asked to report on how they would arrive at their answer to a particular question rather than on what their answer would be. For example, what meaning did they ascribe to the terms used and what context did they apply when considering their answer.

Based on the results from these interviews, we identified many problems that respondents had in understanding questions, and revised the survey questions accordingly. Finally, the survey instrument was vetted and cleared by the Australian Bureau of Statistics (ABS) Statistical Clearance House, a mandatory step for government-sponsored research in Australia.

The aim of Phase Two was to collect and analyse the data, searching for important concepts and their attributes. The flow of main tasks and outputs of this phase are represented in Figure 2, which is explained below.

The data consisted of observations of a number of variables for each of the 1050 organisations surveyed. The sampling frame was drawn from the Dun and Bradstreet database of Australian organisations. The sampling methodology used stratification by business size, so that the final sample included 225 large organisations (100 or more employees), 450 medium organisations (20-99 employees) and 375 small organisations (5-19 employees).

The approach was also intended to cover 15 of the Australian New Zealand Standard Industry Classification (ANZSIC) codes, so initially quotas by industry as well as size were planned. While the allocated sample size was not met for all industries, the number of responses obtained within each of the 15 industries was sufficient for further analysis. The initial sampling frame included 5380 organisations. Advance letters were sent to 2600 of these organisations to increase response rates and 63% of these companies were also contacted by telephone. The response rate was 31%. Details of the actual numbers of responses obtained by industry as well as organisation size are provided in Table 1.

The survey was telephone-based since this technique enabled survey data from a large number of Australian organisations to be gathered within the required time and cost constraints. Also, to alleviate the problem of the time constraint, we contracted a quality-accredited research organisation, here called Social Research Group (SRG), to collect data for the research team using a computer assisted telephone interviewing (CATI) facility. After the first 20 interviews, which were used as a pilot, all fieldwork was completed in time, between 8 June 2004 and 7 July 2004, and we were ready for the data analysis stage.

The purpose of our data analysis was to discover patterns and to detect significant structure in the data. Many modern statistical procedures begin with the simple purpose of detecting significant structure in high-dimensional data; that is, where the dimension of the data is in the hundreds of covariates. The term ‘significant structure’ has many interpretations, the two most useful of which state that a significant structure exists when:

These interpretations reflect the broad principle of parsimony. That is, they attempt to answer the question ‘can the information carried in a high-dimensional data set be adequately understood in a low-dimensional analogue?’ and, further, to explore and interpret relevant relationships within the resultant low-dimensional space.

Structure of the first kind has classically been captured by parametric regression models, which seek to represent the structure in terms of straightforward linear models relating response and covariates. The strength of such models is that they are very compact, low-dimensional representations of the data, but their main drawback is that in cases where the underlying structure is not simple or linear they can fundamentally fail to capture what is really going on in the data. A broad overview of statistical modelling techniques for discovering structure in data of the type we describe in this paper is given in an excellent monograph entitled The Elements of Statistical Learning (Hastie et al, 2001). This book describes basic parametric techniques and discusses in detail the strengths and weaknesses of the classical parametric approach to model fitting.

More modern, non-parametric techniques relax classical linearity and distributional assumptions, thereby allowing more flexible and realistic models, which are inherently data-based. Examples of such methods include:

The key advantage of these techniques is their flexibility — they essentially trade off extremely compact descriptions of the response surface (such as simple lines or planes) against better fitting but more complex models.

In our analysis, the response variable — that is, the construct we wished to model in terms of other properties of the businesses involved — was not initially precisely defined as the outcome to a single survey question but was rather summarised in survey responses to 22 interview questions. In other words, the response space was initially a 22-dimensional space, and the first goal of the analysis was to isolate a smaller space spanned by interpretable variables that could be used as response variables in our model, and which adequately captured the important properties of the original 22-dimensional response space.

We employed a classical multivariate technique, factor analysis, to discover meaningful structure amongst covariates, and to provide low-dimensional analogues to high-dimensional covariate spaces.

Factor analysis seeks to account for variability in the original high-dimensional space by identifying relatively few common, underlying factors (linear combinations of the original covariates) that explain similar variability to that possessed by the original high-dimensional space. Further, factor analysis attempts to assign simple meanings to the resultant factors based on the characteristics of their constituent variables. Factor analysis specifically posits the existence of a small number of common factors that account for a reasonable proportion of the variability in the higher-dimensional space up to a set of ‘unique’ elements that cannot be accounted for by the small set of common factors. Factor analysis is described in detail in many texts on multivariate statistical methods — see, for example, Lattin et al. (2003).

We carried out a factor analysis in an attempt to reduce the complex 22-question response into a form that was both lower-dimensional and easily interpretable in terms of constructs isolated in the foregoing qualitative analysis. A four-factor solution isolated four broad types of value arising from the use of ICT: informational (i.e. increasing the quality, quantity and availability of valuable information), strategic (i.e. creating a competitive advantage), transactional (i.e. leading to efficiencies) and transformational (i.e. enabling organisational change).

The four-factor solution allows a compact representation of the complex response space in terms of easily interpreted factors, the features of which arose from the earlier, qualitative phase of our investigation. These four factors together explained over 60% of the variation present in the original 22-dimensional response space, and so this solution allowed for both significant dimension reduction as well as a confirmation of existing notions of what sorts of benefits might arise from ICT implementation.

The third and final phase of the project was designed to expand and explain concepts emerging from the previous phases and to integrate these concepts into a report. To achieve this aim, we used both quantitative and qualitative techniques. Figure 3 represents the activities of this phase, which are explained in more detail below.

As described above, Phase One was mainly qualitative and Phase Two was mainly quantitative. In Phase Three, however, this distinction becomes to a certain extent meaningless. Once the analysis of our interview data started to result in concept emergence, we were engaged in an almost simultaneous process of comparing ideas and concepts emerging from the interviews with the quantitative data from the survey. Through the constant comparison of these concepts against the raw data collected in Phase Two, we were able to test the conceptual validity of what we considered important. Nevertheless, to explain how we conducted our work, the next two subsections describe the qualitative and quantitative components separately.