One of the most significant issues in developing an effective
eEngagement process is careful planning of the post-implementation
activities for the project. This has two elements:
The Importance of Evaluation
There is little need to reiterate the importance of evaluation in
the public sector. Calls for discussion of debate around and
methodological experimentation with evaluation have been hallmarks of
public sector management reforms for the past decade. Any project
initiated in the public sector today will make provision for evaluation as
a standard operating procedure.
In the context of a new type of activity, however, careful
consideration of evaluation is important. This is because:
-
while most (if not all) governments in Australasia stress the importance of public participation and engagement,
the practical commitment of governments is often quite variable. The relative newness of these activities often creates an
environment in which novel or innovative approaches to community engagement are often subject to higher levels of scrutiny
and assessment. This, combined with the potential to generate greater levels of feedback
about the process itself, can put the innovating public sector manager under a degree of scrutiny not shared by managers
following ‘tried and true’ (but possibly ineffective) strategies to engage the public;
-
the area is new and requires grounded, honest evaluation of the cost and benefits of a range of different approaches.
While it is likely that eEngagement will continue to be an important part of the armoury of public sector managers for the
foreseeable future (if not increasingly important over time as our society develops greater levels of technical sophistication
and complexity), effective and practical evaluation of the vast array of models and techniques will lead to better means to
assess the benefits of one approach over the other, making planning faster, implementation easier and the outcomes more effective;
and
-
the use of ICTs can support new approaches to evaluation, increasing the effectiveness of this part of the management
process and leading to higher levels of understanding about what works and what does not, than in offline activities. This
is a direct result of the interactivity
of the media employed and their capacity to support the automatic collection of user data.
Approaching Evaluation for eEngagement
The exact nature of evaluation will be highly variable depending
on the mechanisms and approaches employed (and objectives). Whyte and
Macintosh provide a useful conceptual tool for evaluating
eEngagement activities, focusing on political, technical and social
outcomes of the project or process. This approach is recommended for any
eEngagement activity and asks the following questions:
- political evaluation:
Did the process follow best practice guidelines for undertaking consultations that are published by government
and were the stakeholders satisfied with the process? The evaluation factors here are similar to those for conventional consultations
but need to be answered by different means;
- technical evaluation:
To what extent did ICT design directly affect the outcomes? In designing the e-consultation there is a need to
take account of the technical skills of the target audience and locality of the participants. Here, we can take as our starting
point established evaluation frameworks from the software engineering and information systems communities, together with considerations
of usability and accessibility; and
- social evaluation:
To what extent did the social practices and capabilities of those being consulted affect the consultation outcomes?
In particular, what bearing do these have on the relevance of consultations to the consulted citizens, the relevance of their
contributions to each other and to policy makers and the nature of the interaction?
Common traps to avoid in developing the evaluation framework
are:
- over-emphasis on technical assessments:
Technical issues are often easy to document and can be clearly presented in terms of equipment ‘up time’, budgetary
management and ease of systems implementation. While these issues are important, it is important to keep them in perspective
and not lose sight of the broader objectives (e.g. technology merely facilitates the process, it is not the end product);
- excessive use of simple metrics:
Many consultation processes are assessed purely on the basis of number of participants, or amount of content generated.
While this has an important role, it is critical to also ask:
- ‘right people’ versus ‘many people’;
- what are the characteristics of the people engaged (e.g.
were ‘new’ people brought into the process, does information flow
though these people to a wider audience – are they
‘influentials’?); and
- picking the right comparisons: If the eEngagement process has been implemented to assess a consultation or participation deficit, the particular approach
used will be assessed against the previous state of play (before-after assessment), rather than with other examples that use
the same technology or methodology – these latter types of comparisons are often of limited value.
What to Consider in Effective Assessment
When developing the assessment approach, it is important to
consider:
-
the extent to which the technology can support longitudinal assessment processes (e.g. performance measurement
over time, or reducing a long ‘end of process’ survey into a series of small polls);
-
user views and experiences (sometimes best expressed qualitatively). Consider allowing the users to develop and
present their own evaluation frameworks
(a variation on self-assessment reporting);
-
‘knock-on’, capacity-building, or social capital
formation outcomes (skills transfer, mobilisation, organisational outcomes and benefits); and
-
the development of real-time
and automatic metrics. A good example of this would be the ability to incorporate comprehensive analyses of user
browsing patterns with respect to online information (e.g. pages viewed, time spent viewing each page, pages with highest
levels of referral to others, etc.). These metrics allow us to analyse (for better or worse) the value of our content in a
way that print run numbers of consultation
documents cannot. These statistics can often be provided by the service provider (such as the website hosting service
or from the telecommunications provider) or through commercial services (e.g. Nielsen//Netratings).
Exhibit 29: Evaluation Example – Local
Issues Forum Success Measures (longitudinal)
|
2-3 Months
|
- forum is still active
- some regular traffic
- experiencing some membership growth
- city / community officials are aware of forum /
may be reading posts
- some community organisations have begun to post
announcements in forum
|
|
6 Months
|
- 25-50 percent growth in subscriptions since
launch
- local media is to paying attention to
discussions
- 10 or more ‘regular’ posters (post at least once
per week)
- participants are starting new
discussions
- regular participation in steering committee
communications and meetings attract a diverse group of
community members
|
|
1 Year
|
- elected officials and city / community staff are
participating – most lurk, but some post
- 50-100 percent growth in subscriptions since
launch
- occasional story in local media that originates
from forum
- some examples of citizen or government action
that have resulted from forum discussions
- you have hosted at least one in-person gathering
or party for participants to meet one another
|
E-Democracy.org 2005, Local Issues Forum
Guidebook