A common complaint about logic modelling is its vulnerability to being captured by conventional thinking. Once a policy or program backbone has been created, it tends to look authoritative and people may think twice about questioning it. Over the last several years, as New Zealand departments prepared their department-wide ‘outcomes hierarchies’ or logic models to be included in Statements of Intent, some concerns have been expressed about the tendency for departments to simply use the logic model format to rationalise and justify their status quo policies and outputs, rather than using it to examine critically their mix of outputs. The litmus test of outcomes-based management, and logic modelling, when used as one of its tools, is whether or not it gives departments a platform from which to make changes in their output mixes to boost effectiveness. It is not yet clear that such changes are occurring as hoped.

Does this absence of discernible impacts mean that logic modelling is either a bad idea or hopelessly unrealistic or both? On one hand, a methodological purist would have to condemn the kind of retrofitting application of logic models that departments are suspected of using in New Zealand. Good practice in intervention logic clearly emphasises the absolute necessity of revealing assumptions and risks associated with each backbone and critiquing each step in the causal logic. It is not meant to be used for rationalising or shoring up either current policy or someone’s favoured proposals for change. On the other hand, no one should be surprised when departments respond to an official request for performance-related information by defending their existing programs. When faced with official reporting requirements, no matter how non-threatening the language of the requirement, rational departmental officials will always use whatever tools are available to weave the most positive picture possible of their department’s development. Officials know that all reporting can and will be used to construct a ‘performance story’ tied to their departments (Mayne 2004), and it is natural for them to want to control that story to the greatest extent possible. ^[3] For this reason, central agencies probably ought to acknowledge quietly that some sugar-coating of departmental performance stories is tolerable, but only if robust procedures are in place for scrutinising these performance stories and asking the kinds of hard questions that will reveal areas of weakness. Parliamentary committees, auditor agencies, and other institutional actors will probably play big roles in this. It may be time for government to direct some of its attention away from making marginal improvements in departmental reporting and direct it toward building more effective scrutiny and feedback arrangements.

Gregory (2004) has argued that political imperatives will always swamp serious efforts by policy advisers to question the government’s policy thinking. Therefore, according to his argument, expecting policy advisers to use logic modelling techniques even for internal policy advice may be unrealistic. Even if Gregory is overstating somewhat the obstacles to free and frank policy advice, it is probably hopelessly idealistic for us to expect departments to publish reports on their websites that contain logic models revealing their current program’s deepest vulnerabilities, particularly if the department is still early in the process of addressing those vulnerabilities. Good practice in logic modelling, if it happens anywhere, is most likely to be found in policy and management teams that are working behind the scenes to improve program effectiveness.

Transparency and accountability are fundamental values in a democratic system, and therefore, central agencies must specify particular forms of reporting that will be uniform across agencies. But at the same time, the art of crafting an effective public sector reporting system requires a delicate balance between Parliament’s and the public’s need for detailed information for assessment, on one hand, and departments’ needs for time and space to carefully analyse and sensibly address problems behind the scenes, on the other hand. Rather than requiring departments to report on current weak points and future risks, I wonder if it might not be better for central agencies to require retrospective reporting from each department about how it has identified and addressed weaknesses in the recent past and learned from them. Knowing that they will be held accountable for evidence of recent learning may encourage officials to look for genuine learning opportunities now and in the future. This sort of retrospective reporting of learning is surely not game-proof, but it may provide a stronger incentive for genuine risk assessment and change management than the more direct reporting requirements provide.

Central agencies in New Zealand (and those advising them, including the author) are learning from their own recent experiences with intervention logic. The guidance documents for agencies around Managing for Outcomes and preparation of Statements of Intent made sustained and explicit reference to intervention logic techniques in 2002, while the 2003 and later versions of these documents did not endorse any particular methods for articulating the rationales linking outputs to outcomes. The more vigorous promotion of intervention logic in previous guidance documents generated backlash among some officials who felt that they were being forced not only to learn a new technique at relatively short notice, but also having to apply it to the broadest possible canvas – a whole department. Although central agencies provided considerable support, the task was probably too much too soon, and there were probably too few positive, internal incentives for departments to really get stuck into the task rather than simply ticking the box.

In hindsight, it may have been better to allow the interest in and enthusiasm for logic modelling to spread at a more natural pace across public service agencies and departments in New Zealand, fuelled by word of mouth and evidence of effectiveness. In addition to being slow, this kind of dissemination is hard to control, of course, and difficult to harness for purposes of reporting and assessment. It surely would have generated a potpourri of practices, very little uniformity, and huge difficulties for anyone trying to compare the effects of logic modelling practices on different agencies. In this case, however, a natural proliferation of practices may have been just what was needed to generate innovation and change, particularly with respect to practical areas where logic models are only recently being applied – including policy design, public management, strategic planning, and project management.

A mysterious and powerful force has often been observed in logic modelling exercises, drawing participants away from thinking about intermediate and final outcomes towards thinking about internal government processes, such as: have we followed procedures? are we setting up the right networks? are we on time and within budget? are the operations people doing what they are meant to be doing, according to the policy? is the minister happy? These are urgent internal matters, and the public management/project management matrix associated with a logic model is suited to addressing these, but they should not to be confused with intended outcomes themselves. Outcomes (intermediate or final) are the consequences of government activity, as experienced by something or someone outside government – such as health status, educational achievement, border security, sustainable fish stocks, and the like. They are not processes of government activity. Logic models should start by thinking about the chains of outcomes that a particular program or policy is meant to produce, before turning to questions of resource allocation, staffing, procedures, networks, and other essential ingredients for making the chain work.

Department-wide logic models may quickly become unreadable if all logical steps are included for all core programs (not to mention assumptions and risks as well). These ‘copulating spider’ diagrams, as current State Services Commissioner, Mark Prebble, once described them, are a common pitfall of logic modelling practice. Not only are they hard to read, but they also tend to miss out on opportunities to cut to the chase. Some programs’ and departments’ causal theories can be summarised simply and elegantly in just a few basic propositions; using a logic model to make these more complicated than necessary is a poor use of time and resources. Logic models are most useful when a department or program group can use them to stay focused on a few vital outcomes and on key stages in the output-outcome conversion process. A good logic model should reveal these logical ‘hinges’ rather than obscuring them (Baehler 2003).

However, those who use logic models to portray policy intentions will often find themselves under pressure to include in their diagrams every possible feedback loop and causal variable that might influence the policy’s effectiveness. This is where policy logic models begin to shade into systems models of the policy environment. The tendency to drift from one technique to the other is natural and not necessarily to be avoided, but at the same time, modellers should try to keep certain important distinctions between the two practices clear in their own minds. The two most important distinctions, in my opinion, are between intention and reality, and between a policy idea and the setting in which it is meant to function.

The world is obviously a very complex place. Much social and even economic behaviour is notoriously difficult to explain and predict. No one can be certain how events and developments will be influenced (if at all) by any single government intervention, much less a complex array of policies and programs. Public policy experts like to repeat the mantra that we are constantly besieged by ‘wicked problems’ that morph before our very eyes and cannot readily be defined let alone solved using standard analytical tools. Brave souls in the social science world are now exploring how insights from complexity theory can be applied to wicked problems to help us understand them. These truly are important developments and worthy of more attention.

At the same time, however, it is important to remember that the basic logic of most government interventions is relatively simple. Policy logic has to be simple and forceful because (1) politicians have to be able to explain policies to stakeholders and citizens in order to win their support, (2) government policy itself is a fairly blunt instrument and delicate operations involving complex interactions of multiple variables are generally the responsibility of frontline staff and implementers (the algorithms of judgment used by frontline staff are virtually impossible to incorporate into policy or to express in a policy logic), and (3) as implementation scholars discovered decades ago, the more moving parts a policy or program has, the more opportunities there are for things to go wrong. For all of these reasons, intervention logic seeks to keep the policy backbone model relatively simple and straightforward. Logic backbones are usually linear to reflect the linear nature of most policy arguments – for example, if we subsidise or otherwise facilitate X, people will consume more of it; if we regulate Y, the harms associated with it will decrease; if we provide a new service Z, the target population will be enabled to function more effectively. As described above, laying out the intended logic of these interventions allows us to test it against what we know about how the real world works. Intended policy logic and social/economic reality need to be closely related to each other, but they are not the same thing.

This is where systems models come in. ^[4] They allow us to map what we know about how a relevant piece of the world works, such as economic development processes, or family formation processes, or cycles of environmental degradation and repair. When policy analysts and evaluators are testing the logical chain of impacts associated with a particular policy or program, they must draw upon knowledge of the many complex and often chaotic influences and drivers that characterise the actual world into which policies and programs interject their resources and rules. Systems models are simply tools for describing what we know about this complex reality. The relationship between logic models and systems models is therefore a reflection of the relationship between intention and actual effect. Logic models plug into a systems model at one or more points with the intention of showing how a policy is expected to break a problem circuit or create opportunities for new patterns of interaction to emerge. Logic models plug into a systems model by way of influencing the system’s incentives, changing the resource mix, reshaping the rules and norms governing the system, or otherwise influencing actors’ tastes, preferences, and choices. This is where the co-production relationship described earlier becomes essential. Policy designers need to understand the ways in which each particular area of policy depends upon group or individual cooperation and action in order to generate outcomes.

Excellent practice in logic modelling, therefore, requires at least some use of systems modelling to map the context for policy intervention. At a minimum, good practice in logic modelling requires that practitioners keep the distinctions between intention and reality, and between the policy idea and its setting, as clear as possible. Confusing or ignoring these distinctions may produce a model that neither communicates the policy’s basic theory nor describes the policy setting adequately.

Where systems are concerned, model builders also should be in regular contact with the department’s operational staff, most of whom interact daily with programs, clients, and their real-life settings. Based on these interactions, operational staff are continually forming and revising their own, often unconscious, models of how various components of a program, service, or policy influence client behaviour, nudge economic trends, shape international relations, or alter patterns of environmental change. Head-office staff cannot begin to understand policy logic, policy systems, or policy complexity without regularly consulting the front lines and comparing what they find there with what the original policymakers promised to deliver.

Logic models are most useful when one has a particular program or policy in mind, but they can be used to generate policy ideas if one has a set of high-level desired outcomes and a working knowledge of the systems model. This approach is sometimes called top-down logic modelling because it begins by mapping the chain of outcomes that are known to contribute to a chosen outcome, rather than starting with a selected policy or program. The process of mapping these outcome chains closely resembles the systems modelling process (I sometimes call it ‘systems lite’) because the intermediate outcomes are simply variables that contribute to the final result. The process of producing the map helps participants organise what they know about the outcome and the conditions that contribute to or detract from it, and sometimes leads to creative ideas for interventions as the chains move down the page.

I have not seen very many departments using top-down logic modelling, and it may be that systems thinking is more appropriate in the kinds of blue-skies policy advice and strategic planning settings where this technique is most likely to be needed.

The frontier for logic practices – their potential contribution to both vertical and horizontal integration of public sector work – has yet to be explored. Can policy models be devised that allow policy designers, implementation planners, project managers, strategists, evaluators, and even ministers all to see at a glance how their separate activities fit together into a coherent whole? If so, what would these models look like? How much detail would they contain? Would they be expandable and contractible, depending upon the desired application? Can website-writing software help us think about the shapes and functions of these sorts of models? One can imagine an entire department’s website organised around the intervention logic for its core outputs. It would say: Click here for the strategic view and high-level outcomes, click here for the intermediate outcomes associated with output X, click here for the research projects designed to test assumptions of this causal link, click here for the plan to manage the risks associated with these intermediate outcomes, click here for a list of partners need to co-produce this outcome, and so on.