Progress in relation to this objective was measured by RPI 1, the proportion of regulations for which the RIS documentation ‘adequately addressed net benefit to the community’. The assessment of the adequacy with which a regulation’s net benefit was calculated was the responsibility of the ORR, which then submitted it to the OSB for inclusion in its annual report on RPI. Over the period 1998-99 to 2004-05, the average annual percentage score for all departments and agencies for RPI 1 was 88% , with a range from 81% to 92% (see Table 3.2). It should be noted that the ORR increased the rigour with which it assessed performance over the period, as departments gained familiarity with the RPI, so that it is not possible to compare year to year performance with any degree of precision, although, where the annual performance increased year by year, despite the increased rigour of the assessments, then performance is likely to have increased.

The use of the phrase ‘adequately addressed net benefit’ in RPI 1, was a clear indication that its designers were well aware of the substantial difficulties involved in costing the likely or actual impact of a regulation. The phrase implies, for example, that cost benefit analysis will be the means for calculating net benefits but it does not specify or require that this must be the case, leaving considerable discretion in the hands of those submitting a proposed regulation to the RIS process. In addition, the use of the word ‘adequate’, rather than more specific criteria for the assessment of net benefit, enables a wide range of calculations of net benefit, more or less precise, to be judged as ‘adequate’. While this might have been realistic, it meant that the value of this measure was distinctly limited for those concerned to see if regulations were likely to achieve a net benefit to the community. It was of somewhat more use for indicating broad trends over time, especially combined with the increasing rigour of the ORR’s assessments. Unfortunately, as indicated in Table 3.2, performance as measured by RPI 2 fell somewhat over the period, suggesting that there was no significant improvement in the estimates of net benefit incorporated in the RIS assessed by the ORR.

Source: the OSB Annual Reports 1998-99 to 2004-05

In practice, the ORR in its publication A Guide to Regulation, did specify the type of costs and benefits to be included by departments in their RIS documentation. It noted that there were considerable difficulties in gaining precise, quantitative estimates of costs and benefits but encouraged departments to at least identify and, where possible, assess the fullest range of costs and benefits, although admitting that quantification was not always possible or necessary — although the onus of proof was on the department to defend the lack of quantification (the ORR 1998: D10).

Hence, it was quite possible that the ORR might judge the calculation of net benefit in an RIS as ‘adequate’, even where little or no quantitative data or assessment was included. If those relying on such performance data were aware of such limitations, it could be argued that at least it gave a rather crude, largely qualitative indicator of some value, although the lack of an upward trend in performance over the period was disappointing. However, even this limited value has to be questioned as, in practice, what was reported in the annual OSB report was only that proportion of a department’s regulations judged as ‘adequate’ in the year in question, in the form of a simple percentage figure, for example, 75%, with very little further explanation or clarification. Further, given that in any one year a department might have submitted only one or two RIS, then the value of providing a percentage indicator as a measure of performance was of questionable value for senior decision-makers and, of course, for Parliament, business and the community.

Objective 2 was measured by two indicators, RPI 2 — the proportion of regulations for which the RIS adequately justified the compliance burden on business — and RPI 3 — the proportion of regulations which provide businesses and stakeholders with some appropriate flexibility to determine the most cost effective means of achieving regulatory objectives — both to be monitored by the ORR. Over the period 1998-99 to 2004-05, the average annual percentage score for all departments and agencies for RPI 2 was 90%, with a range of 81% to 94%, and for RPI 3 was 75%, with a very large range of 39% to 100%. While the average of 90% for RPI 2 might have been reassuring, in the three years previous there was an increased variation with regard to the annual performance, to a low of 81% in 2004-05, which was of concern. In contrast, performance in relation to RPI 3 increased dramatically over the period, even with the increase in the rigour of the ORR’s assessments. Unfortunately the OSB annual reports provide no indication as to why performance improved (or fell), for any of the indicators.

If, as indicated above, it was difficult to provide accurate estimates of the costs and benefits of proposed regulation, then RPI 2 at least required departments to provide clear, logical arguments to support their proposals and (perhaps) to think through the compliance implications for business. If backed up by accurate, quantitative estimates of compliance costs and benefits, so much the better. In practice, its value, as with RPI 1, was limited by the lack of specific criteria against which adequacy would be judged (although the ORR did provide advice to departments in this regard) and the lack of quantitative estimates provided in practice by departments. As the ORR’s annual reports very clearly indicate, departmental estimates of regulatory costs and benefits were often unsatisfactory. Indeed, the Chairman of the Productivity Commission (of which the ORR is a unit), Gary Banks, indicated that in 2004 only 20% of tabled RISs contained even an attempt at quantifying the costs related to proposed regulations (Banks 2005: 10). If Banks was correct in his estimate, then it is difficult to see how, for example, 82% of all regulations assessed in 2004-05 could be assessed as adequate in relation to RPI 1 and 81% in relation to RPI 2 despite the fact that 20% of all RIS did not even attempt to quantify the costs related to the proposed regulations (the OSB 2006:38). It suggests a very flexible interpretation as to the meaning of adequacy and, importantly, makes the value of both RPI very questionable as reliable measures of regulatory performance.

RPI 3 was based on whether or not a department’s RIS contained any one or more of the following measures:

While the intent of RPI 3 is clear — being a measure of the degree of flexibility that a proposed regulation allows businesses in their compliance with regulation — the assumption it contains of the attractiveness of regulatory flexibility to business is problematical. A small business, for example, has limited resources of both money and time. Hence, the use of those resources to determine how it should comply with a regulatory performance or outcome based standard, rather than simply complying in a way prescribed in a regulation, is not necessarily attractive. At the least, it requires the business to design an appropriate strategy, or to purchase a ready-made strategy, or to employ a consultant to design the strategy. Faced with such choices, how many small businesses would not welcome a helpful regulatory prescription as to the required strategy, assuming the prescription is cost effective and meets compliance needs?

In summary, RPI 2 and 3 provided a rather mixed message for decision-makers, suggesting that regulations were providing an increase in the flexibility with which businesses could comply with regulations but, worryingly, a decline in the extent to which departments provided an adequate justification for the regulations in question. When the trend for RPI 2 is combined with the downward trend in RPI 1 — performance related to the calculation of net benefits — it suggests, if rather speculatively, that the efforts of the ORR and the OSB to encourage departments to improve regulatory quality in these important dimensions was unsuccessful, despite their increased efforts owing to a boost to their resources following the Bell Report.

This was to be measured by RPI 4 — the proportion of cases in which external review of decisions (as defined) led to a decision being reversed or overturned — and RPI 5 — the proportion of regulatory agencies whose mechanisms for internal review of decisions meet standards for complaints handling outlined in ‘Principles for Developing a Service Charter’ published by the Department of Finance and Administration. Departments and agencies were to provide the relevant information to the OSB for inclusion in its annual report. Over the period 1998-999 to 2004-05, the average annual percentage score for all departments and agencies for RPI 4 was 59%, with a range from 31% to 70%, but a declining trend from 2002-03. This was a disturbingly poor level of performance. The average annual score for RPI 5 was 68%, with a range from 50% to 77%, with a slight upwards trend in the latter three years, following an initial and very sharp fall in performance. In the case of RPI 5, where departments had only to meet the standards for complaints handling quite clearly specified by the Department of Finance and Administration, the average score of 68% was surprisingly low.

Unlike RPI 1 to 3, RPI 4 focused attention on the performance of departments in administering regulations. In particular, departments were assessed on their adherence to due process rather than administrative efficiency. The assumption was that if the proportion of cases where a decision was reversed or overturned by an external body was low, then the quality of the decision process, at least with regard to due process, was relatively high. While there is no doubt that RPI 4 might provide a valuable, more objective indicator of due process performance, two limitations need to be considered: one, limitations as to the sources of external decisions to be considered and reported upon; two, small business resource limitations.

With regard to the sources of external decisions to be included, the formal advice to departments was that they should only consider decisions of external review agencies that were empowered to overturn or reverse the department’s decision (Department of Industry Tourism and Resources 2006: 7-8). This meant that, for example, complaints from businesses to the Commonwealth Ombudsman would not be considered, even if the Ombudsman supported the complaint, as the Ombudsman had no power to overturn or reverse decisions. In addition, decisions resulting from departments’ internal review processes were excluded, even if they supported the complainant. In this latter case, while a degree of concern as to the objectivity of reviews from internal review sources is not surprising, it is surprising that they were not to be considered, even, for example, where they might have led to the overturning of an earlier decision. If they had been included as a source of performance data, this might have encouraged departments to adopt internal review processes where they did not exist and, where they did exist and indicated poor levels of performance, to improve their procedures and related decisions, especially if such information was made public in the RPI reports.

The second limitation springs from the limited resources small businesses have to take complaints about regulation and its implementation to external review bodies such as the courts or the Administrative Review Tribunal. Faced with this reality, the decision to exclude from consideration those decisions overturned within the potentially less costly and less formal internal review processes of departments was unduly restrictive. In part, the impact of this restriction might have been mitigated by RPI 5, the proportion of regulatory agencies whose mechanisms for internal review of decisions met standards for complaints handling outlined in the Principles for Developing a Service Charter. Some 12 criteria had to be satisfied in order to meet these standards, as determined by the Department of Finance and Administration (Department of Industry Tourism and Resources 2006: 8-9). The assumption was that if a department’s processes satisfied these criteria then it was likely that its internal processes were at least adequate insofar as complainants were concerned. Thus, departments were required only to answer ‘yes’ or ‘no’ with regard to RPI 5 and to provide a brief description of their internal review processes in their first report but not thereafter. It hardly needs pointing out that having sound processes does not guarantee good decisions and, given that the average annual score for all departments for the whole period for RPI 5 was only 68% (on a self-declared basis) then there is room for legitimate concern as to the adequacy of departmental performance in ensuring transparency or fair outcomes — the very objective measured by RPI 4 and 5.

This was to be measured by RPI 6 — the proportion of regulatory agencies having communications strategies for regulation, or formal consultative channels for communicating information about regulation. Over the period 1998-99 to 2004-05 the average annual percentage score for all departments and agencies for RPI 6 was 95%, with an initially impressive performance falling away somewhat after 2001-02.

Departments and agencies provided: a simple ‘yes’ or ‘no’ answer to the question of whether they had communication strategies or formal consultative channels; a brief description of the strategy or channels in question; and, where the department had a formal, written strategy, a copy of that document was submitted to the OSB. Where such strategies existed but had not been in place for the full reporting period, departments were advised to answer ‘yes’ to the question (Department of Industry Tourism and Resources 2006: 11). In practice, there seems to have been no little or no checking by either the ORR or the OSB as to whether the strategies or consultative channels actually enabled businesses to access and understand information on regulations other than as demonstrated in relation to RPI 8 and 9, below. In other words the value of RPI 6 was limited, focusing only on the achievement of procedural targets rather than any assessment of whether or not businesses found the consultative channels accessible or the information provided about regulations understandable. However, assuming that adequate processes might encourage accessibility and understanding, then an average performance level of 95% for the period was reassuring, even with the slight downward trend after 2001-02.

This was to be measured by RPI 7, the proportion of regulatory agencies publishing an adequate forward plan for introduction and review of regulation. Departments and agencies were to provide the relevant information to the OSB for inclusion in its annual report. Over the period 1998-99 to 2004-05, the average annual percentage score for all departments and agencies for RPI 7 was only 60%, with a range from 27% to 78%, although some reassurance could be gained from the consistently upward trend in performance after 2001-02. In other words, assuming that this RPI did provide a reliable indicator as to the ‘surety of future environment’, there was a substantial lack of surety for those businesses affected by the 40% of regulatory agencies that did not publish an adequate forward plan in the period under consideration. In addition, the extent to which such a predictable regulatory environment can be anticipated on the basis of a business knowing that a regulatory review is planned, is questionable, given that the planned reviews could result in substantial change, with at least short term uncertainty.

This was to be measured by RPI 8, the proportion of regulations for which the RIS documentation included an adequate statement of consultation (to be monitored and assessed by the ORR) and RPI 9, the proportion of regulatory agencies with organisational guidelines outlining consultation processes, procedures and standards, with departments and agencies to provide the relevant information to the OSB for inclusion in its annual report. In part, RPI 8 and 9 overlap with RPI 4, above, providing additional information as to the adequacy of consultation processes, based on the assessment by the ORR of the documentation provided in the RIS process. Over the period 1998-99 to 2004-05, the average annual percentage score for all departments and agencies for RPI 8 was 91%, with a range from 82% to 96%, but a worrying, downward trend from 2001-02, to a figure of 82% in 2004-05. The average annual score for RPI 9 was only 62%, with a range from 48% to 74% and a downward trend after 2001-02. Again, RPI 8 and 9 provide performance information only in relation to departmental processes, not their actual impact on business, that is, whether they actually result in accessibility and responsiveness. Hence, given that the average level of process performance for RPI 9 was only 62%, one might speculate that the actual accessibility and responsiveness of consultation processes used by departments was of a rather low order, both for business and the community.