In this section we will present an analysis of 18 international benchmarks that were published in the period 2000-2003. We will look at the focus of the benchmarks, their definition of e-government, their comparison across time and space and the indicators that are used.
3.1. The focus of the benchmarks
The benchmarks could easily be divided into four groups with a clear difference in focus (this is shown in Table 1). A first cluster of benchmarks can be called the supply- oriented e-government measurements. Half of the benchmarks (1-8) inventoried fall into this group. The focus is on the supply of e-government applications, and the success of a country’s e-government is measured by counting the amount of visible applications. A second group of benchmarks (9-11) takes an opposite approach and evaluates the demand side of e-government. These are benchmarks that see e- government success in terms of actual levels of usage (‘take-up’ levels) or levels of customer satisfaction with online services. A third group of benchmarks (12-16) has the Information Society as a focus. These benchmarks evaluate countries, not only in terms of e-government efforts, but also according to their broader policy for (creating enabling conditions for) the Information Society and/or the Knowledge Economy. A fourth and final cluster (17-18) contains two meta-benchmarks in which criteria and indicators for the measurement of e-government are sought and proposed.
Although the segmentation of benchmarks into four groups provides for a first categorization, differences in country rankings cannot be solely attributed to differences in focus, for even within each category country rankings vary considerably.
Rankings should not necessarily be interpreted as ‘good-better-best’ qualifications. The aim of a benchmark is not always to see who has the ‘best’ e-government. Other rationales for e-government benchmarking include:
x Finding out if lessons can be learned from other country’s e-government policies;
x Measuring e-government progress compared to other countries;
x Identifying and learning from best practices in other countries;
x Discovering global trends in e-government;
x Measuring of underlying e-government concepts to identify points of leverage.
This diversity of underlying goals and focus has its effect on the approach and the outcome of each benchmark. One of the issues that has to be decided from the outset is the construction of a definition of e-government
3.2. The concept of e-government
A crucial step in the activity of measuring e-government is a clear demarcation of the concept itself. E-Government definitions abound, so it is crucial to realise that country rankings from different benchmarks are probably based on different definitions of what is being measured. It is not our intention to formulate yet another definition of e- government. We merely give a descriptive account of how e-government is defined in
K. Snijkers et al. / Critical Review of e-Government Benchmarking Studies 75
the benchmarks themselves. These definitions can be placed on a continuum with the following extremes: ‘service delivery on the internet’ (narrow definition) and ‘the use of ICT in the public sector’ (broad definition). Table 1 shows that the supply benchmarks indeed mostly define e-government quite narrowly as online service delivery. The demand benchmarks mostly take a somewhat broader perspective, often including aspects of e-participation. The Information Society benchmarks do not measure e-government in a narrow sense, but in some way look at the enabling environment for ICT, thereby often including e-government as one aspect of a broader scale of policy measures. The two e-government indicator benchmarks, finally, are looking at indicators to monitor broader aspects of ICT development.
Benchmarks that limit their view to online service delivery will obviously take into account different indicators than benchmarks that include aspects of process change, back-office development and e-participation. It seems that the inclination to limit the definition of e-government might partly be explained by the difficulties of gathering the information necessary for a broader conception of e-government.
The meagre attention for e-participation in the benchmark studies may also be the result of the narrow focus of most e-government initiatives on service delivery.
Citizens can be seen from different perspectives: as subordinated to the law, as consumers of public services, and as partners and co-producers of new policies [9]
[10]. Yet, most e-government projects seem to focus on the role of citizens as consumers of public services, and less on the possibilities of ICT to improve the interaction with citizens to develop new policies [11].
Concerning data gathering strategies, it has to be said that most of the benchmarks under consideration use existing, secondary, sources such as studies, national statistics, country reports, website analyses and so on. Again, it seems that –besides the choice of a definition of e-government– also the choice of e-government indicators is often based on the information sources that are easily available.
3.3. Comparing across time and space
Most of the benchmarks under consideration were commissioned out of a concern for comparisons, be it a comparison with one’s own position at a previous point in time or be it a comparison with another country or organisation. The benchmarks under consideration are mostly cross-country comparisons that also have a temporal character because of the recurrent (yearly or bi-yearly) replications of the research. Table 1 shows the variety in the number of countries included. Apart from the two meta- benchmarks and the two demand-benchmarks, all benchmarks compare countries, with a pool of countries varying from 6 to 196.
3.4. Indicators for the measurement of e-government
For each of the benchmarks under consideration an inventory of the indicators used
K. Snijkers et al. / Critical Review of e-Government Benchmarking Studies 76
Table 1. The concept of e-government
Focus of the benchmark
Benchmark Concept of
e-government
Number of countries compared (1) Accenture [12]. e-government leadership:
engaging the customer.
Online service delivery 22 (2) ASPA, UN division for Public Economics and
Public Administration [13]. Benchmarking E- government: A Global Perspective. Assessing the Progress of the UN Member States.
Online service delivery + infrastructure component + human development component
190
(3) Bertelsmann Foundation [14]. Balanced e-government.
Online service delivery + e-participation component
Separate cases from Canada, USA and Europe (4) Central IT unit UK [15]. Benchmarking electronic
service delivery.
Online service delivery + back-office (qualitative study, indicators)
13
(5) Cap Gemini Ernst & Young [16]. Online availability of public services: how does Europe progress?
Online service delivery 17
(6) Kable [17]. Europe’s readiness for e-government. e-gov = ‘when the public sector digitises its processes and interactions’
15
(7) The office of the e-envoy UK [18]. e-Government.
Benchmarking electronic service delivery.
Online service delivery + back-office (qualitative study, indicators)
13 Supply
(8) Worlds Markets Research Center [19]. Global e-government Survey.
Online service delivery 196 (9) Dialogic Innovation and Interaction [20]. E-
government: the demand side. An inventory of the wishes and expectations of citizens on electronic government.
Service delivery + participation
Not applicable, intra country demand (10) Dialogic Innovation and Interaction [21]. Let the
citizen speak. Judgments and complaints about electronic government.
Service delivery + participation
Not applicable, intra country demand Demand
(11) Taylor Nelson Sofres [22]. Government Online:
an international perspective.
Service delivery 31 (12) National Office for the Information Economy
[23]. The current state of play. Australia’s scorecard.
Infrastructure + usage 14
(13) Booz, Allen & Hamilton [24]. International e- Economy benchmarking. The world’s most effective policies for the e-Economy.
Use of ICT to advance social and/or economic development
9
(14) Harvard University [25]. The networked readiness of nations.
ICT environment 83 (15) Ministry of Interior Affairs (Netherlands) [26],
International ICT benchmark 2002.
ICT environment 6 Information
society
(16) The Economist Intelligence Unit [27]. The 2003 e- readiness rankings.
ICT environment 60 (17) Final report of the feasibility study for an ICT-
monitor in Flanders [28].
Indicators to measure ICT environment
Not applicable, metabenchmark e-Gov
indicators (18) Benchmarking Ireland in the Information Society [29].
Indicators to measure production and use of ICT
Not applicable, metabenchmark K. Snijkers et al. / Critical Review of e-Government Benchmarking Studies 77
was made. A global comparison of indicators led to the following categorisation of Indicators:
x Input indicators x Output indicators
x Usage/Intensity indicators x Impact/Effect indicators
x Environmental/Readiness indicators
Input indicators try to measure the resources countries have invested in e-government. Output indicators do not measure financial resources but instead measure the amount of e-government applications realised. Usage indicators do not measure the amount of applications but their actual usage by citizens/businesses.
Impact indicators then, try to measure the impact e-government has had, for example concerning changes in processing time or waiting time. Finally, environmental indicators try to assess the degree to which a country is ‘ready’ for the Information Society and its consequences.
3.4.1. Input indicators
Examples of input indicators
x Amount of financial resources devoted to e-government. Absolute figures, per capita figures.
x IT/e-government spending as % of GDP
x Amount of resources devoted to research and development x Amount of public resources devoted to internet infrastructure
In the benchmarks under consideration input indicators seldom get a lot of attention.
Most benchmarks limit themselves to a statistic of public IT spending, per capita or as a percentage of GDP. It is often not quite clear how these statistics emerge: how can one separate IT and e-government spending? How can one take into account countries where most IT spending is done in decentralized governments? How can one compare accounting systems of countries that deal differently with IT investments (or instead see them as costs)? One can only conclude that if a statistic is found, it is never entirely comparable with statistics found in other countries.
3.4.2. Output indicators
Examples of output indicators
x Number of online services for citizens x Number of online services for businesses
x Percentage of government departments that have a website x Percentage of government websites that offer electronic services
Benchmarks that make use of a broad set of output indicators are mostly those with a limited definition of e-government as online service delivery. The indicators used try to measure the online presence and complexity of services. Complexity is often measured with the categories information, interaction, transaction and integration. Electronic service delivery indeed is one of the most salient features of e-government, so the
K. Snijkers et al. / Critical Review of e-Government Benchmarking Studies 78
output indicators are in no way unimportant. There is a danger though that those governments that base their strategy on benchmarks that only include output indicators tend to forget that e-government is more than online service delivery. When governments try to score in those benchmarks they can often do so by ‘digitalizing’ as many existing services as possible, thereby neglecting the more fundamental process of redefining service delivery in an online environment: you might be better off with less but better services. Pro-active service delivery and so-called zero-stop government might be ingenious ways of approaching government in the information age; output indicators do not value them as they are used in the benchmarks under consideration. A country that has a nice website where citizens can apply for some document online gets higher scores than a country that has improved its back-office and was thereby able to abolish the document (and the need for citizens to apply for it).
3.4.3. Usage indicators
Examples of usage indicators
x Number of individuals that have made use of electronic services offered x Number of businesses that have made use of electronic services offered x Percentage of citizens that has visited government websites to search for
information
x Number of businesses that have made payments online
x Percentage of internet traffic that pertains to electronic service delivery
Usage indicators try to measure the actual usage or ‘take-up’ of electronic services offered. In more recent benchmarks, there seems to be an acceptance of the critique on output indicators. The main critique concerns the fact that countries get good grades for making lots of applications but that it does not matter if these applications are actually used by citizens. This is being corrected more and more by the use of usage indicators and by weighing them together with output indicators. This seems to make sense as the result is an evaluation of both the supply of and the demand for e-government in a country.
The usage indicators furthermore, provide for a good monitoring instrument for governments to evaluate the success of different applications and make corresponding strategy decisions. To arrive at a nuanced view of usage, there are often indicators for information seeking, information provision, and transactions.
3.4.4. Impact indicators
Examples of impact indicators
x Reduction of waiting time at government counter x by y %
x Decrease in case processing time at government organisation x by y % x Citizen/business satisfaction levels concerning e-government
x Survey-type questions, e.g.: ‘do you feel more positive to your government, now that you can contact it by email?’ ‘Has your government become more efficient, now that you can perform services online?’
The use of usage indicators described above already resulted in an overview of actual usage patterns per online service. Impact indicators go even further down the demand side and are used in benchmarks that measure end user satisfaction, but also in
K. Snijkers et al. / Critical Review of e-Government Benchmarking Studies 79
benchmarks that evaluate government organisation’s efforts. They try to establish some form of impact, be it citizens that are ‘happier’ or waiting lists that are shorter because of the introduction of e-government. Only a few benchmarks deal with these kinds of indicators. They are, of course, also the hardest to put into practice and require primary data gathering: interviews with citizens, overall evaluations of organisation’s efforts.
They do measure in a much more direct sense than usage indicators (which represent
‘consumer power’) the actual satisfaction of end users, or more generally the way that things have been improved because of e-government.
3.4.5. Environmental indicators
Examples of environmental indicators
x ICT penetration rates (pc, internet, mobile phone) private households, work, schools
x Indicator that measures ‘fear of invasion of privacy’
x Online shopping rates as an indicator of trust in online environments
x Indicator that measures ‘quality of legislation concerning the information society’
x Telephone tariffs, GSM tariffs, Internet access tariffs
The environmental, or ‘readiness’-indicators do not measure e-government as such, but instead measure some of the preconditions of a successful e-government. They are indicators of the e-society that is the surrounding environment of e-government, and mostly have to do with ICT infrastructure, ICT skills, trust in ICT and the legal environment. ICT infrastructure is one of the basic requirements of online government and can be measured by indicators such as internet penetration rates, broadband penetration, internet access tariffs, amount of public access points, and so on. ICT skills have to do with the way a country’s population is able to handle computers and ICT. A further categorisation here distinguishes ICT skills among citizens, businesses, and civil servants. Another indicator that is sometimes used here concerns the presence of scientific or academic institutions that excel in ICT knowledge.
A third group of environmental indicators indirectly measure trust in online environments by measuring the presence and success of e-business and e-commerce. A final group then, focuses on a country’s legal environment and assesses this in the light of the requirements of the information society, dealing for example with the juridical value of an e-mail and with the issues of online identification, online safety and online privacy.
4. Alternative approaches to e-government benchmarking: qualitative versus