Directory UMM :Data Elmu:jurnal:S:Socio-Economic Planning Sciences:Vol34.Issue4.2000:

Applying fuzzy-set theory to performance evaluation

Salwa Ammar, Ronald Wright*

Department of Business Administration, Le Moyne College, Syracuse, NY 13214, USA

Abstract

Increasing emphasis on performance evaluation creates the need to develop consistent, fair, and robust models. The emerging methodology of fuzzy-set theory provides the tools necessary to address many of the issues relevant to performance assessment. This paper includes three applications of fuzzy-rule-based systems in performance measurement. All the examples require processing surveys and other forms of imprecise information. The evaluations rely on modeling the judgments experts make as they integrate multiple criteria. The paper includes a description of and motivation for the methodology as well as results of the proposed models. Comparisons with other evaluation practices are also included.72000 Elsevier Science Ltd. All rights reserved.

1. Introduction

Fuzzy-set theory has been used over the past 10 years in numerous scienti®c and engineering applications, primarily in control systems and pattern recognition [4]. Fuzzy sets gained popular attention in this decade as shown by an article in Newsweek [8] which reported that the Japanese held thousands of patents on fuzzy systems used to control everything from camcorders and small appliances to subway trains and ships. Application of fuzzy-set theory in the social sciences, however, has been a much slower process. More than a decade ago, Smithson [11] attempted to illustrate the basic concepts of fuzzy-set theory with examples from psychology, sociology, and political science. Yet, just recently, Treadwell [12], while making a case for the use of fuzzy-set theory in the social sciences, stated that `The dialogue between the human sciences and fuzzy-set theory has been scattered, unsystematic, and slow to develop'.

Socio-Economic Planning Sciences 34 (2000) 285±302

0038-0121/00/$ - see front matter72000 Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 8 - 0 1 2 1 ( 0 0 ) 0 0 0 0 4 - 5

www.elsevier.com//locate/dsw

An important area in which fuzzy-set theory has been successfully applied is performance evaluation. It is thus the goal of this paper to provide a description of the methodology and illustrations of its successful application within the context of performance evaluation.

The paper includes three examples of intelligent systems capable of evaluating and/or ranking performance based on expert judgement. The ®rst example describes a complete pilot study on the use of fuzzy logic to evaluate state government performance. The study is part of an ongoing e€ort to de®ne criteria for state management and produce relative rankings of management performance. This, the most recent of the three examples, provides a good background for presenting an overview of fuzzy-rule-based systems. The methodology will, therefore be presented in this context.

The second example is a fuzzy-rule-based model currently in full implementation. The model is used by the corporate information systems at a major northeast utility to evaluate client satisfaction surveys. Model results are used to evaluate performance of departments and to assess allocation of resources.

The ®nal example describes an initial investigation of measuring the economic impact of a technology development organization in New York. The organization, to justify state funding, is attempting to measure the e€ect of its services on tax revenues. The model is developed as a predictor of economic impact, measured as an increase in tax revenue, and is compared with other prediction methods.

All three examples rely on processing survey information. Evaluating survey responses often means interpreting imprecise measures of multiple criteria. The described methodology can accommodate imprecision in both quantitative and non-quantitative measures of performance. It can also incorporate judgment that integrates the di€erent measures by considering interactions among performance criteria. For each application, the paper includes a description of the performance-evaluation problem and a brief discussion of the results. It highlights the advantages of the proposed model vs. previous evaluation practices.

2. Application I: evaluating state governments

In 1996, the Campbell Institute, Maxwell School of Citizenship and Public A€airs of Syracuse University was awarded a 4-year grant from the Pew Charitable Trust to rate management performance of state and local governments and selected federal agencies. The government-performance project builds on a journalism e€ort to publish management ratings of the 50 States [3].

2.1. Why a multi-level fuzzy-rule-based system?

Fuzzy-set theory represents a generalization of classical (crisp) set theory. A fuzzy set is de®ned by a function that ranges between 0 and 1, which assigns the degrees of membership to each element in a set. Intuitively, the degree of membership represents the extent to which an expert's judgment places an element in a set. An element can belong to more than one set with di€erent degrees of membership. This allows a gradual transition among adjacent sets. The mathematical rigor of the theory and comparisons with traditional approaches have received much attention over the last 30 years [4,5,7]. The strength of the theory is in its ability to provide an alternative framework to modeling imprecision. Thus, it allows us to view concepts of possibility and vagueness separate from probabilistic or random uncertainty. The set-theoretic approach also allows the non-linear integration of di€erent domains.

Since measurements of performance are imprecise, any evaluation system needs to be robust and not sensitive to small changes. To achieve fair and consistent evaluation, transition from one category of evaluation to another needs to be gradual rather than abrupt. All these issues motivate the use of fuzzy-rule-based systems in performance evaluation. More speci®cally, in terms of the application at hand, a fuzzy-rule-based system was developed for the following reasons.

. The complexity of the evaluation process and the need to integrate rather than cumulate.

The two key evaluation criteria here are accuracy in budget forecasting and ®nancial structural balance. As we evaluate accuracy in expenditure and revenue forecasting, we might calculate the average forecasting errors. Also, as we evaluate the structural balance between ongoing expenditure and ongoing revenue, we might calculate the average budget surplus. If we were to evaluate each aspect independently and then linearly combine them, we would fail to consider the interaction between the two. We need to be able to evaluate a planned adequate (or even excessive) surplus di€erently from an unplanned one that resulted from poor forecasting. These kinds of interactions are best captured with a set of consistent rules that do not impose linearity.

. The imprecision of expert opinion.

The panel of experts consulted in this study all agreed on the concept of excessive surplus, but they disagreed on their assessments of the level of surplus. It is therefore necessary that the level be de®ned as a range and, more importantly, that there be a gradual transition so that there is no single value at which the surplus abruptly becomes excessive. Fuzzy-set theory provides the appropriate modeling tool here [13].

. The sensitive nature of the project and the requirement of a robust system.

Evaluation using fuzzy-rule-based systems allows for membership in more than one category. This, in turn, allows for multiple rule application, which means that any one conclusion can be derived in many di€erent ways. This ensures robustness in the system, in that small changes in input will not excessively in¯uence the overall results.

. The nature of surveys and the need to account for imprecision in the information.

Much of the survey deals with hard-to-quantify measures such as budget priority setting and planning. Crisp evaluation of these aspects cannot be expected even though many procedures for objectivity (described later) were followed. The fuzzy approach alleviates the need to produce the `exact right' score on each dimension of the evaluation process.

. The need to track and explain each of the state's rankings.

The objective here was the design of a ranking system that inherently encompasses the evaluation process. The multi-level fuzzy-rule-based system allows for tracking through each of the dimensions of evaluation, determining applicable rules, and providing a rationale for ranking at each level. This allowed us, in the end, to tell the ®nancial management `story' of each state.

2.2. The model

Prior to our e€ort to develop this evaluation model, the Campbell Institute convened a panel of practitioners and researchers to be subject-matter experts. Ingrahm et al. [6] have described the consensus that evolved from these sessions and the rationale for the resulting 13 evaluation criteria. The criteria are listed in Table 1. With further assistance from this expert panel, we used these given criteria to develop an evaluation model structure. Fig. 1 shows the complete model structure, which consists of the following three types of evaluation entities.

2.2.1. Numerical data (dashed rectangles in Fig. 1)

These are spreadsheet-calculated values that represent historical averages, budget percentages, forecasting errors, etc. Three ranges were de®ned for each of the numerical measures. The ranges were determined through the consensus of a group of experts in ®nancial management assembled by the Campbell Institute to aid in this pilot project. Fuzzy measures were then de®ned using these ranges. For example, a rainy-day fund was described, on a scale of percentage of expenditures, using three categories: low, moderate, or good. Similarly,

Table 1

Financial management evaluation criteria

1. Accurate revenue and expenditure forecasting

2. Structural balance between ongoing revenues and ongoing expenditures 3. Use of counter-cyclical or contingency planning devices such as rainy-day funds 4. Timely budget adoption

5. Investment and cash management provide appropriate balance between return investment and risk avoidance 6. Appropriate management of long-term debt to ensure solvency

7. Accurate and audited ®nancial statements

8. Engagement in cost accounting so that the entity is able to accurately gauge the cost of delivering programs and services

9. Appropriate balance between expenditure control and ¯exibility for managers in pursuit of established objectives 10. Clear, timely, accurate, and useful ®nancial reporting

11. Multi-year perspective on budgeting and the future ®scal impacts of state actions is considered 12. E€ective management of contracts for delivery of state goods and services

Fig. 1. Financial management evaluation model structure.% GO is the percentage of total debt in general obligation debt.

S.

Ammar,

R.

Wright

/

Socio-Economic

Planning

Sciences

34

(2000)

285±302

expenditure forecasting accuracy was described, on the scale of percentage deviation from actual, using three categories: overestimate, accurate, or underestimate. An example of fuzzy sets de®ning the rainy-day fund is shown in Fig. 2. The fuzzy presentation allows the overlap and gradual transition between categories. For example, a rainy-day fund of 3% is considered both moderate and good, to varying degrees. The degree of membership in each category is de®ned by the function (vertical axis) in Fig. 2. Note that a slightly higher or lower rainy-day fund would still be considered both moderate and good; only the membership degrees will vary minimally.

2.2.2. Scored values (lined rectangles in Fig. 1)

These are scores that represent the evaluation of the non-quantitative survey responses. In order to minimize the subjectivity of these evaluations, we de®ned detailed scoring guidelines with the assistance of a panel of experts. The guidelines represent attributes that the experts determined to be important in assessing the aspect of management performance represented in each survey question. The panel distributed 20 points over each set of guidelines based on relative importance. For example, in evaluating the investment process, the scoring guidelines were de®ned as shown in Table 2. Seven guidelines were stated, each with a maximum possible score of 2±4 points and a maximum total possible score for the investment process of 20 points. Given the prede®ned guidelines, the scoring was performed by a panel of three independent readers. Each reader, for example, was asked to look for evidence of a written investment policy and to assign up to 4 points (see Table 2) based on appropriateness of the policy and statutory mandates. A consensus score was determined for each survey response.

Much like the numerical data, the scores were then converted to fuzzy measures using three categories of poor, fair, and good. These measures varied for each scored entity depending of the distribution of scores and availability of information.

2.2.3. Rule bases (shaded rectangles in Fig. 1)

The model contains nine rule-based systems with two, three or four inputs. Rule de®nition is the most important part of the system design as it is where expert judgment is incorporated. The ®rst step was to de®ne a complete conclusion matrix by considering all possible input combinations and assigning a conclusion to each. An example of such a matrix for the structural-balance evaluation entity is given in Fig. 3. Structural balance is evaluated on four inputs: general-fund balance, rainy-day fund, surplus growth and e€ective surplus. Each of these inputs has three fuzzy categories, hence there are 34=81 possible combinations in this four-dimensional matrix. The cell values represent the judgment describing each combination, where `g' stands for good, `f` stands for fair, and `p' stands for poor. This stage of evaluation allows for judgment that fully integrates the four inputs by considering their interaction. For

Table 2

Scoring guidelines for investment process

Question 8 a, b, c: Investments process scoring guidelines

Guidelines Possible score Comments

Policy 4 Is the policy written, statutory, and appropriate?

Reporting 2 Is reporting frequent, timely, and relevant?

Benchmarking 2 Is benchmarking realistic, regularly tracked and updated?

Independent oversight 3 Does an independent body exist, and does it function appropriately?

Pooled 4 Are all cash funds in a single pool?

Portfolio by objective 3 Are objectives de®ned for liquidity, intermediate, and total return? External managers 2 Are there external managers primarily for intermediate or total return?

Maximum possible 20

Fig. 3. Structural balance rule de®ning matrix.

instance, we can express di€erent judgments on the level of the general-fund balance as we consider the di€erent levels for a rainy-day fund. We can also judge a negative surplus growth favorably in the presence of an excessive surplus and not otherwise. This form of intelligent judgment is necessary in any evaluation model. Any attempt to linearly combine these inputs falls short of considering these interactions and is thus more likely to produce anomalies.

Any one cell in the matrix can be expressed as a rule. For example, the circled cell in the matrix can be expressed as the following rule:

IF the general fund balance is high AND the rainy-day fund is low

AND the surplus growth is negative AND the e€ective surplus is adequate THEN the structural balance is `f' (fair).

The rule matrix was then utilized to describe the evaluation philosophy. This philosophy was discussed and revised by the group of experts consulted for each of the nine rule bases.

In the fuzzy-set-theoretic context, rules apply to varying degrees, depending on the degree of the rule condition. It is important to point out that there is also an inherent robustness in the fuzzy-rule base. Since any input value is likely to fall into more than one category with di€erent memberships, multiple adjacent rules will apply to varying degrees. This implies that small changes in the rule de®nition will not abruptly change the overall conclusion.

In this multi-level system, input to the rule base can be numerical data de®ned on a fuzzy measure, scored values also de®ned on a fuzzy measure, or the output of a lower-level rule base. The rules are applied using the fuzzy-theoretic-extension principle with a resulting fuzzy output [5]. Hence, at every rule base the evaluation of each state is expressed as poor, fair, or good with di€erent memberships. This output is then a natural input to the next rule-base level. Defuzzi®cation and ranking are performed at the ®nal rule base, ®nancial management, using a weighted center of gravity method [10]. A simple example of rule application and defuzzi®cation is given in the Appendix.

2.3. Results and comparison with the journalistic approach

The proposed model was implemented using Excel Visual Basic Application. The system produced a two-page summary report for each state. The reports enabled us to examine each aspect of the evaluation process as it applied to any given state. They also allowed us to summarize the ®nancial management `story' of each state.

Based on our experience here, we suggest that there are many advantages to the fuzzy-rule-based approach. First, there is the issue of consistency. All information and judgments used in the system applied equally to all states. There were no selective or anecdotal inputs. Secondly, our approach is easily repeatable. Beyond the initial investment in system development, the process can be duplicated with minimal resources. The dangers of the `black box' application can be avoided by recognizing that the system incorporates judgments and must be updated and revised periodically. The value is in the ease with which the revisions can be consistently incorporated.

3. Application II: client satisfaction

The corporate information systems (CIS) group of a major northeast utility has used internal client satisfaction surveys for many years, primarily as nebulous measures of satisfaction with the intent to identify areas for improvement. Recently, these evaluations took on new meaning. As the utility industry has become more competitive, pressure was put on units to demonstrate their explicit value and to justify their expenditures. This new competitive environment resulted in management bonuses and layo€s. When upper management began to use the satisfaction surveys as a tool for evaluating individual performance, the entire department developed a renewed interest in the validity of the current procedure for determining scores based on the survey results.

CIS sends the client satisfaction survey to all personnel within the corporation who have used CIS resources. The survey is composed of 35 questions distributed over six groups. Five of the groups deal with functional areas (application development, mainframe computer operation, telecommunications, personal computers and LANs, and the corporate I/S help desk) where the last covers general business conduct. Individual questions focus on services within each function, such as, `How satis®ed are you that corporate I/S reports are delivered on time?' For each question, respondents are asked to indicate both a level of satisfaction with the service and the importance of the service, using a 1±10 scale.

Management is interested in evaluating the level of satisfaction and the level of importance, as well as the relationship between the two. Clearly, it is impractical to provide sucient resources to generate maximal levels of satisfaction for all services. The performance index thus needs to measure the extent to which the most important services (from the users' perspective) receive the highest satisfaction ratings.

3.1. Why a fuzzy-rule-based system?

Many of the motivating factors present in the previous example also apply to this performance-evaluation problem. There are multiple dimensions of performance that must be integrated. Attempts to produce analytical equations that combine client satisfaction ratings with job importance ratings fell short of considering interactions between the two. The outcome contained many unexplainable anomalies and rendered the results unreliable.

There was also the question of transition between the evaluation categories and the arbitrary

interpretation of the overall rating. Why was a score of 8.01 very good and 7.99 only satisfactory?

These concerns about the fairness of the index led to a request for a review of the procedures and the proposal of a new methodology. The proposed fuzzy-rule-based system addresses many of these concerns [2].

3.2. The model

The proposed model was de®ned on three measures of performance (indices): satisfaction, importance, and the relationship between the two. For each index, four fuzzy categories were de®ned: excellent, good, fair, and poor. Membership values were represented by triangular functions, much as in the previous application.

A rule base that incorporates management judgement on performance was determined by completing a rule conclusion matrix. The conclusion was also de®ned using the same four categories. Since there are three inputs and four categories in each, the number of fuzzy rules required for this system was only 43 or 64. An example of one of the rules is:

IF the satisfaction rating is excellent AND the importance rating is good AND the relationship rating is good THEN the overall rating is excellent.

The fuzzy rules were applied for each question in the survey with results suggesting the membership level as excellent, good, fair, or poor. In the ®nal report, the membership levels were represented graphically, emphasizing the fuzzy nature of the rating. Upper-level management still desired a single ®nal number to be used for comparisons. This score was obtained by simply `defuzzifying' the results.

3.3. Results and comparison with averaging methods

Prior to this model, performance rating was determined for each question by ®rst calculating the absolute value of the di€erence between the satisfaction mean and the importance mean. The di€erence was intended as a measure of the extent to which the satisfaction levels matched the importance levels. Management felt that an exact match deserved a perfect score of 10 and that a di€erence of 1 (out of a possible 4) was a `C' and hence deserved a 7. Other scores were assigned linearly from these two points.

For the most part, this method, although a bit arbitrary, produced high scores for good performance and low scores for poor performance. The method, however, produced several anomalies. In some instances, excessively high scores were recorded for responses in which the satisfaction level and the importance level matched but at a low level. In other cases, people felt unfairly penalized by receiving a high level of satisfaction on services considered of less importance. In fact, lower levels of satisfaction would have produced higher scores.

be represented in the ®nal rating as independent factors in addition to the assessment of the relationship between the two. The relationship was thus rede®ned to more accurately measure the extent to which the most important services received the most attention. A preliminary model was developed that determined the ®nal rating as a weighted average of the importance index, the satisfaction index, and the relationship index.

Responses from a recent survey were analyzed using the original index, the weighted-average model, and the rule-based system. The original index continued to produce non-defendable results. Both the weighted averages and the fuzzy-rule-based system eliminated the anomalies that had caused some of the original concerns. However, the fuzzy-index results di€ered from the weighted averages in some signi®cant ways.

The most signi®cant di€erences of the largest magnitude were observed for services in which the satisfaction level was poor. These received, as they deserved to, poor scores using the fuzzy rules. The rule base treated importance as less relevant when the satisfaction was poor. The weighted averages, however, often raised the scores to acceptable fair levels because of the highly rated importance. There were many other examples where the judgment of management was non-linear. The fuzzy-rule-based system, unlike the weighted-average method, was able to incorporate such judgment.

In all cases, the fuzzy index tended to rate services in a manner consistent with the judgments made by the manager upon assessing all available information. Those being evaluated felt that the system was fair and that they fully understood how they will be judged prior to the administration and analysis of the survey.

The rules for this fuzzy system were relatively easy to create. There was a clear understanding by all involved as to what the conclusion should be for each rule. Unfortunately, this is not always the case. The ®nal application thus illustrates a case in which creating the rules was a major hurdle to building the system.

4. Application III: evaluating state-funded agencies

In New York, technology development organizations (TDOs), which provide support to manufacturers and other hi-tech businesses, must regularly show that they have suciently added value to the state economy. Indicators include the number of jobs created, increase in sales, and the increase in state tax revenue. Past e€orts have utilized surveys in which the client estimates the impact of the TDO activity. Clients, however, ®nd it dicult to determine the impact and, in many cases, complain that they are excessively surveyed. There are also few clear guidelines for extrapolating results to cover clients who were not surveyed or did not reply.

4.1. Why a fuzzy-rule-based system?

The Central New York TDO would prefer a system in which sta€ could collect basic information during the consulting process and combine that with the client's initial estimate of the impact of the TDO activity. Sta€ would conduct surveys concurrent with the consulting activity rather than solicit responses periodically. This would allow estimates of ®nancial

impact to be based on all clients and would not require any extrapolation. The TDO also wants the information to be based on quick estimates and not require extensive e€orts from clients. This is particularly important for emerging businesses that do not necessarily have ecient methods for processing corporate information.

The ®nancial impact would be estimated based on a combination of the imprecise but complete information about the company, the client's satisfaction with the consultations, and a judgment by TDO sta€ of the likely ®nancial impact on state revenue. A fuzzy-rule-based system would allow the modeling of ®nancial impact in a manner that accounts for the imprecision of the inputs and the subjective, expert judgment of the TDO sta€.

An example of a rule in such a system might be:

IF the size of ®rm is small AND the current income is minimal AND the type of company is technological AND the type of activity is access to funding AND the client assessment is substantial

THEN the increase in tax revenue is noticeable.

These rules would be created on the basis of expert knowledge obtained from the experience of previous years and then periodically checked using detailed surveys from a sample of the client population. However, producing a complete set of rules would here be a daunting task. In addition, there were concerns about the validity of a fuzzy economic model and whether it could be `sold' to the state government. As a result, an initial step was undertaken in which the results of a fuzzy-rule-based system were compared with those obtained through a study previously commissioned by the State of New York [9]. The next two sections describe the results of that exercise.

4.2. The model

The commissioned study estimated the extent to which TDO services contributed to the value added, from 1992 to 1994, for client companies of all TDOs receiving state funding. For purposes of the study, value added was de®ned as net income plus compensation. Both measures result in increases in state revenue through corporate and personal income taxes, while increased compensation also generates an increase in sales tax.

new multiple regression analysis was then performed using all 380 companies. The service intensity variable was statistically signi®cant, and the resulting coecient was used to predict the increase in value added for each unit of service intensity. This number was used to estimate the total increase in value added for all TDO clients (not just those surveyed).

The goal of the comparison of the regression and fuzzy models was limited to providing some con®rmation that a fuzzy-rule-based model, in this context, could produce results comparable with those already accepted by the state. The data from the 305 companies not receiving TDO services was not available so the comparison was limited to the remaining 75 companies. To provide a fair comparison, the fuzzy model was limited to using only the data represented in the regression model. Hence, the conditional statements in the rules corresponds to the four independent variables in the regression model. Fuzzy sets with triangular membership functions were de®ned for the four variables using three categories in each case. The required rules would thus take a form such as:

IF the increase in employees is large AND

the increase in computer keyboards is medium AND the value added per employee is small

THEN the increase in value added is large.

Normally, these rules would be created using expert judgment. However, the inputs used in the regression model were not those that our experts were likely to use to make their assessments. In addition, the inclusion of such judgments would add input not available in the regression model. A methodology was thus developed to create conclusions to the fuzzy rules [1] utilizing only the available survey information about the four independent variables and without taking advantage of any existing expert knowledge. The resulting model was a single-level rule system much like the client-satisfaction model.

4.3. Results and comparison with the regression model

The results of the two models were compared on the basis of the mean absolute deviation in the predicted increase in value added vs. the actual increase in value added. The two models produced comparable deviations with the fuzzy model showing a small (5.4%) improvement [1]. This indicates that a fuzzy-rule-based system can potentially produce quantitative results that are comparable with those produced by an accepted regression model. However, this simple model does not take advantage of any of the bene®ts we have ascribed to fuzzy systems. It thus uses none of the types of information that TDO personnel envision collecting, and includes none of the expertise they have collectively amassed over many years.

Based on the conclusions of this preliminary study, the Central New York TDO has decided to proceed with the development of a complete fuzzy-rule-based system. The next steps include identifying factors more predictive of the impact of TDO services than those used in the regression analysis, including both quantitative and qualitative inputs. Fuzzy rules and sets are being developed based on the combined experience of TDO sta€ throughout the state as well

as on data previously compiled from many client surveys. Additional work is also being done to enhance the process by which rules are created from a database.

It is expected that this new system not only will provide a more accurate estimate of the increase in value added for TDO clients, but could also be used to predict which clients are most likely to produce the largest increases and hence provide the best return on taxpayer dollars. Besides providing more accurate results, the fuzzy-rule-based system will be far more ¯exible, require less data, and be much less a€ected by the inherent (and often extreme) inaccuracies in the survey data.

5. Conclusion

The three applications described in this paper all rely on survey data but are di€erent in nature. The ®rst application is the most complete and multifaceted of the three. It shows that a complex evaluation process can be structured into several related smaller evaluations. The result is a multi-level rule-based system incorporating expert judgments. It is proposed as an alternative to the subjective journalistic approach to ranking state performances. The second application is a single-rule-based system also incorporating expert judgment. It is proposed as an alternative to an analytical model using a simple linear equation. The ®nal application is a rule-based system where the rules are based on data analysis. The system was developed as a non-linear predictor and is compared with a regression model. The next step in this application is to restructure the evaluation process and incorporate expert judgment on economic impact.

There are many advantages to using fuzzy-rule-based systems in performance evaluation. As we have demonstrated, such systems enable a thoughtful and comprehensive analysis of the evaluation approach. Once this analysis is conducted, the system can be applied fairly to all evaluated subjects. The process can integrate any level of expertise that exists for making judgments. The criteria and the manner in which they are used can be completely explained to those participating in the evaluation. In addition, by tracking the applicable rules and the degree to which they apply, those being evaluated can be informed of the qualities that had the most impact on their rating. Of course, the real value is that fuzzy sets appropriately address, and not simply ignore, the inherent lack of precision in the process. Also, once the initial e€ort has been put into developing a fuzzy-evaluation system, it can be used repeatedly with little additional e€ort and with con®dence that the results will be generally consistent. The dangers of the black-box application can be avoided by recognizing that the system incorporates judgments and needs to be updated and revised periodically. The value is in the ease with which the revisions can be consistently incorporated.

Appendix

A1. Rule application±using the extension principle

matrix shown in Fig. A1. Note that this could be the northwest corner of a four-dimensional matrix.

The nine cells can be combined into the following three rules:

Rule 1 IF Xis ANY AND Yis NOT good THEN the evaluation is `p' (poor) Rule 2 IF Xis NOT good AND Yis ANY

THEN the evaluation is `p' (poor) Rule 3 IF Xis good ANDY is good

THEN the evaluation is `f' (fair)

The degree to which a single rule applies is determined by the minimum membership in all of its conditions. The degree for each conclusion is de®ned to be the maximum degree to which all rules result in that conclusion. For example, consider evaluated entity A, with fuzzy measures onXand Y, as described in Table A1.

Table A1

Evaluated entityA

Membership in categories

Evaluated entityA Poor Fair Good

X 0.0 0.3 0.7

Y 0.0 0.4 0.6

Fig. A1. Combined rules.

By applying Rule 1, we get:

X Y Conclusion

Rule 1 ANY NOT good `p'

EntityA

Membership 1 0.4 0.4

Therefore, the level at which Rule 1 applies is the minimum membership inX and Y, namely 0.4. Note that ANY means that the measure is not relevant to this rule, and, hence, in order to be non-binding, it is assigned the highest membership of 1. Also, NOT is de®ned as a compliment; that is, 1 minus the membership in the set. In this case, the compliment is 1.0ÿ0.6.

Similarly for Rule 2, the level at which the rule applies is the minimum of 0.3 and 1.0.

X Y Conclusion

Rule 2 NOT good ANY `p'

EntityA

Membership 0.3 1 0.3

Following the extension principle, entity A is identi®ed as `p' (poor) with a level of 0.4, the maximum of the degrees to which the poor conclusion applied in Rule 1 and Rule 2.

Also, entity A is identi®ed, using Rule 3, as `f' (fair) with a level of 0.6.

X Y Conclusion

Rule 3 good good `f'

EntityA

Membership 0.7 0.6 0.6

After the complete rule application, each evaluated entity will have a membership in each of the conclusion categories. For this rule subset, entityAis identi®ed as:

poor fair good

with

membership of

0.4 0.6 0.0

A2. Ranking

in many heuristic approaches, appropriate methods can be chosen for speci®c applications. For purposes of this illustration, we will describe a common method for defuzzi®cation, based on calculating centers of gravity of the conclusions.

Each of the three categories is de®ned on an ordinal scale x (from, for example, 0±20) as fuzzy measures. In this method, the area given to each set is proportional to the weight of the corresponding category. An example of these sets is given in Fig. A2. The ranking is determined by calculating the center of gravity of the fuzzy conclusions using the following equation:

Center of Gravityˆ

…

xu…x†d_x=

… u…x†d_x

whereu(x) is the membership function.

As an example, compare entity A, described above, with entity Bwhere entity Bis identi®ed as:

Fig. A2. Conclusion membership functions.

poor fair good with a

membership of

0.6 0.3 0.2

The conclusion membership function for each entity is shown in Fig. A2.

When comparing the centers of gravity, entity A has a center of gravity equal to 7.47 while entityBhas a center of gravity equal to 7.6. Therefore, Branks higher than A.

References

[1] Ammar SH, Wright RH. Using fuzzy rule based economic models to measure the performance of state funded agencies. In: Proceedings of the Annual Meeting of the North American Fuzzy Information Processing Society. New York: Syracuse, 1997. p. 418±22.

[2] Ammar SH, Wright RH. Fuzzy logic: a case study in performance measurement. In: Ayyub BM, Gupta MM, editors. Uncertainty Analysis in Engineering and the Sciences. Boston: Kluwer, 1997. p. 233±45.

[3] Barrett K, Greene R. Grading the states. Governing February 1999;12(5):17±90.

[4] Bezdek JC. Fuzzy models Ð what are they, and why? IEEE Transactions on Fuzzy Systems 1993;1:1±5. [5] Dubois D, Prade H. Possibility Theory. New York: Plenum Press, 1988.

[6] Ingraham PW, Joyce PG. Kneedler AE Managing For Performance. Baltimore: Johns Hopkins University Press (forthcoming).

[7] Klir G, Yuan B. Fuzzy Sets and Fuzzy Logic: Theory and Application. Englewood Cli€s, NJ: Prentice Hall, 1995.

[8] Rogers M, Hoshiai Y. The Future looks fuzzy. Newsweek May 28, 1990;115(22):46±7.

[9] Nexus Associates. Evaluation of the New York Manufacturing Extension Partnership Report. Albany, New York: New York State Science and Technology Foundation, 1996.

[10] Saade JJ. A unifying approach to defuzzi®cation and comparison of outputs of fuzzy controllers. IEEE Transactions on Fuzzy Systems 1995;4:227±37.

[11] Smithson M. Fuzzy Set Analysis for Behavioral and Social Sciences. New York: Springer, 1987.