Lievonen Observations from Even Swaps

(1)

S

ystems

Analysis Laboratory Helsinki University of Technology

Observations from

computer-supported Even Swaps experiments

using the Smart-Swaps software

Jyri Mustajoki

Raimo P. Hämäläinen

Petri Lievonen

Systems Analysis Laboratory

Helsinki University of Technology

(2)

S

ystems

Analysis Laboratory Helsinki University of Technology

Introduction

Even Swaps method

• Hammond, Keeney and Raiffa (1998, 1999)

• Easy-to-use multi-criteria decision analytical (MCDA) method based on value tradeoffs

Web-based Smart-Swaps software

• Procedural support to carry out even swaps

In this study, observations of students using the

method with the help of the software

• Do they like and understand it? • How laborious it is felt to be?

(3)

S

ystems

Analysis Laboratory Helsinki University of Technology

Structure of this presentation

Introduction to the Even Swaps process and the Smart-Swaps software

Observations from computer-supported Even Swaps experiments:

Research questions and experimental procedure

Results and evidence

(4)

S

ystems

Analysis Laboratory Helsinki University of Technology

PrOACT-model

Smart Choices (1999)

Define your decision problem to solve the right problem.

Clarify what you’re really trying to achieve with your decision.

Make smarter choices by creating better alternatives to choose from.

Describe how well each alternative meets your objectives.

Make tough compromises when you can’t achieve all your objectives at once.

Problem

Objectives

Alternatives

Consequences

Tradeoffs

Smart-Swaps software

-this experiment

Introduction to the Even Swaps process and the Smart-Swaps software

(5)

S

ystems

Analysis Laboratory Helsinki University of Technology

Even Swaps elimination process

Carry out even swaps that make

a) Alternatives dominated (attribute-wise)

≡ There is another alternative, which is equal or better than this in every attribute, and better at least in one attribute

b) Attributes irrelevant

≡ Each alternative has the same value on this attribute

» These can be eliminated

Process continues until one alternative, i.e. the

best one, remains

(6)

S

ystems

Analysis Laboratory Helsinki University of Technology

Smart-Swaps software

www.smart-swaps.hut.fi

(7)

S

ystems

Analysis Laboratory Helsinki University of Technology

Example

Office selection problem

(Hammond et al. 1999)

Dominated by

Lombard Practically

dominated by

Montana

(Slightly better in Monthly Cost, but equal or worse in all other attributes)

78 25

An even swap Commute time removed as irrelevant

(8)

S

ystems

Analysis Laboratory Helsinki University of Technology

Supporting Even Swaps with

Preference Programming

Support for

1. Finding candidates for the next even swap 2. Identifying practically (i.e. almost) dominated

alternatives

Both tasks need comprehensive technical screening

Idea: supporting the process – not automating it

(9)

S

ystems

Analysis Laboratory Helsinki University of Technology

Decision support in Smart-Swaps

More than one remaining alternative

The most preferred alternative is found

Trade-off information Even swap suggestions Practical dominance candidates Initial statements about the attributes

Problem initialization

Updating of

the model

Make an even swap Eliminate irrelevant

attributes

Eliminate dominated alternatives

Yes No

Even Swaps _ProgrammingPreference

(10)

S

ystems

Analysis Laboratory Helsinki University of Technology

Previous studies of Even Swaps

Comparison of Even Swaps and MAVT

• Belton et al. (2005): “MCDA in E-democracy. Why weight? Comparing Even Swaps and MAVT.”, TED Workshop, May 19-22, Helsinki

Environmental planning

• Gregory et al. (2001): “Bringing stakeholder values into environmental policy choices: a community-based

estuary case study.” Ecological Economics 39, 37-52.

Strategy selection in a rural enterprise

• Kajanus et al. (2001): “Application of even swaps for strategy selection in a rural enterprise.” Management Decision 39(5), 394-402.

(11)

S

ystems

Analysis Laboratory Helsinki University of Technology

Thoughts about Even Swaps process

From a cognitive point of view, Even Swaps

process has several characteristics that are of a

special interest, e.g.

• What kind of swaps the DMs tend to carry out?

• How the DMs understand the alternatives with the revised consequences?

• Does the DM end up with the same result following different paths of even swaps?

Smart-Swaps software available

» Easy to study how the DMs carry out the process in practice

(12)

S

ystems

Analysis Laboratory Helsinki University of Technology

Questions of interest in our experiment

In this study, the focus is on:

1) Issues related to carrying out the process with the Smart-Swaps software or (manually) with

Microsoft Excel?

2) Issues related to the size of the problem?

3) What are the benefits of using the Preference Programming functionality of Smart-Swaps software, if any?

(13)

S

ystems

Analysis Laboratory Helsinki University of Technology

Material and methods

• Subjects consisting of engineering students

as DMs

• Controlled experiments included in courses

1) Assignments in applied mathematics 2) Decision making and problem solving

• Questionnaires with open and scaled

questions

(14)

S

ystems

Analysis Laboratory Helsinki University of Technology

Experimental procedure

After a brief introduction to the Even Swaps process each DM carried out two decision analytical assignments:

1) The Even Swaps process on a small introductory problem.

On this assignment,

A. half of the DMs used Excel manually

B. the other half conducted the process with the Smart-Swaps software

2) The Even Swaps process on a much larger problem. Every DM used the Smart-Swaps software but

A. half of the DMs were instructed to ignore the Preference Programming functionality (‘recommender’)

(15)

S

ystems

Analysis Laboratory Helsinki University of Technology

1

st

: a small introductory problem

You are choosing an apartment. After preliminary

elimination there are three alternatives: Lombard, Baranov and Montana. You are in a happy position, as all want you as a tenant. It all boils down to what do you want. You

have ended up with four criteria to base your decision on. You gather up a consequences table:

Lombard Baranov Montana

Distance to

school 25 min 20 min 45 min

Condition Poor Good Excellent

Size 35 m2 42 m2 23 m2

(16)

S

ystems

2

nd

: a large primary problem

Your company is about to choose the facility for a new office. After serious thinking you consider eight attributes relevant:

a1: size of the office a2: rental costs

a3: renovation need

a4: car park opportunities a5: means of commuting a6: distance to city center

a7: other facilities in the neighborhood a8: habitability

(17)

S

ystems

Analysis Laboratory Helsinki University of Technology

1

st

study

2

nd

study

both

1. Small

problem

A (Excel)

10 DMs 6 DMs 16 pers.

B (Smart-Swaps)

2. Large

problem

A (without recomm.)

B (with recommender)

Overall sample sizes

(18)

S

ystems

Analysis Laboratory Helsinki University of Technology

Hypotheses on the experiment

H1.1: The DMs using the Smart-Swaps software evaluate the results more positively than the DMs using Excel. H1.2: The DMs using the Smart-Swaps require less time to

get the result than the DMs using Excel.

H2.1: The support by Smart-Swaps is evaluated to be more useful in large problems than in small ones.

H2.2: The DMs tend to eliminate rather dominated alternatives than irrelevant attributes.

H3.1: The even swap recommender is evaluated as useful. H3.2: Utilizing the recommender results in fewer swaps. H3.3: Practical dominance propositions are seldom

neglected.

(19)

S

ystems

Analysis Laboratory Helsinki University of Technology

DMs arrived to various decisions

Decisions in the 1st (small) and 2nd (large) problems. Areas indicate

frequencies of subjects that chose certain alternative.

1 2 7 2 9 1 3 5 1

X0 X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12

All DMs (N=31, NA=9)

5 6 29 All DMs (N=40)

Montana Baranov

(20)

S

ystems

H1: Smart-Swaps software

vs.

Excel

?

• The DMs using Smart-Swaps were faster

• No significant difference between no. of swaps

• Note that the problem was small

• No significant difference between decisions

made

(21)

S

ystems

Analysis Laboratory Helsinki University of Technology

The DMs using Smart-Swaps were faster

Decision time comparison in the 1st_{(small) problem. Areas indicate}

frequencies of subjects that completed the task in certain time.

Smart-Swaps software vs. Excel:

4 1 1 2 7 2 5 1 1 7 3 1 1 1 1 1 1

1 3 1 4 2 1 1 1 1 1

3 1 1 2 4 2 5 1 3 1 1

0 min 5 min 10 min 15 min 20 min 25 min 30 min 35 min

Median

All DMs (N=40)

DMs using MS Excel (N=16)

(22)

S

ystems

No significant difference between no. of swaps

Number of swaps on the 1st_{problem. Areas indicate frequencies of}

subjects that completed the task with certain amount of swaps.

Smart-Swaps software vs. Excel:

1 1 1 14 12 1 8 1 1

4 7 1 2 1 1

1 1 1 10 5 6

0 swaps 1 swaps 2 swaps 3 swaps 4 swaps 5 swaps 6 swaps 7 swaps 8 swaps 9 swaps

Median

All DMs (N=40)

DMs using MS Excel (N=16)

(23)

S

ystems

H2:

Issues related to the

size of the problem

• In large problems, it may be very difficult

to manually carry out the even swaps

process

• Smart-Swaps is regarded as helpful in

applying the Even Swaps process also in

large problems

• DMs tend to eliminate more dominated

alternatives than irrelevant attributes

(24)

S

ystems

Analysis Laboratory Helsinki University of Technology

The Even Swaps process is seen to apply

best to small problems

Areas indicate frequencies of subjects that gave certain answer on Osgood-scale.

Size of the problem:

2 5 4 3 2

6 6 1 1 1 1

2 1 3 3 6

3 1 2 2 4 4

0 1 2 3 4 5 6 7 Median

The apartment selection problem could be solved applying Even Swaps process with pen and paper (weakly-strongly) (N=16, N/A=4)

The office selection problem could be solved applying Even Swaps process with pen and paper (weakly-strongly) (N=16, N/A=4)

Smart-Swaps helps applying Even Swaps process to small problems (e.g. apartment selection problem) (weakly-strongly) (N=15, N/A=5)

(25)

S

ystems

Analysis Laboratory Helsinki University of Technology

Elimination by dominance was used

more than irrelevance

Number of all eliminations in the 2nd_{problem studies are presented in the table.}

Size of the problem:

No proposer With proposer

ALL ALL-% A A-% B B-%

Removal 66 11% 53 21% 13 4%

Dominance 345 59% 130 52% 215 64%

Irrelevance 163 28% 58 23% 105 31%

Practical dom. 11 2% 7 3% 4 1%

(26)

S

ystems

Analysis Laboratory Helsinki University of Technology

H3: Benefits of using Smart-Swaps

• The DMs using the Preference

Programming functionality (the

recommender) were faster

• The DMs using the recommender made

fewer swaps

• The recommender was rated high

• The DMs using the recommender gave

more positive opinions on the method

(27)

S

ystems

Analysis Laboratory Helsinki University of Technology

DMs using the proposer were faster

Decision time comparison in the 2nd_{(large) problem. Areas indicate}

frequencies of subjects that completed the task in certain time.

Benefits of Smart-Swaps:

2 1 1 3 6 1 3 1 1 1 3 2 1 2 1 1

2 1 1 1 2 1 1 1

2 1 1 3 4 2 1 1 3 1 1

0 min 5 min 10 min 15 min 20 min 25 min 30 min 35 min 40 min 45 min _Median

All DMs (N=30, N/A=10)

DMs without proposer (N=10, N/A=9)

(28)

S

ystems

DMs using the proposer made fewer swaps

Number of swaps on the 2nd_{problem. Areas indicate frequencies of subjects}

that completed the task with certain amount of swaps.

Benefits of Smart-Swaps:

2 1 1 4 3 4 2 1 1 1 1 1 2 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1

2 1 1 3 3 4 2 1 1 1 1

0 swaps 10 swaps 20 swaps 30 swaps 40 swaps 50 swaps 60 swaps

Median

All DMs (N=31, N/A=9)

DMs without proposer (N=11, N/A=8)

(29)

S

ystems

Preference programming functionality was rated high

Benefits of Smart-Swaps:

1 4 7 9

2 4 8 3 2

1 1 3 5

2 2 7

2 4 5

3 4 3

1 1 1 1 6

0 1 2 3 4 5 6 7

Median

I regard proposer as (cumbersome-convenient) (N=21)

I regard proposer as (unreliable-reliable) (N=19, N/A=2)

I regard proposer as (difficult-easy) (N=10)

I used the proposer (not at all - much) (N=11)

I did the swap the proposer suggested (never - always) (N=11)

I used proposer more for (irrelevance-dominance) (N=10, N/A=1)

I accepted practical dominance

suggestions (never - always) (N=10, N/A=1)

Opinions on the recommender functionality of the Smart-Swaps software.

(30)

S

ystems

DMs without proposer vs. DMs using proposer

Benefits of Smart-Swaps:

Differences in opinions between DMs ignoring the recommender and DMs utilizing it. Areas indicate frequencies of subjects that gave certain answer on Osgood-scale.

2 5 3 4 12 7 4

2 5 2 1 5 2

1 3 7 5 4

2 4 3 4 11 7 8

2 3 2 2 6 3

1 1 2 5 4 8

0 1 2 3 4 5 6 7

Median

The method is suitable for problems that have a large number of attributes - all opinions (weakly-strongly) (N=37, N/A=3) Opinions of DMs ignoring the proposer (N=17, N/A=2)

Opinions of DMs utilizing the proposer (N=20, N/A=1)

I regard the method as (easy-difficult) to use - all opinions (N=39, N/A=1)

Opinions of DMs ignoring the proposer (N=18, N/A=1)

(31)

S

ystems

DMs without proposer vs. DMs using proposer

Benefits of Smart-Swaps:

1 1 3 8 1 4

1 1 3 3

5 1 4

2 1 6 7 12 9 1

2 1 4 4 5 2

2 3 7 7 1

2 5 9 10 10 1 1

2 2 7 3 3

3 2 7 7 1 1

0 1 2 3 4 5 6 7 Median

Software helps justifying decisions to others - all opinions (weakly-strongly) (N=18, N/A=2)

Opinions of DMs utilizing the proposer (N=10)

Method improves my decision making - all opinions (weakly-strongly) (N=38, N/A=2)

Opinions of DMs utilizing the proposer (N=20, N/A=1)

I trust the method (mildly-strongly) - all opinions (N=38, N/A=2)

(32)

S

ystems

Conclusions

Our first experiments on the Smart-Swaps

software suggest:

• The support provided by the Smart-Swaps software is perceived to be useful

• Preference Programming approach provided considerable help

• A significant difference between the opinions of the group utilizing the recommender and the group instructed to

ignore it

» Further studies needed to get a deeper insight of how the DMs perceive the approach in practice

» In addition, we got a lot of usability feedback to help improve the Smart-Swaps software

(33)

S

ystems

Analysis Laboratory Helsinki University of Technology

References

V. Belton, G. Wright and G. Montibeller (2005). “MCDA in E-democracy. Why weight? Comparing Even Swaps and MAVT.” Presentation at the TED Workshop on

e-Participation in Environmental Decision Making, May 2005, Helsinki. (Downloadable at http://www.ted.tkk.fi/presentations/Belton-TED.ppt)

Hammond, J.S., Keeney, R.L., Raiffa, H., 1998. “Even swaps: A rational method for making trade-offs.” Harvard Business Review 76(2), 137-149.

Hammond, J.S., Keeney, R.L., Raiffa, H., 1999. Smart choices. A practical guide to making better decisions. Harvard Business School Press, Boston.

Mustajoki, J., Hämäläinen, R.P., 2005. “A Preference Programming Approach to Make the Even Swaps Method Even Easier.” Decision Analysis. (to appear) (Downloadable at www.sal.hut.fi/Publications/pdf-files/mmus04.pdf)

Salo, A., Hämäläinen, R.P., 1992. “Preference assessment by imprecise ratio statements.”

Operations Research 40(6), 1053-1061.

A. Salo and R.P. Hämäläinen (1995). ”Preference programming through approximate ratio comparisons.” European Journal of Operational Research 82(3), 458-475.

Applications of Even Swaps:

Gregory, R., Wellman, K., 2001. “Bringing stakeholder values into environmental policy choices: a community-based estuary case study.” Ecological Economics 39, 37-52. Kajanus, M., Ahola, J., Kurttila, M., Pesonen, M., 2001. “Application of even swaps for