She Ji. The Journal of Design Economics

(1)

V

o

lu

m

e

2

, I

s

u

e

2

S

u

m

e

r

2

0

16

(2)

Editorial Scope

She Ji is a peer-reviewed, trans-disciplinary design journal with a focus on economics and innovation, design process and design thinking. The journal invites papers that enrich the understanding and practice that enable design innovation in industry, business, non-profit services, and government through economic and social value creation. These papers may explore how design thinking can inform wider social, managerial, and intellectual discourses with an added focus on strategy and management. She Ji also publishes articles in research methods and methodology, philosophy, and philosophy of science to support the core journal area.

She Ji is fully open access. Tongji University and Tongji University Press support She Ji as a contribution to the design field and a public service to design research. Authors are not liable for any publication charges and all published articles are accessible free of charge from the journal web site.

A CC user license which manages the reuse of the article (see http://www. elsevier.com/about/open-access/open-access-policies/ oa-license-policy). All articles will be published under the CC BY-NC-ND license: Creative Commons Attribution-NonCommercial-NoDerivs (CC BY-NC-ND): for non-commercial purposes, lets others distribute and copy the article, and to include in a collective work (such as an anthology), as long as they credit the author(s) and provided they do not alter or modify the article.

Author Enquires For enquiries relating to the submission of articles please visit this journal’s homepage at URL:

http://www.journals.elsevier. com/she-ji-the-journal-of- design-economics-and-innovation

Also accessible from here is information on copyright, submission instructions and more.

Contact details for questions arising after acceptance of an article, especially those relating to proofs, will be provided by the editorial office. Advertising information If you are interested in advertising or other commercial opportunities please email majin.sheji@ icloud.com and your enquiry will be passed to the correct person who will respond within two working days. Permissions

Permission from Tongji University is required for resale outside the institution and for all derivative works for commercial purposes, including compilations and translations (please consult [email protected]). Notice

No responsibility is assumed by She Ji nor Elsevier for any injury and/or damage to persons, property as a matter of product liability, negligence, or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein. Because of rapid advances in medical sciences, in particular, independent verification of diagnoses and drug dosages should be made. Although all advertising material is expected to conform to ethical (medical) standards, inclusion in this publication does not constitute a guarantee or endorsement

(GLWRULDO2IÀFH Correspondence She Ji

c/o Jin Ma Tongji University College of Design and Innovation

Building No. 2, Room IS-112 281 Fu-Xin Road

Shanghai, 200092 China Tel: +86 21 6598 7990 [email protected] About the Cover Every autumn, the Tongji University College of Design and Innovation organizes the Emerging Practices Conference (EPC). Each EPC is a rich, interactive conversation on design research and design education. The theme of EPC2016 was Design in Complexity. This conference continued the exploration of design for complex sociotechnical systems that the DesignX Collaborative initiated in 2014 and expanded at EPC2015. Jiaying Hu developed the cover of this issue for EPC2016. It depicts the challenge of complexity that design and designers face today.

She Ji: The Journal of Design, Economics, and Innovation is a peer-reviewed, open access journal published quarterly (spring, summer, autumn, and winter).

For information about She Ji, visit:

http://www.journals.elsevier. com/she-ji-the-journal-of- design-economics-and-innovation

Full text available on ScienceDirect®.

She Ji: The Journal of Design, Economics, and Innovation Volume 2, Issue 2, Summer 2016

(3)

The latest issue of She Ji exemplifies several aspects of design, economics, and in-novation. In our view, design functions on a continuum. At one end, we find sim-plicity in the form of artifacts, processes, products, and services. At the other, we find complexity, often in complex adaptive systems, networks, and processes that straddle the traditional boundaries of many organizations. Designers work with all of these—and all come into being through the application of economics, innova-tion, and ingenuity.

Today many human beings are required to work in tandem with machines, in large-scale, networked systems. Increasingly, digital systems are connecting organizations and remote actors in clusters and constellations, and on massive online platforms. Projects, products, and services bridge and engage stakeholders, producers, and users from multiple groups. Kyle Behymer and John Flach ask how to make progress as we move “From Autonomous to Sociotechnical Systems: Designing Effective Collaborations.” Following the article, Hugh Lubberly, Derek Miller, and Susu Nousala offer views and comments in response.

As the design field expands its scope to become a research discipline, the ques-tion of how to develop and deploy theoretical foundaques-tions has become a vital issue. In “Examining Practical, Everyday Theory Use in Design Research,” Jordan Beck and Erik Stolterman seek a grounded understanding of theory in design research. Danah Henriksen, Jeffrey Bardzell, and Deirdre Barron comment on this article in a series of short, focused essays.

Cara Wrigley’s article “Design Innovation Catalysts: Education and Impact” focuses specifically on generating innovation in design, and on how we educate designers to work with this crucial factor in successful design.

In “De-Colonizing Design Thinking,” Jerry Diethelm examines differences between science and design. In a time when research is essential to our field, the concept of design science has several meanings. The cultures of science and design intersect and cooperate at the same time that they often conflict and draw apart. Diethelm considers four root metaphors—problem, intervention, limits, and satis-fice. He invites us to think about the ways in which we can—and ought to—think about science and design.

In this issue, we launch our book review section. Luke Feast and Donald Norman discuss two different books by Per Mollerup. The first book is Data Design: Visualizing Quantities, Locations, Connections. The second is Simplicity: A Matter of Design.

We also introduce a second new feature to She Ji with our letters section—a forum for debate and inquiry. While the editors invite comments on selected arti-cles, our readers also comment on ideas and issues in letters to the journal.

The Design Continuum—From

Simplicity to Complex Systems

Publishing services by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

(6)

In this issue, Stuart Walker offers “Concerns about Quantifying Design Re-search Quality” in response to an article by Gerda Gemser and Cees de Bont. The authors reflect on the questions and concerns in their reply.

She Ji is a community of readers, writers, and editors. We’re glad to have you with us. We welcome your letters on the articles in this issue. We also welcome suggestions for the book review section. Let us hear from you!

(7)

Keywords

Human-autonomy interaction Collaborative systems Human-machine teaming

Received June 6, 2016 Accepted September 8, 2016

Emails Kyle J. Behymer (corresponding author) [email protected]

John M. Flach MRKQÁDFK#ZULJKWHGX

From Autonomous Systems to

Sociotechnical Systems: Designing

Effective Collaborations

Kyle J. Behymer, Infoscitex, USA

John M. Flach, Wright State University, USA

Abstract

Effectiveness in sociotechnical systems often depends on coor-dination among multiple agents (including both humans and autonomous technologies). This means that autonomous technologies must be designed to function as collaborative systems, or team players. In many complex work domains, success is beyond the capabilities of humans unaided by technologies. However, at the same time, human capabilities are often critical to ultimate success, as all automated control systems will eventu-ally face problems their designers did not anticipate. Unfortunately, there is often an either/or attitude with respect to humans and technology that tends to focus on optimizing the separate human and autonomous compo-nents, with the design of interfaces and team processes as an afterthought. The current paper discusses the limitations of this approach and proposes an alternative where the goal of design is a seamless integration of human and technological capabilities into a well-functioning sociotechnical system. Drawing lessons from both the academic (SRK Framework) and commercial (IBM’s Watson, video games) worlds, suggestions for enriching the coupling between the human and automated systems by considering both technical and social aspects are discussed.

Publishing services by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

The peer review process is the responsibility of Tongji University and Tongji University Press.

(8)

Introduction

The goal of this paper is to argue that eff_{ective collaboration is critical to the} suc-cess of human-machine teams, and to provide a framework (illustrated in Figure 1) for addressing the coupling between humans and machines such as autonomous agents. This is particularly important in the context of sociotechnical systems where multiple agents must collaborate to solve complex or wicked problems. 1 We will begin with a brief example to illustrate some of the dynamics of eff_ective collaboration.

In 2005, Playchess.com hosted a chess tournament in which teams of human players could use computer assistance during matches. 2 The chess super computer Hydra was also entered into the competition, and after recently defeating Grand Master Michael Adams 5 ½_–½_{in a six game match, was considered to be the} pro-hibitive favorite. Surprisingly, Hydra was eliminated before the semi-finals, with three of the four semi-finalists consisting of Grand Master-led teams equipped with supercomputers. Even more surprising was the fourth semi-finalist and eventual winner, team ZachS, composed of two relatively amateur chess players named Steven Crampton and Zackary Stephen using ordinary computers. 3

The Elo Rating system 4 —a method of rating chess player skill level based on head to head results—puts team ZachS’s victory into perspective. Table 1 lists Elo ratings ranging from novice to world champion. Current world champion Magnus Carlsen obtained the highest Elo rating (2882) in history for a human player (Garry Kasparov’s best was 2851, Bobby Fischer’s was 2785). 5 At the time of the tourna-ment, Hydra’s estimated Elo rating was 3000, and the runner up team was led by two 2600+ Grand Masters. Crampton and Stephen’s Elo ratings were 1685 and 1398 respectively. 6

Team ZachS was vastly outclassed in chess skill and computer hardware, yet overcame Hydra and the Grand Masters armed with super computers by quickly

and effi_{ciently manipulating their machines to deeply} explore relevant positions and shrink the search space for their chess computers. 7 The higher skill level of Hydra and the Grand Masters equipped with super computers was not enough to overcome the seamless collaboration between the less skilled ama-teurs and their weaker computers. As Garry Kasparov stated, “Weak human + machine + better process was superior to a strong computer alone and, more re-markably, superior to a strong human + machine + inferior process.” 8

Synergy

In fact, the human-machine combination has the potential to outperform hu-man-alone and computer-alone in many domains. For example, human forecasters at the National Weather Service can improve the accuracy of computer precipi-tation forecasts by 25% and computer temperature forecasts by 10% over comput-er-only forecasts, 9 and human-computer teams have the potential to outperform both doctors and computer algorithms at correctly interpreting mammograms. 10 However, as the chess example illustrates, group performance is more than the sum of the abilities of the individuals that compose the group. For example, the collective intelligence of a group of people is more highly correlated with the group’s social sensitivity, equality in turn taking, and the number of women in the group than the average intelligence of group members or the IQ of the group’s Table 1. Chess Elo Ratings.

Elo Skill Level

1 For discussion of ‘wicked problems’ see Richard Buchan-an, “Wicked Problems in Design Thinking,” Design Issues 8, no. 2 (1992): 14–19.

2 Various names have been proposed for this type of chess, including advanced chess, cyborg chess, centaur chess, and freestyle chess.

3 Clive Thompson, Smarter Than You Think: How Technology is Changing Our Minds for the Better, reprint ed. (New York: Penguin Books, 2014), 4–5.

4 The Elo Rating system, developed for chess by Arpad Elo in 1978, has also been used to measure skill level in many sports including baseball, basket-ball, footbasket-ball, soccer, and tennis.

5 Ratings from http://www. chessgames.com/chessstats. 7 Garry Kasparov, “The Chess Master and the Computer,” The New York Review of Books, February 11, 2010, accessed September 17, 2016, http://www. Noise: Why So Many Predictions Fail—But Some Don’t (New York: Penguin Books, 2012), 125.

10 Mark Gaynor, George Wyner, and Amar Gupta, “Dr. Watson? Balancing Automation and Human Expertise in Healthcare Delivery,” in Leveraging

(9)

smartest person. 11 _{Similarly, cardiac surgery e}_ffi_{ciency is more dependent on the} surgical team’s cumulative experience than the individual experience of the at-tending surgeon. 12 _{Teamwork beats talent when talent doesn’t work as a team, as} the saying goes. Pairing the best human with the best computer won’t necessarily result in the best performance. Thus, there is much that the designers of auton-omous technologies can learn from the literatures on team effectiveness 13 and distributed cognition. 14

The human-machine team is like a pair of scissors cutting through the fabric of the work domain. Sharpening one or the other of the blades—increasing the capabilities of either the human or the machine—might lead to cleaner cuts. But if there were no screw—no effective interface—to hold the blades together, no matter how sharp they were, the scissors would not cut at all.

As shown in Figure 1, the distributed sociotechnical system includes multiple potential loops, where each loop is bounded in terms of access to information or perception (P) and action or control capability (C). The quality of the observer and control processes depends on the quality of coupling among the multiple loops. If the coupling is rich, then the sociotechnical system can be a better observer than any of the components. For example, each loop may provide unique information about the state of the problem domain—so redundant data across the loops can be useful in filtering sampling noise (averaging, common mode rejection). Also, rich coupling allows coordination of multiple actions to achieve a common goal. Without coupling, the actions of each loop will be a potential disturbance relative to other loops. If the coupling within the network of collaborating agents is rich—if there is effective communication—then the whole can serve as a more effective control system than any of the components. If the coupling within the network is poor, then there is a potential for the whole to be worse than the best component due to interference between loops.

Unfortunately, system developers often focus on increasing the capabilities of autonomous agents, without giving sufficient consideration to how they will in-terface with human operators. This approach often fails to recognize the technical limitations of autonomous components and the potential of a human-machine team. Additionally, focusing on improving the technical capabilities of autonomous agents without considering how these will interact with human operators often leads to poor coupling within the human-machine team. This paper proposes an al-ternative approach—frame the problem as interfacing to the problem domain and

the other ‘agents,’ not only to improve observability and controllability, but also to

11 Anita Wooley et al., “Evi-dence for a Collective Intelli-gence Factor in the Performance of Human Groups,” Science

330, no. 6004 (2010): 686–88, DOI: http://doi.org/10.1126/ science.1193147.

12 Andrew Elbardissi et al., “Cumulative Team Experience Matters More than Individual Surgeon Experience in Cardiac Surgery,” Journal of Thoracic Cardiovascular Surgery 145, no. 2 (2013): 328–33.

13 For example, see Eduardo Salas and Stephen M. Fiore, eds.,

Team Cognition: Understanding the Factors that Drive Process and Performance (Washington, D.C.: American Psychological Association, 2004).

)RUH[DPSOHVHH-DPHV

Hollan, Edwin Hutchins, and David Kirsh, “Distributed Cognition: Toward a New Foundation for Human-Comput-er IntHuman-Comput-eraction Research,” ACM Transactions on Computer-Hu-man Interaction 7, no. 2, (2000): 174–96.

Figure 1 Framework for Hu-man-Machine teams collaborat-ing to solve a complex domain

(10)

take the technical and social aspects involved in enriching the coupling between components into account.

The Prosthetic/Substitution Approach: The Technical Limits

Replacing the human user with autonomous systems, or at the very least, designing autonomous systems to either compensate for or overcome the limitations of a human user has been referred to as a prosthesis approach 15 or substitution-based approach, 16 and is based on the idea that designers should identify human weak-nesses and replace them with automated strengths. 17 The origin of this approach can be traced to a 1955 Dartmouth manifesto in which a group of artificial intelli-gence scientists—John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon—proposed a goal of discovering how machines could solve the kinds of problems that had previously been the domain of skilled humans, without con-sidering if/how these machines would interact with people. 18 _{The question wasn’t} how to design an autonomous system that could collaborate with a person to com-plete a task; rather, the question was how to design an autonomous system that could substitute for human capabilities. 19

Advocates of the prosthetic/substitution approach often present humans as poor decision makers, citing studies in which human participants in contrived lab-oratory tasks perform poorly compared to mathematical decision-making models like Bayes’ Theorem. 20 These studies conclude that human rationality is bounded and is therefore limited. What is often underrepresented is the fact that autono-mous systems are bounded as well.

In 2005, the Defense Advanced Research Project Agency (DARPA) created a research program called COORDINATORS, whose goal was to develop hand-held au-tomated agents that would help geographically distributed warfighters coordinate and adapt mission plans in response to unexpected events. 21 In 2008, a capstone ex-ercise 22 was conducted to compare two different automated agent approaches—de-veloped by two separate teams—with the performance of a control team of human operators. 23 One automated agent approach removed the human from the loop entirely; the team of humans it supported acted as actuators, only taking actions the agent assigned to them. This approach performed significantly worse than the control team of human operators, and the developers concluded that if automated agents are not provided with appropriate situation constraints, they will inevitably trend towards a subpar solution in the face of a highly dynamic environment. 24 The second automated agent approach fared slightly better, but only because its design allowed its developer team to input a human-devised strategy tailored to the specific scenario prior to the exercise. According to the designers of this approach, the automated agents still failed because they did not have an effective method of narrowing the enormous search space of the exercise’s dynamic environment. The designers’ key takeaway is worth quoting verbatim: “The most interesting result of the evaluation is that it is so difficult to outscore humans in a complex planning and scheduling problem.” 25

When an autonomous system is presented as the answer to the “problem” of human-bounded rationality it is inevitable that the technology will eventually reach its own limits. Consider Watson, the IBM supercomputer designed to com-pete in Jeopardy!—a television quiz show in which three contests comcom-pete to earn the most money by answering trivia questions. After two rounds, Watson was soundly defeating two of the best human Jeopardy! players ever, Ken Jennings and Brad Rutter, 26 _{by a score of $36,681 to $2,400 and $5,400 respectively. Then came} Final Jeopardy! The category was U.S. Cities, and the clue was “Its largest airport is named for a World War II hero; its second largest, for a World War II battle.”

15 Emilie Roth, Kevin Bennett, and David Woods, “Human Interaction with an ‘Intelligent’ Machine,” International Journal of Man-Machine Studies 27, no. 5 (1987): 479.

16 Erik Hollnagel, “From Function Allocation to Function Congruence,” in Coping with Computers in the Cockpit, ed. Sidney Dekker and Erik Hollna-gel (Aldershot: Ashgate Publish-ing Company, 1999), 29–53.

17 Sidney Dekker and David Woods, “MABA-MABA or Abracadabra? Progress on Human-Automation Co-Ordina-tion,” IEEE Intelligent Systems 29, no. 5 (2002): 240–44.

18 Susan Epstein, “Wanted: Col-laborative Intelligence,” $UWLÀFLDO Intelligence 221 (2015): 36.

19 Gary Klein et al., “Ten Challenges for Making

Auto-PDWLRQD¶7HDP3OD\HU·LQ-RLQW

Human-Agent Activity,” IEEE Intelligent Systems 19, no. 6 (2004): 91–95.

-RKQ)ODFKDQG)UHG9RRU -horst, What Matters?: Putting Common Sense to Work (Dayton: Wright State University Librar-ies, 2016), 169–74.

21 Bob Kohout, “The DARPA COORDINATORS Program: A Retrospective,” in Proceedings of the 2011 International Confer-ence on Collaboration Technol-ogies and Systems, ed. Waleed Smari and Geoffrey Fox (New

-HUVH\,(((DOVR

available at http://ieeexplore. ieee.org/document/5928708/.

:KLOHHPSOR\HGDW-;7 $SSOLFDWLRQV,1&WKHÀUVW

author managed a DARPA Phase II STTR (Small Business Tech-nology Transfer) that developed the user interface used by one of the automated agent devel-oper teams during this capstone exercise.

23 Rajiv Maheswaran et al., “Multi-Agent Systems for the Real World,” in Proceedings of the 8th_{International Joint}

Conference on Autonomous Agents and Multiagent Systems, vol. 2 (Budapest: AAMS, 2009): 1281–82.

(11)

Jennings and Ritter correctly answered “What is Chicago?”, while Watson answered “What is Toronto?????” 27

Watson’s response elicited an audible groan from an audience full of IBM pro-grammers likely thinking, “Toronto isn’t a U.S. city.” Except, as Watson was all too aware, Toronto is a U.S. city—in Illinois, Indiana, Iowa, Kansas, Missouri, Ohio, and South Dakota. Adding to the confusion, Watson was programmed to de-emphasize category names, as they are often only weakly tied to the content of the clue, and can contain puns or other forms of wordplay. So while this question was relatively easy for a human player, it proved to be Watson’s Achilles’ heel.

In their haste to replace irrational humans with rational machines, advocates of the prosthetic/substitution approach have failed to recognize that autonomous systems also have limits—what’s more, they are overlooking a better solution. Imagine a warfighter teaming with a DARPA COORDINATOR agent. Imagine Brad Rutter teaming with Watson to play Jeopardy!. The critical point is that the ratio-nality of all agents—human and machine—are bounded with respect to the com-plexity of many work domains. Thus, it will often be necessary to combine the capabilities of multiple agents, each with unique bounds and capabilities, in order to meet the demands for effective performance reflected in Ashby’s Law of Requi-site Variety. 28

The Prosthetic/Substitution Approach: Unintended Social Consequences

“John Henry hammered on the right-hand side. Steam drill kept driving on the left. John Henry beat that steam drill down. But he hammered his poor heart to death.” 29

—Kennedy, Kennedy, and Baker, Knock at a Star

Thirty-five years ago, Weiner and Curry noted that the general public had two opinions in regard to automation: skepticism about its capabilities and fear of its consequences—widespread unemployment at best, and Orwellian dystopia at worst. 30 _{The prosthetic/substitution approach did little to alter these opinions in} subsequent years, with coverage in the media being divided between fear mon-gering and disdain. 31 The story of John Henry’s epic battle and ultimately Pyrrhic victory over the steam engine exemplifies the fear that people have of being re-placed (or even destroyed) by automation, a fear that is omnipresent in popular culture. The first cinema robot appeared in Fritz Lang’s 1927 silent film Metropolis, a Maschinenmensch (German for machine-human) created by the evil scientist Rotwang to replace Maria, an activist working to better the lives of the workers on whose backs the gleaming city of Metropolis has been built. The Maschinenmensch is designed to look exactly like Maria and has a single goal: to destroy Maria’s repu-tation among the workers. The Maschinenmensch sows chaos among the workers and they riot, causing floods and destroying parts of the city. Eventually the subter-fuge is discovered and the Maschinenmensch is burned at the stake. Similar themes are present in modern films such as Terminator, The Matrix, and Avengers: Age of Ul-tron—a machine designed to replace humanity turns on humanity. Both Schafer et al., 32 and Parasuraman and Riley 33 have argued that these fictional portrayals have influenced society’s perception of autonomous systems and may create dissonance when people interact with autonomous systems in the real world.

Another unintended social consequence of failing to take social factors into account when designing automated systems is disdain. In 1993, Microsoft started the Lumiere project, with the goal of developing an automated capability that could detect a user’s goals based on their actions and provide assistance to the user to meet his or her goals. 34 _{In 1997, after more than 25,000 hours were spent on}

of the 9th_{International} Confer-ence of Autonomous Agents and Multiagent Systems (AAMAS 2010), ed. Wiebe van der Hoek et al. (Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems, 2010): 1338.

25 Maheswaran et al., “Multi-Agent Systems,” 1282.

-HQQLQJVZRQDUHFRUG

straight games in 2004. Rutter has never been defeated by a human player in Jeopardy! and is not only the all-time Jeopardy! money winner but also the all-time game show money winner.

27 The number of question

PDUNVLQGLFDWHVWKHODFNRIFRQÀ -dence Watson had in its answer. See “What is Toronto?????” YouTube video, 1:38, from Jeopardy! on February 15, 2011, posted by Loginer, February 15, 2011, https://www.youtube.com/ watch?v=7h4baBEi0iA. 28 W. Ross Ashby, An Intro-duction to Cybernetics (1956; Principia Cybernetica Web, 1999), 206, accessed September 17, 2016, http://pespmc1.vub. ac.be/books/introcyb.pdf.

;-.HQQHG\'RURWK\

Kennedy, and Karen Lee Baker, Knock at a Star: A Child’s Intro-duction to Poetry (New York: Little, Brown and Company, 1999), 21.

30 Earl Wiener and Renwick Curry, “Flight-Deck Automation: Promises and Problems,” Ergo-nomics 23, no. 10 (1980): 996.

31 Epstein, “Collaborative Intelligence.”

32 Kristin Schaefer et al., “The Future of Robotic Design: Trends from the History of Media Representations,” Ergonomics in Design 23, no. 1 (2015): 13–19.

33 Raja Parasuraman and Victor Riley, “Humans and Automation: Use, Misuse, Disuse, Abuse,” Human Factors 39, no. 2, (1997): 230–53.

(12)

usability testing, Clippy 35 was released as part of Office 97. 36 It was so despised that its removal from Office was included in the later Windows XP system sales pitch. 37

Microsoft failed to realize how Clippy was perceived. “I HATED that clip. It hung around watching you with that nasty smirk. It wouldn’t go away when you wanted it to. It interrupted rudely and broke your train of thought. It never actually had an answer to questions I had.” 38 _{Microsoft spent 25,000 hours testing the}

tech-nical capabilities of Clippy, but ignored the social components critical to ensuring a rich coupling between Clippy and the human user, dooming Clippy to failure.

Thus, in addition to considering the technical aspects related to the collabo-ration between humans and automated machines, it is also necessary to consider the social aspects. What does it mean for an automaton to be effective as a team player? How does an automaton earn the trust of an operator? How is it possible to strengthen the bonds among human and autonomous teammates? How can an automaton assist, without interrupting human processes or undermining human capabilities?

A Collaborative Systems Approach: Complementing Capabilities

“Basically, meaningful interaction with an environment depends upon the existence of a set of invariate constraints in the relationships among events in the environment and between human actions and their effects.” 39

—Jens Rasmussen, “Skills, Rules, and Knowledge”

If there is a rich coupling between the components outlined in Figure 1, the hu-man-machine team will jointly bridge Hutchins, Hollan, and Norman’s Gulf of Eval-uation and Gulf of Execution 40 (see Figure 2A). However, if there is a poor coupling between the humans and technologies, then another gulf is introduced, creating additional uncertainties for each component (see Figure 2B).

On one side of these gulfs resides the human-machine team, with their goals and intentions. On the other side is the work domain. The size of the gulf of execu-tion depends on the effectiveness of the actions or controllability the human-ma-chine team has to achieve his or her goals. The size of the gulf of evaluation depends on how well the human-machine team can observe, perceive, and under-stand the state of the world in terms of their intentions—and find the best move on the chessboard. The goal-directed interaction between the human-machine team and the physical system is dependent upon constraints. 41 To span the gulf of evaluation it is vital to make the constraints of the work domain salient to the hu-man-machine team. 42 To span the gulf of execution the constraints associated with the human-machine team—designing controls and displays that are consistent with a human operator’s reasoning capabilities—must be considered. 43 Rasmussen’s Skills, Rules, Knowledge (SRK) framework 44 defines three ways in which people represent constraints, signals, signs, and symbols which, in turn, distinguish three levels of human performance—skill-based, rule-based, and knowledge-based.

Skill-based behavior consists of sensory-motor tasks without conscious control. 45 In rule-based behavior, an individual has a set of predetermined solutions that are triggered by specific conditions, or signs. Knowledge-based behaviors occur when an individual encounters a novel unexpected situation for which no procedure exists. 46

Consider a chef preparing a meal—chopping vegetables or continuously ad-justing the gas flame of a burner to perfectly fry an egg are skill-based behaviors, and following a recipe is rule-based behavior. Now imagine that the recipe calls for vanilla extract—but when the chef looks in the pantry, the bottle is empty. The chef considers his or her options: (1) leave the cooking process, which is currently in a critical stage to acquire more vanilla extract; (2) skip the vanilla extract; or (3)

35 “Clippy” is a nickname;

´&OLSSLWµLVLWVRIÀFLDOQDPH

36 Eric Horvitz, “Lumiere Project.”

37 Brian Whitworth, “Polite Computing,” Behaviour & Information Technology 24, no. 5 (2005): 359.

38 Ibid.

-HQV5DVPXVVHQ´6NLOOV

Rules, and Knowledge; Signals, Signs, and Symbols, and other Distinctions in Human Perfor-mance Models,” IEEE Trans-actions of Systems, Man, and Cybernetics 13, no. 3 (1983): 258.

(GZLQ+XWFKLQV-DPHV

Hollan, and Donald Norman, “Direct Manipulation Interfac-es,” Human-Computer Interac-tion 1, no. 4 (1985): 319. 41 Kim Vicente, Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work (Boca Raton: CRC Press, 1999), 10.

.HYLQ%HQQHWWDQG-RKQ

Flach, Display and Interface Design: Subtle Science, Exact Art (Boca Raton: CRC Press, 2011), 114.

43 Ibid.

44 Rasmussen, “Skills, Rules, and Knowledge,” 257–66.

45 Ibid., 258.

(13)

find a suitable substitute. When the chef decides to try almond extract as a substi-tute, he or she is exhibiting knowledge-based behavior. 47

The prosthetic/substitution approach attempts to replace the user at all three of these modes of interaction. While some researchers have suggested that au-tonomous systems are easiest to develop for skill-based behaviors and hardest for knowledge-based behaviors, 48 _{building autonomous systems that replace humans} in all three categories can be very challenging. For example, cutting vegetables is simple for an experienced sous chef, but very difficult for an autonomous system. 49 Rather than replace the human, the focus should be on designing autonomous systems that support the human in all three modes of interaction—skill, rule, and knowledge-based.

Autonomous systems can be designed to augment a human’s skills, rules, and knowledge (SRK) behaviors in several ways. First, the autonomous system should help the human explore the state space. IBM is currently repurposing Watson to help chefs do this very thing. Imagine a chef that has just returned from the garden with a plentiful bounty of sweet corn, lima beans, zucchini, and onions, and no idea of what to prepare. Using the Chef Watson app, 50 the chef can enter these ingredients and get back a variety of recipes featuring these ingredients—including zucchini tacos, zucchini fricassee, zucchini curry.

The chef can then collaborate with Chef Watson by adding constraints and narrowing the search space. For example, if the chef is in the mood for a specific type of cuisine, he or she can select the “Pick a Style” option. By default, Watson will recommend styles based on the ingredients—in this case Watson suggested Peruvian, Basque, Nuevo Latino, Moroccan, Tailgating, Tex Mex, Nashville, Cajun, and Israeli—but also provides the option for the chef to select an out of the box cuisine for these ingredients, like Japanese. Table 2 illustrates how Chef Watson has modified the zucchini taco recipe to infuse the dish with Japanese flavors. For example, Manchego cheese, which has a flavor profile similar to miso, has replaced goat cheese.

47 Note that if the almond extract replacement is deemed a success, the chef will likely switch to rule-based behavior in a similar situation in the future—“I am out of vanilla extract; I will use almond extract instead.”

48 Mary Missy Cummings, “Man versus Machine or Man + Machine?,” IEEE Intelligent Systems 29, no. 5 (2014): 66.

49 Ian Lenz, Ross Knepper, and Ashutosh Saxena, “DeepMPC: Learning Deep Latent Features for Model Predictive Control,” in

Proceedings of Robotics: Science and Systems (Rome: Sapienza University of Rome, 2015): 1–9, also available at http://www. roboticsproceedings.org/rss11/ p12.html.

50 Available at https://www. ibmchefwatson.com/.

(14)

At this point, should the chef decide to make the Japanese zucchini tacos, Chef Watson has a recipe queued up and ready for the chef to access. And if the chef heads to the pantry and can’t find any caraway seed, luckily Chef Watson has it covered—potential substitutes are generated upon request. The chef, aided by his or her sous chef, Chef Watson, can now get busy cooking.

The Chef Watson program illustrates how an automaton can be used both to complement cooking skills, and potentially stimulate a chef’s creative ability to invent new solutions to the cooking problem. Note that the point here is neither to use Chef Watson as a substitute for a human, nor to use it to enforce procedural compliance by a human. Rather, Chef Watson becomes a creative partner—it helps the ‘team’ explore the cooking problem efficiently and effectively, think produc-tively, and experiment with innovative alternatives.

A Collaborative Systems Approach: Creating Social Cohesion

Developers must also consider social aspects in order to facilitate a rich coupling between the human and autonomy. In 2004, Valve Corporation released Half-Life 2

(HL2), the successor to their massively successful 1998 debut Half-Life. HL2 is a first person shooter game in which gamers play the role of Gordon Freeman, a scien-tist who finds himself inspiring a resistance movement against a conquering alien force in a dystopian future. The HL2 series deftly filled the shoes of its beloved predecessor by developing several innovative gameplay elements, including a fully realized AI sidekick named Alyx Vance. Alyx received almost universal praise, often ranking at the top or near the top of lists ranking the greatest non-playable charac-ters of all-time, even years after her introduction. 51

Alyx’s success is a credit to Valve’s intense development and play testing pro-cess. Over 100 actresses were auditioned to provide Alyx’s voice, with developers seeking a voice actress that could be charming and warmly intimate, but could also be strong, confident, and believable. 52 During play testing it became clear that having Alyx be capable of providing assistance to the player wasn’t enough. Developers initially tried to create a sense of urgency by having Alyx say things like “Hurry up!” and “Keep moving!” but players felt like they were being nagged,

-RH0DUWLQ´7RS&RP-puter Game NPCs,” bit-tech. QHWODVWPRGLÀHG-XO\

http://www.bit-tech.net/gaming/ pc/2008/07/18/top-10-computer-game-npcs/5; Adam Dodd, “Top 10 Video Game Sidekicks,” Cheat Code Central, accessed May 1, 2016, http://www.cheatcc. com/extra/top10videogameside-kicks2.html.

52 David Hodgson, Half-Life 2: Raising the Bar (Roseville: Prima Games, 2004), 153.

Table 2. Recipe differences (highlighted in gray).

Zucchini Taco Japanese Zucchini Taco

Egg Egg

Lima Bean Lima Bean

Onion Onion

Corn Corn

Zucchini Zucchini

Vegetable Oil Vegetable Oil

Butter Butter

Flour Tortilla Flour Tortilla

Thai Chile Chile de Arbol

Lemon Grass Jalapeno Pepper

Pineapple Juice Lemon Juice

Goat Cheese Manchego Cheese

(15)

and ended up hating Alyx. This led to a major design change—having Alyx almost

always follow the player, rather than leading the way. 53

Ultimately, making Alyx likeable was just as important—if not more so—than making her capable. The developers designed multiple scenes to humanize and endear Alyx to the player, and each scene had multiple variables that had to be just right. “If you don’t like Alyx, you’re not going to have much fun with Episode 1. So Alyx being likeable was one of our most crucial design goals. Little moments like

the Zombine 54 _{joke are designed to make Alyx more endearing…. Surprisingly,}

lighting was really important too. Under red light, Alyx’s self-deprecating groan looked more like she was sneering at the player for not getting the joke. Changing the lighting to blue and then adjusting the direction of the light so that it changed

the shadows on her face fixed the problem.” 55

By the time the player reaches the end of HL2: Episode 2 they will have spent

many hours working with Alyx towards a common goal, and most players will have developed an emotional attachment to her, which Valve uses to devastating

effect. At the end of HL2: Episode 2 the player watches helplessly as Alyx’s father is

brutally murdered right in front of her. As Alyx—who is also restrained—screams in rage and agony, there’s an incredibly brief moment when she glances back at the player, whispering “Gordon,” her eyes pleading with the player for help. This gut-wrenching sequence continues as Alyx clings to her father’s lifeless body, and

her desperate sobs remain even after the screen fades to black. 56 This is how

devel-opers can create an AI that connects with the user on an emotional level.

Everything Valve got right with Alyx Vance, Microsoft got wrong with Clippy. Valve realized during the design process that Alyx should follow the player’s lead and not control the action. Clippy would show up uninvited, take control of the user’s mouse cursor, and keep coming back no matter how many times the user

sent it away. 57 While Valve discovered that in the wrong lighting, what was

in-tended to be a humanizing groan could be perceived as looking down on the player, Clippy’s tone always seemed to convey that it knew better than the user. Valve spent thousands of hours perfecting Alyx’s interaction with the player, resulting in one of the most beloved video game characters ever. Microsoft did not—and ended up with perhaps the most notorious automated assistant ever.

Conclusions

As many researchers have noted repeatedly, 58 effectiveness in sociotechnical

systems will often depend on whether the technologies function as collaborative system team players. In many complex work domains, success is beyond the capa-bilities of un-aided humans, yet human capacapa-bilities are often critical to ultimate success. An important motivation behind the Cognitive Systems Engineering ap-proach was the realization that no matter how carefully designed, all automated control systems will eventually face situations that were not anticipated at the time

of their design. 59 _{Thus, at some point the human operators of those systems will be}

called upon to complete the design. In other words, the human operators will need to intervene to creatively deal with the requisite variety that was not anticipated by the designers of the automated systems. The SRK framework is specifically geared towards drawing attention to user interfaces, and ways to design representations so that human and automated systems can work together to creatively respond to the inevitable, unanticipated variability endemic to complex work domains.

In sum, the challenge is to move beyond an either/or attitude with respect to humans and technology—the classic “Humans are Better at/Machines are Better at” lists—that tends to focus on optimization of separate human and autonomous com-ponents as the top priority, and leaves the design of interfaces and team processes

53 “Half-Life 2: Episode One— Developer Commentary— Undue Alarm,” YouTube video, 10:25, from Half-Life 2 Episode 1: Commentary by Matt Wood, posted by Grey Torch, April 13, 2015, https://www.youtube.com/

ZDWFK"Y JX$Y;K'*IF.

54 “Zombine” is a portmanteau of zombie and combine. The main antagonist force in Half-Life 2 is known as The Combine.

55 “Half-Life 2 Developer Commentary. Episode One [full, 2016 remake]” YouTube video, 50:45, from Half-Life 2 Developer Commentary. Episode One [full, 2016 remake]: commentary by Erik Wolpaw, posted by Steady Eddie, September 26, 2016,

https://www.youtube.com/ watch?v=1I8jCv8WT8g.

56 Scene available at https:// www.youtube.com/watch?v=_ zbrkhcF4_8 starting at 6:37.

57 Whitworth, “Polite Comput-ing,” 360.

58 Klein et al., “Ten Challenges,” 91–95; Dekker and Woods, “MABA-MABA,” 240–44.

(16)

as an afterthought. The alternative is to take a holistic perspective, and to begin thinking in terms of both/and, where the goal of design is a seamless integration of human and technological capabilities into a well-functioning sociotechnical system. Success in complex domains will ultimately depend on the ability of humans AND technologies working together as well coordinated teammates—each contributing unique abilities to create a team with the potential to be greater than the sum of its parts, and thus jointly bridge the gulfs of execution and evaluation in order to address the requisite variety of complex domains, or wicked problems.

Acknowledgments

The authors would like to thank Brian McKenna for comments on an earlier draft of this paper, as well as two anonymous reviewers.

Commentary

Value-Pluralism and the

Collaboration Imperative

in Sociotechnical Systems

Derek B. Miller, The Policy Lab®, and The Pell Center for International Relations and Public Policy, Salve Regina University, USA

[email protected]

doi:10.1016/j.sheji.2016.12.001

In their article “From Autonomous Systems to Socio-technical Systems: Design Effective Collaborations,” Behymer and Flach make the case for the “seamless integration of human and technological capabilities into a well-functioning sociotechnical system.”1 The appeal is driven by the potential for increased eff ec-tiveness in a range of desirable actions that might result from better human-technology cooperation. The authors are also driven by a concern for the state of ongoing neglect. As they explain, there is currently too much attention paid to the “optimization of sepa-rate human and autonomous components” to systems thereby leaving “the design of interfaces and team processes as an afterthought.”2

Should this appeal be heeded, it could result in a new commitment to an agenda of attending to this human-technology nexus, thereby enriching the possibilities for what they are calling sociotech-nical systems. Importantly, though, the authors are not merely pointing to a contemporary gap in the

human-technology working relationship, but instead are suggesting that the separation of technology from human support—full autonomy, as it were—is some-thing of a chimera, because while technology can indeed be automated, it cannot by its nature respond to the range of experience that would make it func-tion well in all circumstances:

“No matter how carefully designed,” they argue, “all automated control systems will eventually face situations that were not anticipated at the time of their design. Thus, at some point the human operators of those systems will be called upon to complete the design. In other words, the human operators will need to intervene to creatively deal with the requisite variety that was not anticipated by the designers of the auto-mated systems.”3

On that basis, it is essential that the human-tech-nology team process and interface not be treated as an afterthought, but addressed specifically.

What I would like to do, here—albeit briefly—is provide support to Behymer and Flach’s appeal for an enriched agenda on this matter by situating the prob-lematic they observe in a wider context, and demon-strating that not only will automated control systems eventually face these complex situations, but humans will too. And furthermore, we always will.

(17)

also the UN Institute for Disarmament Research, where I worked for almost a decade.

This perspective on the challenges of autonomy problematizes—and raises the urgency of—the otherwise technical but morally banal observation about the need for people to “creatively deal with … requisite variety.” If weapons systems are empow-ered by their designers to act lethally, but ultimately cannot be trusted to do so autonomously because the range and variety of experience needed will always be beyond their design parameters—and if the team process of collaborating with people is hindered and treated as an afterthought—then we have a big problem.

One familiar response to observed gaps or insuffi-ciencies in autonomous systems and artificial intelli-gence is to claim that the field simply has not solved for certain challenges yet, but the rate of learning and the momentum in the field points to success later. This line of reasoning might conclude that Behymer and Flach are correct in their observations at the moment, but that technological developments will eventually close those gaps, meaning that special attention to team processes is only a temporary mea-sure, and one situated in the current state of affairs rather than a permanent condition of technology itself and hence something in need of a separate and dedicated agenda.

I find this to be a reasonable argument—but it is also the wrong one. The issue does not lie in the state of technological development, but rather in the way that human beings reason, and how we create and direct machines to act as our proxies. This proposition can be defended in three ways—computers are always programmed from a cultural perspective that is not universal; within any given cultural system, there are not simply unanticipated challenges, but actually conflicts of values that we might never program away if we cannot overcome them ourselves; and eventu-ally, over time, systems that we create from partic-ular cultural perspectives will very often operate in inter-cultural contexts,4

further complicating their relevance, legitimacy and effectiveness, and their ethical foundation for action. Put simply, the “gaps” are not merely on the machine side—the problems with effective autonomy are also ours. And if that’s true, no level of technological development will solve for what is inherently a challenge in the human and social condition. In other words, the agenda to im-prove sociotechnical systems is necessary because of gaps and errors in us.

The philosopher and historian of ideas Isaiah Berlin argued for a theory of metaethics called value

pluralism. According to Berlin (as quoted from the Stanford Encyclopedia of Philosophy),

“Genuine values are many, and may—and often do—come into conflict with one another. When two or more values clash, it does not mean that one or another has been misunderstood; nor can it be said, a priori, that any one value is always more important than another. Liberty can con-flict with equality or with public order; mercy with justice; love with impartiality and fairness; social and moral commitment with the disinter-ested pursuit of truth or beauty (the latter two values, contra Keats, may themselves be incom-patible); knowledge with happiness; spontaneity and free-spiritedness with dependability and re-sponsibility. Conflicts of values are ‘an intrinsic, irremovable part of human life’—the idea of total human fulfillment is a chimera. ‘These collisions of values are of the essence of what they are and what we are’—a world in which such con-flicts are resolved is not the world we know or understand.”5

These conflicts of values are active within socio-cul-tural systems. However, they can also exist across socio-cultural systems, and even a cursory compar-ison of Liberalism with Nazism, modern Jihadism, or Stalinist Communism should prove to even the casual observer that the world has a variety of value systems that cohere into ways of life that both differ and conflict.

What value-pluralism and inter-cultural commu-nication challenges underscore for systems design is that the act of design is always positional, relational, and grounded in a particular socio-cultural and dis-cursive system that invariably will reach its mean-ingful limits in terms of universality and therefore applicability. If we take that notion of meaningful limits seriously, it means that any sort of handover mentality to automation on matters of moral or social importance is questionable at best.

(18)

If this argument holds, it means that the “rich coupling” Behymer and Flach say we need to allow for “coordination of multiple actions to achieve a common goal” should not only be seen as a means for increasing communication and control, but as a necessity for any sophisticated system facing com-plex problems (of a type) because those very systems will need a means to negotiate the virtue of possible actions—virtues that may conflict and even cause harm—to achieve a measure of satisfaction, however unsatisfactory it may ultimately prove to be.

1 Kyle Behymer and John M. Flach, “From Autonomous Systems to Sociotechnical Systems: Designing Effective Collaborations,” She Ji: The Journal of Design, Economics and Innovation 2, no. 2 (2016): 113–14.

2 Ibid., 113. 3 Ibid.

4 This is certainly true in the case of autonomous lethal weapons, currently.

5 Joshua Cherniss and Henry Hardy, “Isaiah Berlin,” The Stanford Encyclopedia of Philosophy (Fall 2016 Edition), ed. Edward N.

=DOWDODVWPRGLÀHG6HSWHPEHUhttp://plato.stanford.edu/ archives/fall2016/entries/berlin/.

Distinguishing

Between Control and

Collaboration—and

Communication and

Conversation

Hugh Dubberly, Dubberly Design Office, USA Paul Pangaro, College for Creative Studies, USA

[email protected]

doi:10.1016/j.sheji.2016.12.002

In their paper “From Autonomous Systems to Soci-otechnical Systems: Designing Effective Collabora-tions,” Kyle J. Behymer and John M. Flach remind us “the goal of design is a seamless integration of human and technological capabilities into a well-functioning sociotechnical system.” 1 Recent trends—the sensor revolution, big data, machine learning, and intelligent agents, for example—make their reminder timely.

However, the idea of “seamless integration” has a history in design discourse and discourse about

computing. Architect Christopher Alexander made “fit” the organizing concept of his first book.2_HCI pioneer Douglas C. Engelbart focused his life’s work on “augmenting human intellect,” which he described as “increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems … in an integrated domain where hunches, cut-and-try, intangibles, and the human ‘feel for a situation’ usefully co-exist with powerful concepts, streamlined terminology and notation, sophisticated methods, and high-powered electronic aids.” 3 Computer pioneer J.C.R. Licklider wrote about “man-computer symbiosis,” which he described as “cooperative interaction between men and electronic computers.” 4_{And architect Nicholas Negroponte} explored the possibility of building machines that could collaborate with designers, stating that “the partnership is not one of master and slave but rather of two associates that have a potential and a desire for self-improvement.” 5

Negroponte made a critical distinction—a master “controlling” a slave differs substantially from one colleague “collaborating” with another. The first is ex-ploitative—the second is generative. Historically, most discussions about man-machine interfaces have been framed around control loops, treating machines as slaves. What’s fascinating is that more than 50 years ago, Licklider and Engelbart envisioned an alternate, more humane relationship, a collaboration between man and machine.

Behymer and Flach build on the idea of collab-oration, proposing a model comprised of actors or “agents,” both human and “automaton.”6_{In this} model, each agent is part of a simple control loop with sensor (“Perception”) and actuator (“Control”) flowing through a “Wicked Problem Domain.” This simple control loop is a useful approximation of designing, first introduced in the mid-1960s.7 Behymer and Flach add multiple actors and communications between them. Behymer and Flach also note that the quality of the communication between the actors de-termines the quality of perception and control of the entire system—with “rich” communication, the whole can be more effective than any of the parts.

(19)

“Wicked Problem Domain” in the first figure. That way, we might avoid suggesting that the systems given as examples in the article interact with “wicked problem domains.”

A model that treats wicked problems must grapple with their “wickedness.” Horst Rittel taught us that wicked problems “are inherently different from the problems that scientists and perhaps some classes of engineers deal with … which are definable and separable and may have solutions that are find-able…. [Wicked problems] are ill-defined; and they rely upon elusive political judgment for resolution.” 8

Second, the model does not include a “goal,” a key component of any control system model. Goals might be presumed inherent in the actors (human or automaton); however, the question arises: Where do their goals come from? The goals could be taken as given for simple problems (as in student assignments or entry level jobs). Yet, most professional work in-volves agreeing on goals. And what makes wicked problems intractable is the great difficulty of agreeing on goals (i.e., the problem framing).9

Third, the model does not say much about the nature of communication between the actors, except that it should be “rich” and “effective.” How do we achieve that?

Negroponte points to “conversation,” having included a section by cybernetician Gordon Pask on Conversation Theory in his book Soft Architecture Ma-chines, which describes “(1) the computer as a designer, (2) the computer as a partner to the novice with a self-interest, and (3) the computer as a physical envi-ronment that knows me.” 10

Pask’s model of “conversation” is worth distin-guishing from Claude Shannon’s model of “commu-nication.” Shannon described a process of sending sig-nals. Pask describes higher-level processes, whereby learning systems (including people) make distinc-tions, share and understand them, agree that they understand, and then act on their agreement. His model further distinguishes between conversations about goals and those about means, and it might be expanded to conversations about creating new lan-guage and new processes needed to deal with new challenges (or disturbances) that require “innovation.”

Fourth, while the paper elsewhere introduces cybernetician Ross Ashby’s very useful concept of “requisite variety”,11_{the model misses an opportunity} to connect to the idea. Variety is a property of the system’s sensors and actuators—and also a property of the environment.

Variety refers to the capacity of a system to main-tain itself in the face of disturbances. All systems

have limitations—that is, even the largest and most robust system can be overwhelmed, given a larger and more robust disturbance. Requisite variety refers to the capacity required to overcome disturbances the system is “likely” to encounter. When an automated system is overwhelmed, human operators must come to its aid. We might say that the automated system lacked variety, and the humans increased the variety of the combined system. Deciding how much variety to include is a design decision. Are we designing for a hundred-year flood or a magnitude-8 earthquake? We weigh the likelihood of the disturbance against the cost of including the variety required to resist it.

We may apply the concept of requisite variety to higher-order learning systems—teams, AIs, and socio-technical systems—while noting that Ashby originally developed it to describe first order systems. Behymer and Flach’s essential argument is that neither tech-nology systems alone nor human systems alone have as much variety as both systems might have together, particularly if they are coordinated well. This is the argument for diversity on teams. And while it brings benefits, variety also has costs. As the number of team members grows, so does the complexity of their lan-guage, the potential for miscommunication, and thus the need to design conversations.

1 Kyle Behymer and John M. Flach, “From Autonomous Systems to Sociotechnical Systems: Designing Effective Collaborations,” She Ji: The Journal of Design, Economics and Innovation 2, no. 2 (2016): 113–14.

2 Christopher Alexander, Notes on the Synthesis of Form (Cambridge, MA: Harvard University Press, 1964).

3 Douglas C. Englebart, “Augmenting Human Intellect: A Concep-tual Framework,” Stanford Research Institute Summary Report AFOSR-3223 (Menlo Park: SRI, 1962), accessed October 16, 2016, http://www.dougengelbart.org/pubs/augment-3906.html.

4 Joseph C. R. Licklider, “Man-Computer Symbiosis,” IRE Transactions on Human Factors in Electronics HFE-1, no. 1 (1960): 4.

5 Nicholas Negroponte, preface to The Architecture Machine (Cam-bridge, MA: MIT Press, 1970).

6 Behymer and Flach, “Autonomous to Sociotechnical,” 105–14.

7 See for example John Chris Jones, Design Methods, 2nd ed. (New York: John Wiley and Sons, 1992); Horst W. J. Rittel, “The Universe of Design: Faculty Seminar,” College of Environmental Design, Institute of Urban and Regional Development, University of California at Berkeley, 1964; and Don Koberg and Jim Bagnall, The Universal Traveler: A Soft-Systems Guide to Creativity, Problem-Solving, and the Process of Reaching Goals (Los Altos, CA: W. Kaufmann Inc., 1972).

8 Horst W. J. Rittel and Melvin M. Webber, “Dilemmas in a General Theory of Planning,” Policy Sciences 4, no. 2 (1973): 160.

9 Peter G. Rowe, Design Thinking (Cambridge, MA: MIT Press, 1987).

10 Nicholas Negroponte, author’s note to Soft Architecture Machines

(Cambridge, MA: MIT Press, 1975).

(20)

Ashby and Alexander, who relied heavily on Ashby’s Introduction to Cybernetics in his Notes on the Synthesis of Form.

Collaboration and

Sociotechnical

Systems—A Question of

Context and Framing

Susu Nousala, Tongji University, China

[email protected]

dor:10.1016/j.sheji.2016.12.003

In the paper “From Autonomous Systems to Soci-otechnical Systems: Designing Effective Collabo-rations,” Kyle Behymer and John Flach discuss the challenge of finding better solutions for human and technical interaction. The central issue is a collab-orative balance between the technical systems and human actors in sociotechnical systems.

While these are timely issues, it is important to position the arguments. There is a gap between the introduction and the discussion of human and technical interaction in the article. Let’s compare the opening with the subsequent discussion.

In the introduction, the authors describe “the context of sociotechnical systems where multiple agents must collaborate to solve complex or wicked problems.” 1_{This statement is important. It implies} that there are solutions to complex or wicked prob-lems. However, the term “complex” should be defined. It holds different meanings in different contexts.

For example, consider the phrase “complex and wicked problems.” To describe complex or wicked problems might imply that these terms are interchangeable. To describe complex and_wicked problems might suggest differences that may be significant when considering human and machine interactions. Beyond this, the authors seem to assume that it is in fact possible to solve complex or wicked problems.

The term “complex” may have any of several meanings. Merriam-Webster’s defines the relevant aspects of the noun as “a whole made up of compli-cated or interrelated parts,” or “a group of obviously related units of which the degree and nature of the re-lationship is imperfectly known.” The adjective means

“composed of two or more parts,” “formed by union of simpler substances,” or “having many interrelated parts, patterns, or elements that are hard to separate, analyze, or solve.” 2_{In terms of the kinds of design} issues that Behymer and Flach describe, complexity also involves complex adaptive systems—and a range of issues too large for a short comment.

In colloquial terms, the word complex may simply mean “intricate or complicated,” and therefore hard to understand. In the context of solutions, this leaves a significant question unresolved. Are the au-thors describing wicked problems as complex in one of these technical meanings, or complex because they are hard to understand?

In their 1973 paper titled “Dilemmas in a Gen-eral Theory of Planning,” Rittel and Webber offer a definition of wicked problems—they “are ill-defined; and they rely upon elusive political judgment for resolution. (Not ‘solution.’ Social problems are never solved. At best they are only re-solved—over and over again.)” 3In this context, it is not possible to solve problems in a technical sense, because “social problems cannot be meaningfully correct or false.” 4 For Rittel and Webber, there are no immediate or ultimate tests for solutions to wicked problems. The kinds of problems that they describe typically involve one-off operations. It is impossible to know the out-come in advance. It is often difficult to know in retro-spect whether the solution works. In their introduc-tion, Behymer and Flach discuss finding solutions for complex or wicked problems. They should be clearer.

Bounded rationality limits human problem solving. Given these limits, many solutions involve what Herbert Simon describes as satisficing.5_{As we} all know, humans aren’t always rational. Reinhard Blutner states that “boundedly rational agents experi-ence limits in formulating and solving complex prob-lems and in processing (receiving, storing, retrieving, transmitting) information.” 6 We need these to iden-tify the boundaries of rationality and collaborative relationships. Even so, we always experience limits.

Herbert Simon, paraphrasing Chester Barnard,7 said “the decisions that an individual makes as a member of an organization are quite distinct from his personal decisions.” 8