106 Computer
E N T E R T A I N M E N T C O M P U T I N G
A
common limitation of con- ventional video games is that players quickly learn the positions and behavior of computer-controlled charac- ters, which usually take the form of monsters. Software developers pre- program these characteristics so, after playing the game several times, the player comes to know exactly how and when the monsters will act. The game eventually becomes boring because the player need only execute a learned script to defeat the monsters and overcome all the hurdles he or she faces.This pattern has sparked increasing interest in using computational intelli- gence techniques to control the actions of computer characters, rather than relying on simple heuristics or rule- based systems. In particular, evolu- tionary computation and neural networks are being adapted to let soft- ware characters learn from their own experience, predict what a player might do next, and take appropriate action to meet their own challenges. In this way, a game can remain perpetu- ally novel, posing new tests for the player each time he or she plays.
TRAINING BLINDFOLDED
In pursuit of this goal, game devel- opers can combine evolutionary com- putation and neural networks to let
software agents learn appropriate behaviors even in complex strategy games, without resorting to prepro- grammed features provided by soft- ware engineers a priori. The evolu- tionary process can even invent its own features for describing situations in a game and learn to weight those fea- tures and act based on patterns it rec- ognizes.
To place this capability in context, consider the following thought exper- iment. Suppose you sit down at a table to play a game with an opponent.
You’ve never played a game of any type before. You face a two-dimensional board with eight squares on each side, with alternating colors, and, as Figure 1 shows, pieces occupy every other square. You’re told the rules: The game takes place in turns, red moves first, and pieces move forward one square at a time, diagonally. Further, pieces that reach the back row become kings, a status that lets them move forward or backward one square at a time. Finally,
when the possibility of jumping over an opponent’s piece becomes available, you must take that jump move and remove the opponent’s piece from play.
What you aren’t told is the object of the game. Instead, your opponent gives you the first move and responds in kind. After some number of moves, you’re told that the game has ended and you’re challenged to play again.
Naturally, you’d like to know the out- come of the first game before playing a second one. This information, how- ever, is withheld from you until you fin- ish a random number of games. Only then do you learn that you earned, say, eight points. Eight points is better than
seven and not as good as nine. You aren’t told, however, which games gar- nered you points and which were less successful.
Given these parameters, consider how many games it would take you to learn to play at the level of human experts—people who know the object of the game, can study and read books about it, converse with other experts, and hone their skills through practice.
EVOLVING A CHECKERS EXPERT The game in this case is checkers, and this is essentially the task that Kumar Chellapilla and I gave to an evolutionary algorithm that uses arti- ficial neural networks as checkerboard evaluators. I describe the experimental design and its implications in depth in my book, Blondie24: Playing at the Edge of AI (Morgan Kaufmann, 2002).
We presented only the raw informa- tion from the checkerboard as inputs to the neural networks: the position,
Evolutionary
Entertainment with Intelligent Agents
David B. Fogel, Natural Selection Inc.
When she’s not busy surfing
or skiing, Blondie24 plays a
mean game of checkers.
type, and number of pieces. The neural networks operated on these data, pro- cessing them through two hidden lay- ers of 40 and 10 nodes, respectively, to ultimately derive an output value for any board that would range between –1 and +1. Higher numbers indicated more favorable positions.
In the evolutionary algorithm’s first generation, we set all the neural net- works’ connection weights at random values. Each neural network played five games as red against a random col- lection of other neural networks in the initial population. As a result, the games proceeded at first with random moves, with minimax pruning of nodes and branches to provide a rationale for choosing which move works best in a particular setting when evaluating some number of moves into the future.
Even with random initial moves, some moves proved better than others.
The neural networks could earn or lose points in each match: They gained one point for a win, lost two points for a loss, and received no points for a draw.
We withheld this specific information from them, however, and used only each neural network’s overall score gained after all neural networks com- pleted their games to evaluate their rel- ative performance.
Culling the weak
Based on their scores, we killed off the lower half of the neural network population and used the upper half as the basis for generating offspring in the next generation. We kept variation to a very simple mutation of each of the weights of every parent neural net- work, along with a mutation of the neural network’s assessment of the kings’ importance. Each parent gener- ated a single offspring, and we refrained from using recombination operators in this experiment.
Nevertheless, after 10 generations the best-evolved neural network could defeat both Kumar and me, its creators.
Subsequent efforts extended this basic paradigm to include a prepro- cessing layer of nodes in the neural net-
works, with each node being devoted to a spatial region of the checkerboard.
These regions covered all 3 ×3, 4 ×4, and larger subsections in an effort to allow the neural networks to learn about the game’s spatial characteristics.
Head-to-head with humans After the evolutionary process had undergone 840 generations of self- play, we tested the best-evolved neural network against real people online at www.zone.com. None of the human opponents knew they were playing against a program.
Players at the www.zone.com game and entertainment site receive rating points corresponding to the system adopted by the United States Chess Federation, in which players earn points based on the results of out- comes against other players. A player earns more points for defeating play- ers with a higher rating and loses more points when defeated by players of lesser presumed strength. The point system defines a nomenclature of
• grand masterfor anyone with a point score above 2,400;
• masterfor those in the range of 2,200 to 2,400 points;
• expertfor those in the range of 2,000 to 2,200 points; and
• successive class levels of A, B, C, and so on in descending intervals of 200 points.
When you log into www.zone.com, you must adopt a screen name. Our initial choices of David1101 and Kumar1202 generated only minimal interest from other players. Changing our screen name to Obi_WanTheJedi generated significantly more interest, but also instances of less than sports- manlike behavior from opponents who lost to the neural network and swore at us in the chatbox provided for online communication.
An AI with personality …
After suffering much berating, we adopted a new screen name: Blondie24.
Next, we invented her personality, mak- ing her an attractive, single, 24-year-old UC San Diego math major who surfs and skis. Using the chatbox, we explained that she became really good at checkers by teaching herself how to play during the six months it took her to recover from breaking her leg in a ski- ing accident.
This information comprised a ver- sion of the truth because the program had in fact run for nearly six months on a Pentium II 400-MHz computer as it completed the 840 generations of its evolution. The players on the Internet site greeted this new personality much more favorably. My book contains many detailed stories of these interac- tions.
Currently, Blondie24 generates moves in the software product Evo- lutionary Checkers Starring Blondie24, published by Digenetics. Figure 2 pro- vides a screenshot of the game inter- face, which offers simultaneous two- and three-dimensional views of the board in conjunction with live-motion video of an opponent framed by a real- istic background. This background can be animated and the video segments Figure 1. The initial board position in checkers. Pieces move forward diagonally, one square at a time. Upon reaching the back row, the pieces become “kings” and can move forward or backward one square at a time. Jump moves over the opponent’s pieces are compulsory, and the jumped pieces are removed from play.
June 2003 107
1 2 3 4
9 10 11 12
5 6 7 8
17 18 19 20
13 14 15 16
25 26 27 28
29 30 31 32
21 22 23 23
Red
White
by Competing Against Chinook:
Experiments in Co-Evolving a Neural Checkers Player,” Neurocomputing, vol. 42, 2002, pp. 69-86).
Many years ago, at an early stage of research into artificial intelligence, such an achievement appeared impossible.
The famous computer science pioneer Allen Newell once said, “It is extremely doubtful whether there is enough information in ‘win, lose, or draw’ when referred to the whole play of the game [such as checkers] to per- mit any learning at all over available time scales” (Marvin Minsky, “Steps Toward Artificial Intelligence,”
Computers and Thought, McGraw- Hill, 1963).
We present an even higher hurdle here because we provided no specific win, loss, or draw information—only a final point tally to each neural net- work over a series of games. Nor were the neural networks afforded the lux- ury of any human expertise beyond the position, number, and type of pieces on the board. The neural networks had to invent any other features they would use on their own, through the process of evolution.
T
he evolutionary technology we have developed can be used to control characters in intelligent board games and other software enter- tainment products. Efforts are under way to incorporate self-learning tech- niques in other games, such as chess, and may provide superior methods for designing continually novel games in the future. ■David B. Fogelis the CEO of Natural Selection Inc. Contact him at dfogel@
natural-selection.com.
108 Computer
E n t e r t a i n m e n t C o m p u t i n g
Figure 2. A screenshot from Evolutionary Checkers Starring Blondie24, which embeds the results of evolutionary computation technology in a virtual reality interface that provides simultaneous two-dimensional and three-dimensional viewing of the playing area, coupled with lifelike continuous-motion video of the opponent.
Editor: Michael Macedonia, Georgia Tech Research Institute, Atlanta; [email protected] blended together, triggered by actions
on the board or even player inaction.
The effect can be quite striking, and some people have asked, after playing a few moves, “Where is she?”
… and daunting expertise
After completing 165 games, each played by manually entering opponent moves into the neural network and again entering moves suggested by the neural network to the human oppo- nent on the Internet, Blondie24’s rat-
ing settled in the expert category at 2,045. This placed the program in the top 500 of 120,000 players on www.zone.com. In a subsequent 10- game match, Blondie24 played against the novice version of Chinook, the world-champion checkers program developed by the University of Alberta team led by Jonathan Schaeffer. This match generated evidence corroborat- ing her www.zone.com rating (D.B.
Fogel and Kumar Chellapilla,
“Verifying Anaconda’s Expert Rating