Stay Informed:
Baskin Engineering COVID-19 Information and Resources
Campus Roadmap to Recovery
Zoom Links: Zoom Help | Teaching with Zoom | Zoom Quick Guide

Computer game design professor Michael Mateas discusses AI

Computer game design professor Michael Mateas discusses AI
Computer game design professor Michael Mateas discusses AI
Tuesday, January 9, 2007

Artificial intelligence (AI) occupies roughly the same scientific realm in the popular imagination as genetic engineering and space exploration: we can't help but prepare for it to backfire spectacularly. But, as is often the case with scientists and intellectuals who understand these advances on a very deep level, Michael Mateas, the new professor of computer game design at UC Santa Cruz, doesn't think we have anything to fear just yet.

'Today's AI is pretty dumb by science fiction standards,' he says, and he would know, because he's done as much as he humanly can to make AI smarter for his entire adult life. He brings with him the credentials of a maverick in the field, arriving from Georgia Tech, where he founded the Experimental Game Lab and participated in several pioneering collaborations, including the groundbreaking game Fa'e.

'The real ethical issues that exist with AI now relate to data mining and surveillance technologies,' Mateas says. 'It's not like there's going to be a super-smart Terminator running around trying to kill you. What you've got are these not-so-smart, but still-sophisticated computer programs that can pick out patterns about your life. I think there are major privacy concerns about that. Someone could take the full database of every phone call made in the U.S. and use it to look for terrorist cells.' Even the best artificial intelligence can be undermined by, shall we say, a lack of presidential intelligence, but in an odd way that's a comfort.

'Anyone who's actually worked in AI, and not just theorized about it, comes away with a respect for how hard it is, and how far we are still,' Mateas says. He sees his move to UCSC as a bit of a West Coast homecoming'he grew up in Carson City, and got his first college degrees in Stockton and Portland, Ore.'as well as an opportunity. 'Since UCSC was founded to be the interdisciplinary UC campus, it has a history of combining different disciplines and exploring the edges, which is very much what computer game designing requires.

'It's a total art,' he continues. 'People used to say that about opera, that it combined so many different disciplines. Games are that for today, combining almost every kind of art form as well as extremely complicated technical expertise.'

In his push to program computers with something more than digital intelligence'after all, drama requires more emotion than logic'he might just break through this era's one great, frustrated, unrequited relationship: We love our computers, but can they learn to love us back'

Scientists and artists call the crossing point between fiction and reality 'the uncanny valley.' It is so named because of a well-known graph, made by Masahiro Mori in 1970, charting human acceptance of increasingly lifelike human facsimiles'people react favorably to cartoons, and are obviously OK with other real people, but report sharp feelings of disturbance when the representation is simultaneously live and dead, flesh and mechanics. The uncanny valley is the root of our dread of zombies and cyborgs, as well as the reason some animated films like Final Fantasy and The Polar Express seem inordinately creepy.

Computer games are constantly doing battle with this uncanny valley, especially with the current crop of processors allowing modern computers and new game systems like the Playstation 3 and Nintendo Wii to render hyper-realistic graphics. Tiger Woods, nearly a machine in real life, becomes positively Terminator-like in the latest PGA Tour game from EA Sports, with everything from the fist pump to the smirk rendered to mathematical perfection. Unfortunately, this Tiger is incapable of a 19th hole press conference. While graphics have evolved to near-realism, conversations with the characters aren't much more technologically advanced than Pong. Today's computer games are all beauty, no brains.

Mateas is one of the very few game designers working on the brains, and he approaches the problem not from the angle of making Tiger Woods speak fluently, but from the angle of making Tiger Woods actually feel the joy mimed by that programmed fist pump. Not that Mateas is interested in making golf games. He's on to bigger things.

'I want to make games that are not about princesses and monsters, but about people,' he says. 'The goal with interactive drama is to have games be about all the things cinema can be about. Graphics and simulation alone don't cut it. You need to start biting off serious AI to express just one little bit of what it means to be human.'

He considers AI code to be a potential medium. 'Novelists work with words,' he says. 'Directors work with shots and editing and the language of film. AI is an expressive language, too. You don't have to solve all of AI to work with it. That would be the equivalent of telling a painter to solve the general theory of painting before putting a brush to canvas. Certainly we're nowhere near the intelligence of the human brain, but still I'm writing, creating little universes of human meaning with AI code. Each program is a little theory you can play and experience. That approach leaves you feeling much more open-ended about things.'

It's this shift in thinking that leads to a Mateas game being a completely different beast from, say, a Will Wright game. Wright is the creator of the ubiquitous Sim games'SimCity, SimEarth, The Sims (the bestselling game of all time), and the highly anticipated Spore. He's also the gaming industry's recognized leader in creating AI-based gaming, as opposed to action-based or puzzle-based games. However, Wright's AI is really just a blend of action and puzzle, a closed simulation with complex but clear rules that encourage improvisation only within the narrow bounds of what choices he's built in. They are staggering achievements, but represent the intelligence of the creator more than the creation. If you spend countless hours with The Sims, you quickly discover that you're not playing with people, or even dolls, but with little problems, each with a little solution. Mateas, however, embraces something wider.

The Art of Code

One of the goals Mateas has for his time at UCSC is building a bridge between the computer sciences discipline (which has not just its roots but also its classrooms in the engineering building) and the arts. 'I'm interested in theorizing code, and exploring what code means, the aesthetics of it,' he says. 'The funny thing about code, if you think of it as a kind of writing, is that it simultaneously means something to human beings, who read it and interpret it, and the machine, which executes it. Strictly speaking, those two meanings are not identical. Obviously they're related, but there's all sorts of tension between the human reading of the code and the machine reading of the code.'

Mateas has the air of the wild dramaturge about him, as if his computer coding geekery were subsumed by a larger urge to portray the drunken porter in the next Shakespeare Santa Cruz production of 'Macbeth.' When describing the problem of creating artificially intelligent characters, Mateas suddenly begins to mimic the smooth but limited range of arm and head motions we associate with cyborgs; his intense wintergreen eyes become fixed and dead; his normally rapid-fire voice slows to a rhythmically strict cadence. 'All these little details,' he says, suddenly snapping free of his robot self, 'when you're working with actors, you get them for free, because they're human. But when you're working with artificial actors, all of that has to be put in.'

Computers only react to what is put in'that's where the art of AI comes into play. Mateas says the decision to forgo any input menus in Fa'e, while creating a monstrous challenge for him and Stern, was essential to the project because it preserved the art, the improvisational acting, of the game. When you present a menu, it automatically shapes the player's thinking,' he argues. He references the ambitious role-playing game Fable, which has an enormous menu of possible actions intended to make the playing experience more open-ended. 'Some of the actions were put there for fun. Like fart. You could push a button, and your character would fart, and everyone around would hold their nose.' The problem, he says, is that some players, when stuck in a particular situation, would begin to wonder if they were supposed to fart.

Mateas sees the avoidance of the menu, the broadening of acceptable input, as a fertile field for creativity. For example, he co-created a robotic sculpture called Office Plant #1, which sits on a desk and responds to the ambient light and sound of the room'and the level of e-mail activity on the worker's computer'by means of slow movements and low-level sounds.

More recently, he created a program called Terminal Time. In academic speak, it 'constructs ideologically biased documentary histories in response to audience feedback.' In practice, it involves a group of people gathering in a theater to watch a Ken Burns'style documentary about a given topic. The program edits, narrates and assembles the documentary on the fly, pausing every so often to ask the audience a question. However, the questions have nothing to do with the content on screen. A typical question goes, 'Which of these things best describes you' A) Things were better in the time of my ancestors. B) Things are good and they keep getting better every day.' The questions get tougher and tougher as the documentary responds by becoming more and more biased. 'Is it okay for two men to kiss in public' A) Yes. B) No.'

'Let's say you think no,' Mateas says. 'Are you going to hoot and holler about that publicly, in the midst of a highly-biased documentary'' It's the AI art experiment equivalent of the rodeo scene from Borat, and it carries similar questions about the nature of group thinking, biased situations, and our susceptibility to provocation. Except in this case, Borat is a computer.

Fa'e and Substance

If you happen to have a computer robust enough'and we're talking top-of-the-line, just-out-of-the-box, 1.6GHz processor robust'you can download and play Fa'e, the game Mateas and Stern developed to demonstrate their theory of interactive drama at work. (The game is available for free at In Fa'e, you assume a first-person perspective not unlike Doom, but instead of being armed to the teeth in order to explode aliens and stuff, you go to a dinner party hosted by Trip and Grace, two friends of yours. They are in the midst of a tense time in their relationship. Without menus, screens, or instructions, you walk through the door, and the characters begin talking to you. You can click stuff, or you can type things to say. When you speak, Trip and Grace respond, not just with their voices, but with appropriate facial expressions and emotions, in context. When I hug Grace, is that a flash of annoyance that crosses Trip's face' When I fail to respond enthusiastically enough to the apartment's redecoration, does Grace give me the cold shoulder'

In 20 to 30 minutes, a drama unfolds that completely incorporates your words and prompts, with the end varying wildly depending on what you do. I spent one session trying to get a bottle of wine open the whole time, and the party was predictably dull. Another session, I confessed my crush on Grace, and the sparks flew from there. It's kind of like life.

It's not impossible to 'break' Fa'e. Introducing a wild new plot element''My dog just got run over outside''will not elicit an intelligent response. However, the story never feels scripted, nor like pushing buttons on a choose-your-own-adventure. Despite the paper doll graphics'which Mateas says were intentionally low-fi in order to both save processor speed and avoid the uncanny valley'Trip and Grace are 10 times more realistic, personality-wise than Tiger Woods.

While the breakthrough is perhaps subtle for the player, it's tremendous for the programmer. Anyone can go out and record a bunch of phrases for semi-intelligent playback. But there's Madden NFL'in which the announcers quickly become repetitive and occasionally blurt out nonsensical color comments (OK, so that's actually pretty lifelike)'and then there's Fa'e.

The characters were physically programmed from the ground up, to move and express in a human-like manner. This aspect is the one road that's been somewhat traveled by computer game programmers, though Fa'e takes a decidedly inside-out approach'having the characters move based almost solely on internal decisions, rather than outside events or forces.

The entire Fa'e universe is overseen by a deity, called the Drama Manager, which provides oversight and gooses the plot with the release of important information at key points, massaging the improvised story to a natural climax and denouement. The fluid mind of the programmed god is not apparent to the player'at no point does the game halt and a screen come up saying 'now everyone becomes aware that Grace is pregnant''but the effect is much like the cassette tape in a How to Host a Murder Mystery party game, deciding when and how to mete out the information that creates dramatic tension. 'In most games, the drama is handled modularly, with some sort of cut scene that interrupts the game when you defeat a monster,' Mateas says. 'We went for something much more organic. It doesn't railroad a player down an arc, but builds an arc around the player.'

Finally, the characters and the Drama Manager'in essence, the entire program'has to understand human language. And not just standard, complete-sentence, third-grade book report language. Fa'e knows what to do with the phrase, 'Dude, you're totally bumming me out right now.'

The trick was to stop trying to teach computers to read like humans'left to right, merging word order with syntax and grammar'and start teaching computers how humans think about language. First, Mateas and Stern took the menus away and put '20 or 30 discourse acts' behind the scenes, so the program is able to categorize a situation as flirting, or arguing, or making small talk. This is the context that's never put on the page, the between-the-lines that makes written communication work (as anyone who's had an e-mail's tone misinterpreted knows). The program follows the train of conversation to assign each dialogue input one to four contexts.

With the sentences themselves, the program's approach is to successively comb the sentence for structure. First, it looks for critical words and finds a synonym. Then it looks for pattern clues for where the sentence is directed (like the 'dude' in the above example signals a direct communication as opposed to a philosophical pondering). The program continues to dig in this manner, searching for critical words within successively smaller fragments of the sentence, until it 'gets' the meaning.

'It's like the program is looking for islands of meaning,' Mateas says. 'It tries to chain those islands together, to get the full meaning, but if it doesn't, that's fine. At least it got something out of the sentence.' The program might not figure out exactly what the player said, but it can at least pick out overall disagreement or dissatisfaction, and do its best to work from there. Fa'e is like being on stage with a highly-skilled actor who only knows a smattering of English. The show still goes on, and sometimes, you're hardly even aware that there's a foreigner on the stage.

Of course, that foreigner is speaking in phonetically-learned phrases. Instead of making Trip and Grace cobble together sentences like a voice mail menu narrator, Mateas and Stern made their actors record hundreds and hundreds of potential sentences, and let the computer make the decisions when to play them back. 'Imagine being a writer who's given a big pool of sentences, and you can't deviate from those sentences, but you can string them together in whatever combination you need in order to tell a story. There's still authorial talent involved.' What Mateas describes is eerily similar to the story of how Dr. Seuss wrote 'The Cat in the Hat''the publishers gave him a list of words kids ought to know, and it was up to him to make a story. It just took an artist to transcend Dick and Jane and create the enduring tale of an anarchist feline.

When attempting to describe the underlying structure for all of this, Mateas began speaking of decision clusters, which appeared in my mind as foamy white stardust clouds in an inky blackness, galaxies of meaning the computer would telescope toward when looking for answers, and the vertiginous similarities to the search for intelligent life on other planets made me dizzy. Is Mateas an inventor, or an explorer' He's certainly more than a mere programmer (or, to be more accurate, mere programming is more than we give it credit for), but I lean toward explorer.

Someone who works in the medium of AI, attempting to apply it toward interactive drama, must have a keen understanding of linguistics, storytelling, human emotional psychology ' the very topics philosophers have struggled with since the beginning of recorded time. While it's tempting, on the surface, to lump a computer game design degree in with such pleasure-oriented disciplines as beer making and pornography studies, in the hands of Mateas, the goals and results of the program will be much loftier.