The Art of Human Computer Interface Design,
B. Laurel, editor,
NY: Addison Wesley, 1990
Realness and Interactivity
Legend has it that a gentleman once approached Picasso
on the street and criticized his paintings as distorting reality. Seeming
to change the subject, the artist asked the gentleman if he had a girlfriend.
He did, and produced a small picture of her from his wallet. "She's beautiful,"
replied Picasso, "but she's so tiny."
This paper is about how reality is a funny thing, now
that we are beginning to simulate it, and where the clues may lie in understanding
styles of function and aesthetics about virtual realities.
Virtual Realities Require Realness And Interactivity
Realness and interactivity have various meanings. We
usually use the term "realness" to mean our sense-ability: what our senses
sense. This "sense of presence" is independent of whether the material
is derived from the real world or from a fantasy based one (perhaps "thereness"
is a more appropriate word than "realness"). The term "interactivity"
usually means our effect-ability: what our effectors affect. Though interactivity
always requires information flowing in both directions, it is our input
and its effect that distinguishes it from non-interactivity, where the
flow is always one-way.
The ultimate virtual reality is, by definition, indistinguishable
from reality, where all our effectors are sensed and all our sensors are
affected. This is a rather formidable task. Such models exist in the fantasy
literature: the family media room in Bradbury's Veldt and cyberspace in
Gibson's Neuromancer are popular examples of total immersion in virtual
realities (and most certainly include created worlds as well as real ones).
Other models, theoretical ones, have been proposed from the points of
view of media technology (Negroponte 1981), cybernetic theory (Bricken
1983), and drama (Laurel 1986). The combination of realness with interactivity
is a new frontier where experienced pioneers today are few, and come from
such diverse worlds as flight simulation, dream research, and experimental
Clearly there is an attraction to experiencing the jaw-dropping
impact of a seven story high IMAX movie controlled by a video game-style
steering wheel (which is very much what present-day flight simulators
really do). Or having a conversation with a Disney-style talking head
interacting in a way that dwarfs the ELIZA psychoanalyst program (which
very much doesn't exist today in any form of media). We can expect such
progress on the technological front, but there is a bit of a communication
The sense-ability experts and the effect-ability experts
come from different worlds. And their uses of each others' terms are grossly
The Movie World Understands Realness But Not Interactivity
If you want a mediated experience to look and sound
(and feel and smell) real, go to the motion picture industry. From the
beginning, the trend has been toward better sound, color, scale, depth,
spatial resolution, and temporal resolution. Today we have various "enhanced-format"
theaters at theme parks and worlds' fairs offering "more reality." These
include big movies like IMAX and OMNIMAX (horizontally formatted 70mm
film), panoramic movies like CircleVision (nine 35mm screens arranged
in a cylinder), deep movies like Twin-70mm stereo (two 70mm projectors
and polarized glasses), and smooth movies like Showscan (70mm film shot
and projected at 60 frames per second).
There are other tricks known by the motion picture industry
to achieve even more reality. The use of a concave mirror to achieve infinity
focus is used in today's flight simulators to allow the users' eyes to
focus at infinity, as they normally do when viewing landscapes. Disney
pioneered relief projection, projecting movies on a screen the same shape
as the image, such as the "talking heads" in the Haunted Mansion. He also
pioneered the use of a single room-size half-silvered mirror to produce
3D ghost-like images seen in the Haunted Mansion's ballroom. Lucasfilm
recently introduced the use of a motion platform synchronized with image
and sound in their "Star Tours" ride giving the audience a parallel kinesthetic
sensory input channel.
What has been restricted to the film world has begun
to enter the video world. High definition television, with its various
and opposing formats, has forced some of these parameters to be surveyed
and formalized (Schrieber 1984).
So when people from the motion picture industry say
they can give you a sense of reality, they can, as they say, knock your
socks off. But if they say they can give you an interactive experience,
don't come running.
The world's first interactive movie (as it was billed)
was produced by the Czechs for EXPO '67 in Montreal. The audience had
"yes/no" buttons and every few minutes they were given a choice. This
database of movie scenes was so sparse that after the end, the audience
was asked if they would like to see the other end ("vote yes or no"),
which acknowledged what the experience really was: a clever and perhaps
prophetic gimmick. (Incidently it was more clever and less technological
than the audience may have imagined. Each choice was carefully structured
so that either option would end back at the same next choice, where the
audience vote signalled the projectionist on which of the two synchronized
projectors to put the lens cap.)
Today most of the interactivity known to the motion
picture industry is in the form of interactive videodiscs, since they
share common production hardware and techniques with conventional cinema.
Most interactive videodisc applications are poke-and-see movie snippets
with just slightly more options than the Czech's film. In a Sony advertisement
a few years ago, a Cuisinart kiosk was shown with a monitor and ten pushbuttons
numbered one through ten, presumably allowing the user one of ten short
linear programs. (Sardonically, the words on the screen flashed "PUSH
BUTTON SIX." One can't help but wonder who's controlling who here.)
You can't conduct a symphony orchestra, or carry on
a rich conversation, or drive a car this way.
The Computer World Understands Interactivity But
If you want an interactive mediated experience, go to
the computer world. But since it is younger and less evolved than the
world of cinema, it has addressed interactivity more in the context of
here-and-now practicalities than in the context of future sensory-rich
virtual realities, where theoretical factors such as simultaneity, memory,
and interruptability become essential (Lippman 1985).
All interactive media experiences require user input.
With a little imagination, one can envision human-machine interaction
beyond a keyboard and mouse to the natural and kinesthetic way we encounter
the real world. Input devices for our hands, arms, head, eyes, body, and
feet can sense position, gesture, touch, movement, and balance.
Video games are primitive examples of this sort of interactivity.
Driving games such as "Pole Position" offer user control as tight linked
as real world power steering. Even the first arcade video game, "Pong,"
had an input channel many times the bandwidth of practically any "interactive"
videodisc system today.
But don't ask computer people about a sense of media
reality: their perceptions are warped by current hardware constraints.
Go to any computer graphics film show and listen when
the audience applauds. A few years ago I was at the annual Siggraph film
show, the largest of these events, standing next to a young security guard
in a mobbed colosseum. After a couple minutes a shot of a (fractal) mountain
scene appeared on the screen and the audience applauded. I noticed the
security guard getting tense. Shortly after that, (ray traced) shiney
steel balls were shown hovering over a chessboard. More applause. Then
a (atmospheric) foggy outdoor forest scene appeared and the audience exploded.
At this point the security guard snapped. He grabbed my arm ("something
weird is going on here man") and demanded an explanation.
More recently I attended a demo of a new chip capable
of unlimited realtime movement through a 3D database of limited cartoon-like
quality. A very excited representative of a well-known computer maker
kept exclaiming how real this was, and continued to refer to the little
green cones as "trees."
Who're we kidding?
Bandwidth Is Not Enough
But if virtual reality-making follows moviemaking (or
any other media form), then throwing more bandwidth at the medium doesn't
automatically make it better. Being artful with what you have does.
In early motion pictures, shots were long in duration,
camera movement, if any, was slow, and scenes were edited for perceptual
continuity. Audiences were cautiously led through the virtual space of
cinema, careful not to get "lost." As the language of cinema evolved and
audiences grew more sophisticated, more inferences could be made. Conceptual
continuity replaced perceptual continuity. One might speculate that the
Lucas/Spielberg-style fast-paced thriller would have been impossible for
early movie watchers to have followed. The evolution of aesthetic style
results in massive bandwidth compression.
We have little basis for aesthetic style in virtual
realities yet, and we have much to learn. For starters, we still have
only a meager understanding of the use of scale in media. Nor do we have
more than a hint of a cinematic language of interactivity. We have little
understanding of the roles of acculturation and habituation in media.
And we have no experience at all in creating behavior as well as image.
We do know that these media issues have something to
do with context and its relationship to medium (remember McLuhan), as
we are creating media forms with vast new capabilites. This has created
a particularly critical role for artists today, if we are to believe that
artists of the future will create landscapes that you walk through and
portraits that you converse with.
Bricken, W. (1983) A Model Interface Model. Atari Research
Memo #XX. Sunnyvale: Atari Sunnyvale Research Laboratory.
Laurel, B.K. (1986) Toward The Design Of A Computer-Based
Interactive Fantasy System (PhD dissertation). Department of Theater,
Ohio State University.
Lippman, A. (1985) Imaging and Interactivity. Media
Lab report. Cambridge: Massachusetts Institute of Technology.
Negroponte, N. (1981) Media Room. SID 22, No. 2, 109-113.
Schrieber, W. (1984) Psychophysics and the Improvement
of Television Image Quality. SMPTE 93, No. 8, 717-725.