michael naimark

From VSMM98 Conference Proceedings
4th International Conference on Virtual Systems and Mulitmedia
Gifu, Japan

Invited Paper



Field Cinematography Techniques for Virtual Reality Applications


Interval Research Corporation, 1801-C Page Mill Road , Palo Alto CA 94304 USA



Abstract. "Virtual Reality" (VR) is practically defined by the requirement of 3D computer models and realtime control by the user. While such properties afford interactive navigation and manipulation, the imagery is relatively simplistic or cartoon-like. Cinema is the opposite, with little or no interactivity but with "photo-realistic" imagery. Most of the efforts around VR have originated from the computer culture. This paper describes complimentary efforts from the cinema culture, including techniques, case studies, challenges, and social implications.


  1. Virtual Reality and Cinema

1. "Be Now Here" in Creative Time's "Art in the Anchorage" exhibition, New York, 1997.

(photo: T. Westenberger)

1.1 Aesthetics of Telepresence

Much of the historical work in representation has concentrated on conveying a sense of place. We can trace one strand relating to visual representation from landscape and mural paintings, to the panoramas and cycloramas of the nineteenth century, to the special-venue cinema formats of today such as Imax, Showscan, and CircleVision. These formats exploit high spatial and temporal resolution, wide-angle and surround fields of view, multitrack audio, and 3D stereoscopy. Their goal is to convey a sense of "being there" – that is, of telepresence.

The "there" in most cinema is an actual physical place, since the nature of cameras is to record whatever is in front of the lens (even if that is the contrived environment of a studio). The aesthetics of cinema are biased toward representations of actual places; imaginary places must be created with additional work (i.e., special effects). Furthermore, virtually all cinema is linear and non-interactive; the technology of film makes it difficult to give the audience any control.

Hence, the aesthetics of telepresence from the point of view of cinema are geared to present high sensory realism in images of physical rather than imaginary places, with little or no interactivity.

1.2. Cinema Culture and Computer Culture

The aesthetics surrounding the computer culture has traditionally taken a contrasting point of view: low sensory realism in images of imaginary rather than physical places, with interactivity as a key element.

People often associate computer graphics with simplistic or cartoonlike imagery. Developers must build computer-graphics models from scratch, using drawing and painting tools as well as libraries of primitive shapes and textures. Techniques such as photographic texture mapping help such models to approach photorealism, but the imagery is still far from conventional camera-based images alone. Current work in image-based rendering shows promise but also demonstrates the complex and problematic challenge of making 3D computer models from images.

Also, since such computer models must be built up from nothing, it is generally easier to make imaginary and fantasy places than to model physical environments. Hence, the aesthetics of the computer culture has tended more toward the imaginary and fantasy, whereas the cinema culture – particularly the documentary cinema culture – has tended toward the actual world.

A more subtle difference between the computer and cinema cultures is that a person must spend many hours in front of a workstation to make computer models, whereas filmmakers must interact directly with environments and people.

Many people who work with computers think that interactivity is a critical characteristic, for both navigation and manipulation (e.g., when the user specifies, "move that chair to the right"). Sensory and physical realism is secondary. People who work in cinema often have the opposite priorities [1].


2. Cinematic Techniques for Interaction and Immersion

2.1. Photorealism

Realness, or sensory realness, or photorealism is relative to representation. In theory, we could apply a photorealism Turing test to types of imagery: We would ask viewers whether the representation is indistinguishable from its subject. The problem is that virtually all current forms of visual representation would fail. Even 3D Imax images are obviously still only a movie, compared to reality. The human eye is extremely difficult to fool. (The ear is much more fallible; we've all mistaken a voice on the radio for the voice of a person present in the room.)

Consider the current range of photorealism for dynamic visual representation: web- and MPEG- level video, broadcast-quality video, theater-quality (e.g., 35mm) film, and special-venue (e.g., 70mm and multiple-screen formats) film. Imax is advertised as having 10 times the resolution of standard 35mm film, and several special-venue formats have twice the resolution of standard Imax [2]. To make an ultimate CAVE, with four walls, ceiling, and floor all in Imax-quality stereo, would require 12 times Imax resolution, or 120 times 35mm film. Such a level of photorealism is orders of magnitude greater than low-end formats.

It is also noteworthy that the degree of perceived realness is usually correlated with quality of content. When a presentation is compelling, it seems real. Conversely, higher resolution does not automatically make a presentation more convincing. The relationship between photorealism of form and quality of content is complex.

2.2 Panoramas

Panoramas are generally regarded as wide-field images; often, they represent a complete 360-degree field of view (FOV). A panorama represents a single point of view and is by definition two dimensional. Panoramas allow a viewer to look around (angular movement, i.e., panning and tilting), but not to move around (lateral movement, i.e., dollying and tracking).

We can make panoramic photographs using a single lens (such as a fisheye, in conjunction with a convex mirror, or with a rotating slit mechanism). We can tile together multiple images, but if they are not taken from a single point of view (i.e., the nodal point of the camera), then distortion is inevitable.

Panoramic imagery offers limited navigational interactivity. A viewer can pan and tilt through a panoramic scene, but can neither move laterally nor manipulate the imagery.

2. Le Cinéorama, a 10-screen panoramic film theater by Raoul Grimoin-Sanson, Paris, 1900.

(gravure: La Nature)

2.3 Moviemaps

Moviemaps offer another kind of limited navigational interactivity. They are filmed by stop-frame cameras that move along a path and are triggered by distance from the viewed scene (typically by a sensor attached to a wheel), rather than by time. Distance triggering maintains constant speeds during playback at constant frame rates, which is often not practical or possible during production with conventional (time-triggered) movie cameras. The result is the transfer of speed control from the producer to the viewer, who controls the frame rate through an input device such as a joystick or trackball.

In addition to speed control, limited control of direction is possible if registered turns are filmed at intersections. With match-cutting between a straight sequence and a turn sequence, the user can "turn" from one route to another. The developer must be careful to minimize visual discontinuities, such as sun position and object (e.g., cars and people) transience. The goal is to make the cuts appear seamless.

Moviemaps are "look-up" media, where all possible views are pre-recorded and accessed via computer. Hence, they offer only limited navigability: you can view only images that have been pre-recorded (i.e., you cannot leave the paths). Like panoramas, moviemaps limit interaction to navigation; viewers cannot manipulate the imagery.

2.4 Stereoscopy and Multiple Perspectives

Stereoscopy, the sense of 3D that we get when we perceive a scene through both eyes, requires two unique points of view – one for each eye. People make stereoscopic photographs and cinema using two lenses (and often two cameras) typically separated by the normal human interocular distance. Two separate images must be recorded and kept synchronized from recording to playback to give the viewer the sense of 3D.

In theory, stereoscopy is successful only if the viewer’s head is not allowed to move, because it represents only two points of view. If head motion is allowed, every new perspective encountered must be displayed. Although a system that can provide all these views has been demonstrated in a limited way with pre-recorded imagery [3], creating one is problematic because every possible point of view cannot be filmed. Unlimited navigation is possible with 3D computer models.

2.5 Orthoscopic Displays

Many VR displays combine three important sensory elements for a maximum sense of presence: wide-angle FOV (for immersion), stereoscopy (for 3D), and orthoscopy (for proper scale), often called wide-angle ortho-stereo [4]. These displays fall into two main groups: special-venue film formats (such as Imax Solido, 3D Imax, and Showscan 3D) and VR displays (such as head-mounted displays [HMDs] and CAVEs). The film formats provide ultra-high resolution and group viewing but are not interactive, whereas the HMDs and CAVEs are lower resolution but allow the possibility of multiple perspectives through head-tracking and 3D computer models.




3. Case Studies - "See Banff" and "Be Now Here" [5]

3.1 See Banff

3. Johnston Canyon trail near Banff.

(stereo pair from See Banff - view cross-eyed)


See Banff! is a unique stereoscopic moviemap that grew out of an exploration of field recording for VR [6]. We used two stop-frame 16mm film cameras with wide-angle lenses, mounted for stereoscopy on a "baby jogger" carriage, with an optical encoder attached to one of the wheels to trigger the cameras at programmable distance intervals.

The imagery was recorded entirely in the field, outdoors in the Canadian Rocky Mountain region surrounding Banff, Alberta. As is sometimes done in documentary film, we made no attempt to control lighting or action: The goal was to record the environment as it is.

In addition to recording the beauty of the landscape, documenting the proliferation of tourists was an integral part of the intention. As we worked in the field and interacted with both local residents and tourists, it became apparent that there was lively controversy surrounding tourism and growth; this dialog was part of the experience of being in Banff. Aesthetically, conceptually, and technically, having tourists appear in the foreground and the landscape in the background added a strong sense of depth and presence. Over 100 paths were recorded during a 6-week period.

4. "See Banff!" camera rig and kinetoscope playback system.

(photos: L. Psihoyos and M. Naimark)


The display system for See Banff! mimicked a 100-year-old cinema viewing device: the kinetoscope. Thus, it mimicked the limitations of the old stereoscopic viewing systems by using a stationary eye-hood that prevents a viewer’s free head motion, as well as providing nearly orthoscopic optics. The display also conformed to the limitations of the one-dimensional travel along the paths by providing only a one-dimensional user input device: a crank on the side of the system. The crank employed a force-feedback brake that would freeze at the beginning and end of each sequence. The user selects the sequence to view by manipulating a lever near the eye-hood. The prerecorded material was stored on a single laserdisc using field-sequential stereo and LCD shutter optics.

Hence, the See Banff! kinetoscope provided a broadcast-quality video wide-angle ortho-stereo viewing experience with one-dimensional navigational control for a single user. It could not provide unlimited navigation or any form of manipulation of the imagery.

3.2 Be Now Here

5. Orlando Column in Dubrovnik, covered for protection from bombing.

(stereo pair from Be Now Here - view cross-eyed)

Be Now Here (Welcome to the Neighborhood) is a unique stereoscopic panorama. We used two full-motion 35mm film cameras mounted for stereoscopy on a motorized tripod that rotated at 1 revolution per minute (rpm). To enhance the sense of telepresence, we ran the cameras at 60 frames per second (fps), rather than the standard 24 fps, and employed wide-angle (60-degree horizontal FOV) optics.

The imagery was recorded in public plazas in the four cities designated "In Danger" by the UNESCO World Heritage Centre: Jerusalem, Dubrovnik (Croatia), Timbuktu (Mali), and Angkor (Cambodia). The intention was to record these beautiful and troubled environments.

The production concept was simple: Find in one public plaza in each city a single spot that best represents each place, then film several panoramas from that spot during the course of the day without moving the camera system. The multiple times of day would be perfectly registered and allow seamless intercutting, with only the lighting and transient objects changing.

Due to the potentially hazardous and controversial nature of the project, production was as inexpensive, fast, and quiet as possible, relying on prearranged local staff at each site and help from UNESCO to cross borders with the 500 pounds of film gear. Through a great deal of planning and collaboration, all four sites were filmed in 1 month. ("Highlights" included a bomb scare in Jerusalem resulting in evacuation of the entire plaza, a drive through a strip of Bosnia in the middle of the night during wartime, a negotiation with Taurig camel drivers in Timbuktu about issues of appropriation, and a bribe to get the hired driver out of jail after he had gotten lost after dark and had been found by the Cambodian military [7].) Miraculously, all the footage survived.

Working with local collaborators was a critical element in ensuring the quality of imagery. The selection of the sites was heavily informed by local knowledge. More important, filming in the middle of public plazas is a conspicuous activity, and the fact that local collaborators knew many of the people in the plaza helped to make everyone feel comfortable. Local people, particularly children, didn't appear self-conscious – they simply did what they would normally do in such places.

6. "Be Now Here" camera rig and installation.

(photos: G. Tassé and C. Dohrmann)

The display system for Be Now Here employed a large (12- by 16-foot) front-projection screen capable of maintaining polarity, two video projectors driven by laserdisc players, four-channel surround audio, and a simple input pedestal that allowed a user to choose the location as well as the time of day. The input pedestal was positioned at the orthoscopically correct point for a 60-degree FOV of the screen.

We recreated the sense of camera rotation by rotating the entire floor in sync with the imagery. A 16-foot diameter rotating floor was used as the viewing platform, with the input pedestal in the center. This space was totally dark except for the screen, resulting in a strong visceral illusion: Viewers believed that the screen was rotating around them, rather than that they themselves were rotating. The effect is similar to the feeling of motion that you get when you are sitting in a stationary train in the station and an adjacent train begins to move.

After several public and private screenings, it was apparent that the 1-rpm rotation of the floor was too fast for some people to ignore. Tests suggested that, at 0.5 rpm, almost nobody would feel dizzy, but the illusion of a rotating screen would remain intact. The floor was slowed and the laserdiscs were remastered at one-half the original speed (30 fps). An unintentional result was that all motion in the images – of people, animals, and vehicles – was now in "slow motion," making the representation less real and more abstract, an effect which could be construed as more arty and less techy [8]. Nevertheless, most viewers reported a more compelling immersive experience.

Be Now Here provided a twice-broadcast-quality video wide-angle ortho-stereo viewing experience with limited navigational control (discrete choice of place and time) for group viewing. Like See Banff!, it could not provide unlimited navigation or any form of manipulation of the imagery.


4. The Challenge of Converging Cinema and Computing

4.1 Dimensionalization: Making 2D into 3D

Dimensionalization is making a 3D model from one or more 2D images. Image-based rendering may fulfill a common dream in many VR circles: to wave a camera around an actual place and to end up with a 3D computer model. But, as we are learning, many obstacles prevent us from realizing this fantasy.

Perhaps the most difficult problem is how to resolve occlusions, the "holes" that are left after we after aggregate all 2D images into a 3D model. Simply put: How do we fill in a blank when we have no information? In many classes of imagery, occlusions are inevitable, even with many 2D views, such as of forests, crowds, street scenes, or almost any complex and unstructured environment.

Another question is for what purposes 3D models are necessary. Several different 2D panoramic formats currently exist on the web, including QuickTime VR, PhotoBubbles, and IPIX. Since panoramas represent a view from only one point in space, they are relatively easy to record. As 2D databases, they require much less storage than a comparable 3D database. But they allow only angular, rather than lateral, navigation, and they afford no manipulation. For some applications, panoramas alone may provide sufficient virtual reality.

4.2 Segmentation - Making Non-Semantic "Models" into Semantic Models

Segmentation – adding higher-level or semantic knowledge to an image – can be done by hand or by computer. One particular class of 3D visual databases consists of only points in space, and includes no semantic knowledge by the system. This class includes light-fields and 3D images made with depth-maps. Standard 3D computer models, built from primitives, are semantic models: The system "knows" the contents of the database. Semantic models are required for any kind of interactive manipulation of the imagery. Light-field and depth-map 3D databases are nothing but "clouds" of pixels (whether they are even "models" is debatable) As such, they allow unlimited navigation but no manipulation.

Like full 3D models, semantic models may not be necessary for all applications. Real-world precedences exist for viewers to enjoy navigation without manipulation (through nature trails, ancient ruins, religious temples, and so on).

4.3 Automation or Human Intervention

Making 2D into 3D (dimensionalization) and making non-semantic into semantic models (segmentation) are both possible when there are humans in the loop. Some of the processes have been automated and some will be automated. But it's not clear that all the decisions required should be automated.

Much of the human labor today associated with dimensionalization and segmentation is by default: The work is neither desirable nor enjoyable, but automation doesn't exist. The Hollywood special-effects community relies on such human intervention. Clearly, such processes would be best automated.

There is, however, another class of decisions for which human intervention is desirable, particularly regarding segmentation and semantic modeling. In some cases, determining what are the "most important" elements in a scene is a matter of human expression and of art.

4.4 Immersive Virtual Environments

The two most prominent immersive virtual environments today are created with HMDs or with CAVEs. Both techniques are problematic. HMDs are relatively low resolution and encumbering; CAVEs require several projectors and space. Because of head-tracking, both HMDs and CAVEs are optimized for only one user, even though CAVEs can comfortably accommodate several other viewers.

Before we can have high-quality, ubiquitous immersive virtual environments, we need to overcome several technological hurdles. For example, we need high display resolution and brightness, no viewer encumbrance, and accurate head tracking.

4.5 Cameras of the Future

Although we may be able to do a limited amount of VR conversion from pre-existing images, cameras used for most VR applications today fall into two categories: (1) standard, mass-produced cameras that have been modified, specially mounted, or instrumented; or (2) extremely heavy, expensive contraptions, such as CyberScans, motion-control rigs, or time-of-flight lasers.

No one has designed an inexpensive camera specifically for VR applications. A golden opportunity exists here.


4. Work in the Real World

The state of the world is precariously uneven in terms of resources. Although many may believe that computers will save the world, North American scientists have access to almost 40% of the world's R&D investment, while the entire continent of Africa only has 0.5% [9], and less than 10% of the children of the world will have access to computers and the Internet by the year 2000 [10].

The state of the world is also unimaginably rich in terms of culture. VR can be an important communications medium for world culture, but only if those of us lucky enough to have access to the tools are sensitive enough to work with and learn from local expertise. If not, the loss will ultimately be ours.



[1] M. Naimark, Realness and Interactivity. In: B. Laurel (ed.), The Art of Human Computer Interface Design. ISBN 0-201-51797-3. Addison Wesley, Reading, MA, 1990, pp. 455-459.

[2] M. Naimark, Expo '92 Seville, Presence vol. 1 no. 3 (1992) 364-369.

[3] S. S. Fisher, Viewpoint Dependent Imaging: An Interactive Stereoscopic Display, SPIE vol. 367 (1982) 41-45.

[4] E. M. Howlett, Wide-Angle Orthostereo, SPIE vol. 1256 (1990) 210-223.

[5] M. Naimark, A 3D Moviemap and a 3D Panorama, SPIE vol. 3012 (1997) 297-305. (online at www.interval.com)

[6] M. Naimark, Field Recording Studies. In: M. A. Moser and D. MacLeod (eds.), Immersed in Technology. ISBN 0-262-13314-8. MIT Press, Cambridge, MA, 1996, pp. 299-302. (online at www.interval.com)

[7] M. Naimark, Trip Reports from the Be Now Here production. (see http://www.naimark.net/writing/trips/bnhtrip.html)

[8] M. Naimark, What’s Wrong with this Picture? Presence and Abstraction in the Age of Cyberspace. In: Roy Ascot (ed.), Consciousness Reframed: Art and Consciousness in the Post-biological Era. ISBN: 1 899274 03 0. University of Wales College, Newport, 1997. (online at www.interval.com)

[9] F. Mayor, Science and Power: A New Commitment for the 21st Century, UNESCO Director General's address to the Association for the Advancement of Science, Washington, D.C., 25 June 998 (see www.unesco.org).

[10] N. Negroponte, 2b1 Foundation Mission Statement (see http://www.2b1.org/mission.html).