‘Minority Report’ Inspires Technology Aimed at Military

 

By JONATHAN KARP

 

Los Angeles

IN THE FUTURISTIC movie “Minority Report,” Tom Cruise gestures with his gloved hands to sift through crime-clue data that are displayed on giant screens. With the twist of a wrist he can move information from one column to another or delete items.

Pamela Barry, then a Raytheon Co. engineer, had a eureka moment while watching the 2002 Steven Spielberg sci-fi flick. Ms. Barry believed such a system could be a boon to the military as it tries to parse reams of information in the heat of a battle.

Raytheon then hunted down the scientist who was behind the movie technology, John Underkoffler. Raytheon decided to fund an effort to try to turn his film fantasy into reality and explore its potential for speeding up intelligence analysis, says Allan Mattson, the defense company’s director of national space programs.

The fruits of that investment are housed in a darkened room in a converted Los Angeles factory. There, a man wearing reflective gloves uses hand gestures to manipulate pictures projected on a panoramic screen. He slides an index finger forward to zoom in on a street scene; swivels a horizontal hand to the right to scroll through a video; sweeps both hands to the left to clear the screen.

Raytheon believes such ‘gesture technology’ can help solve one of the military’s biggest problems: information overload. Commanders are increasingly unable to process the massive flow of intelligence from satellites, sensors and soldiers. To tackle that challenge, Mr. Underkoffler and Raytheon are devising ways to visually display and manage the date in a user-friendly way to quicken combat responses.

The technology also has commercial potential. Software engineers developing everything from three-dimensional modeling programs to videogames have long strived to liberate consumers from the computer keyboard and mouse.

“Keystrokes and mouse clicks limit your degree of freedom,” says Mr. Underkoffler, who earned his doctorate at the Massachusetts Institute of Technology. By communicating with a computer through gestures, hands can do as much as five or six mice, he adds. “Your hand becomes a Swiss Army knife,” he says.

Raytheon, which has licensed Mr. Underkoffler’s technology and unveiled it to Air Force and intelligence officials last week, aims to adapt it for use in future command centers. The idea is to streamline the disjointed and limited functions currently performed by scores of soldiers manning banks of individual PCs. In Raytheon’s vision, real-time video and maps will be fused with database information on large interactive screens to assess battle situations.

Raytheon isn’t alone in chasing the command post of the future. And it isn’t the only company injecting Hollywood into this race. Silicon Graphics Inc., which is known for special effects in movies, is working with the Army to develop the computing firepower that command centers will need.

Raytheon appears to be among the leaders in applying gesture technology. The “gestural interface” is part of a broad quest to find intuitive ways for people to interact with the digitized world of computers. Such efforts encompass speech recognition and touch-screen technologies.

Mr. Underkoffer’s work is more complex because it involves hand motions in the air. Early gesture-technology efforts in the 1980s used “datagloves,” which were tethered to a computer by wires. “It was sexy, but it had many disadvantages,” says Andries van Dam, vice president for research at Brown University, which has long studied melding gestures with computers. Mr. Underkoffler’s system uses gloves with reflective heads and infrared cameras that detect the motion by bouncing light off the beads, similar to motion-capture technology used in film animation.

Mr. Underkoffler, a goateed 37-year-old, did pioneering research at MIT’s renowned Media Lab. His work with holography and light built on concepts of virtual reality, leading to what he dubbed the “Luminous Room,” where people can work with electronic data on ordinary walls or tables rather than being glued to the tube and keyboard. “The idea is to force graphics out of the monitor and into the real world,” he says.

Among his innovations is software, in used at MIT, that lets architects place physical models on tables where digitally produced shadows, reflections and wind flows are projected. Move the model, and the shadows adjust, allowing the designers to quickly stimulate the spatial impact of a building.

Mr. Underkoffler was discovered at the Media Lab by Mr. Spielberg’s “Minority Report” production team in 2000 as they scouted ideas for the film. Mr. Underkoffler soon relocated to Los Angeles as the film’s scientific adviser, launching a Hollywood career that includes work on “the Hulk” and “Aeon Flux.”

One of Raytheon’s goals in trying to create a version of the technology for the Defense Department is to sell more of the cameras and sensors it makes, which are used on spy satellites and military planes. Raytheon won’t disclose how much it is investing in the project.

One recent morning, Kevin Parent, a former MIT colleague of Mr. Underkoffler’s, donned black gloves to demonstrate gesture technology.

“Shall we step into the ring,” he said, referring to a metal truss holding several infrared cameras and a projector. Standing before the wide screen, Mr. Parent raised one hand, his index finger pointing like a pistol. Pressing his thumb down while moving his finger across the screen is like clicking and dragging an icon with a mouse.

So far, Mr. Underkoffler has invented a vocabulary of more than 20 gestures, each corresponding to a mathematical formula. As Mr. Parent pans westward across a projection of the downtown Los Angeles skyline by moving his hand to the left, he raises it to simultaneously enlarge the buildings. “We’re panning and zooming,” exclaims Mr. Underkoffler. “And you still have another hand.”

Mr. Parent toggled between a rooftop video of downtown Los Angeles and an overhead map to analyze the scene from different angles. With hand flicks, he zipped forward and backward through the video clip and noticed the gradual movement of a window-washing crane atop a distant skyscraper, something that was undetectable at the video’s regular speed. In a battle situation, that crane could be a piece of slow-moving heavy artillery.

Mr. Mattson believes that removing the normal computer interface allows the brain to combine space and time data faster to identify threats. “This toold isn’t a panacea, but it’s part of the solution,” says Gerald Perryman Jr., a retired Air Force general who is Raytheon’s vice president for intelligence and surveillance systems. “This may give us a fighting chance to put the clues together.” Mr. Perryman acknowledges that gesture-dominated command centers with three-dimensional display aren’t likely for a decade.

Raytheon is working on more immediate applications, such as a device called a Common Tactical Blackboard to offer a portable bird’s-eye view of a battle zone and software that suggests combat responses. But Mr. Underkoffler retains the right to pursue commercial uses, such as command-and-control operations for railroads and ports, and virtual wind tunnels for industrial designers. Videogames are also in the mix. With similar but less advanced technology, Sony Corp. already markets the EyeToy, in which a camera captures a person’s movements and incorporates them into the game on TV.