Bot & Dolly is a San Francisco based, design and engineering studio that specializes in automation, robotics, and film making.

Their mission is to advance motion control and automation as a creative medium, and build world-class tools that enable others to do the same.

At the core of their technology is an integrated software/hardware platform that provides precise and expressive control of 6-axis industrial robots.

On top of this core platform they provide industry-specific tool sets such as IRIS to support in the creative process, from prototype to production.

To date, their robotic tools have been used in feature films, national television ads, Las Vegas shows, and large-scale art installations.

Bot & Dolly calls their combination of robotic arms and projection mapping a “kinematic projection platform”

Box movie

Box is Bot & Dolly’s live performance film which documents a first-ever live synchronized performance using 3D projection mapping, robots, and actors

The images are projected onto the panels and surrounding environment as the performers move about and interact with them.

This short film is the first of its kind to combine projection mapping of 3D computer graphics, robotics and an actor into a synchronised performance.

Box explores the synthesis of real and digital space through projection-mapping onto moving surfaces.

It is the culmination of multiple technologies, including large-scale robotics, projection mapping, and software engineering.

The short film documents a live performance, captured entirely in camera

.

For BOX’s animated content, Bot & Dolly brought on none other than Bradley G Munkowitz, no stranger to Motionography. His trademark attention to detail is on full display in every frame of the project.

Designed to control camera movement, activate lights, and shift set pieces with ever-repeatable motion, this precision allows the virtual and physical worlds to unite and CG elements to match in real time.

Not limited to film, their motion control advances have applications in live performance, experiential installations, and architecture.

.

Box – Behind the scenes

Gravity – the movie

If you have seen the latest marketing for GRAVITY, you know they are capitalizing on the groundbreaking nature of the film.

Everyone from James Cameron to NASA scientists have weighed in on how realistic the portrayal of being in space is in Alfonso Cuaron’s film.

There have been lots of different techniques used to create the illusion of space including harnesses and green screen.

Ron Howard famously filmed Tom Hanks and the cast of APOLLO 13 in hundreds of flights in their anti-gravity simulation airplane.

But, Cuaron has done a lot more and the results are amazing

The actors, Sandra Bullock and George Clooney, were filmed acting inside a light box that had projected from it the correct environment the actor was meant to be in.

Thus, if the actor was meant to have the earth providing bounce light from their left, on the screen on that side of their body was a giant earth image, digitally controlled and correctly aligned.

.

“Here, the light box LED panels showing the interior of the space station providing the correct lighting around the actor.”

This combination of flying the camera and lighting Clooney and Bullock by the light box would then allow an accurate performance to filmed of each of the two lead actors.

But for a shot where they were in their suits, only their faces would be real, the rest of their bodies, suits, hands/limbs, environments etc were all digital and fully 3D animated.

The light box, resembling a hollow cube, stood more than 20 feet tall and 10 feet wide. The interior walls were constructed of 196 panels, each measuring approximately 2 feet by 2 feet and each holding 4,096 tiny LED lights.

“The actor’s cage inside the light box”

It was based on the technology commonly used to put massive images on screens behind rock bands or at sports grounds. “Each pixel is made out of an LED that we could control,” explains Webber. The lights could not only project light, but also varying colors, and the lights could also be programmed to move at any speed.

“What the actors saw”

As its name suggests, the purpose of the light box was to cast the appropriate illumination on the actors inside, no matter where they were or how fast they were moving in relation to the light source.

“The IRIS robotic camera rig outside of the light box”

The light box walls could also be moved and were at times placed on robotic arms.

To have the actor ‘fall’ towards the camera, the actor would not necessarily move but instead the camera would fly towards the actor, with the aid of the IRIS robotic camera rig from Bot & Dolly.

Below is a diagram of the side and top view of the camera arm that would move towards the actors to simulate them flying at the camera on screen.

“It can’t be done.” This is what Alfonso Cuarón heard when speaking of his vision of weightlessness for the film Gravity. While these words discourage some, they kindle others.

The making of Cuarón’s Gravity was a formative project for Bot & Dolly.

The film pushed us to evolve our robot platform to the specific needs of cinematographers and visual effects artists. By leveraging our IRIS platform, including four robots and Bot & Dolly’s engineering team, we were able to make that impossible vision a reality.

Our work on Gravity began mid-2011 after the Head of Visual Effects at Warner Brothers heard about IRIS and reached out to Bot & Dolly. He and his team were looking for the technology needed to bring Cuarón’s ambitious film to life, and after an initial test shoot in San Francisco our robots boarded a cargo plane headed to London.

Bot & Dolly provided Cuarón and Framestore with the tools necessary to execute complex cinematography based on computer previsualizations, in live action with industrial robots and other onset hardware.

Because our tools integrate tightly with industry standard software Maya, Framestore was given control of camera, lighting and other set elements from within their established animation environment.

The entire story of Gravity was animated in CG prior to filming.

Through the use of BDMove and our four IRIS robots, we were able to reproduce the zero-gravity motion created by Framestore’s team of animators.

Cuarón and Webber took a unique approach to simulating weightlessness. Instead of moving an actor through a set using traditional wire rigs, they achieved the illusion of zero-gravity by moving the world around the subject.

This could only be done by synchronizing lighting, LED backgrounds and actor pose with frame accurate camera positioning.

Bot & Dolly enabled this approach by unifying technology on set with a single common timeline. Beyond camera control, we were responsible for driving LED graphics, cueing actors and technicians, real-time compositing, and implementing an interactive playback system.

Each day, the challenges on set inspired us to expand the functionality of our tools and bring new levels of a technology to film production.

CCG tools

Much of Prime Focus’s conversion work takes place in the space capsules, where live action and some CG elements from Framestore made up the shots.

That meant that a major challenge for Prime Focus was interacting back and forth with Framestore to produce the stereo conversions for those shots.

“We were being sent 2D tracks of the plates and geo from Framestore,” says Baker, “and we’d convert that. Then we developed some really accurate stereo camera development tools. We could then pass these stereo cameras back to Framestore and they could use these in their Maya scenes.”

“Framestore had their own proprietary stereo camera format,” explains Prime Focus’ Rajat Roy. “The original footage was shot at 3K, and all the repositioning and scaling wasn’t applied when they tracked their cameras. So we had to take their live asset, like a camera, apply a non-final repo and scale to it, do our conversion according to the new reposition and scale of the image, apply any changes as they arrived, make sure it was version tracked, then hand back to them stereo cameras that matched our conversion – so that they could stereo render and re-apply their comp to our conversion background.”

To help with having the most consistent and accurate elements for conversion, Prime Focus was able to leverage cyberscans of say Sandra Bullock’s head for positioning that could be put their View-D pipeline. Traditional roto and spline work was carried out with a multitude of tools including in Fusion and NUKE.

The result was a consistent feel to the stereo in the film, in particular the stereo CG renders by Framestore, and a clear enhancement to the sense of weightlessness and vastness of space. “I really thought that that enhancement worked so well,” notes Roy. “For example when Sandra was sitting on a chair, they exaggerated the difference between her and the chair to make it look more like she was more in a floaty environment.”

“There was actually a lot of subtlety to it inside the capsule,” adds Bristowe. “But also there were a lot of technical challenges that needed to be resolved before we even got into the creative side

Technical Background

The biggest obstacle to making Alfonso Cuarón’s space epic “Gravity” was the natural phenomenon itself.

The Mexican director, along with his longtime cinematographer Emmanuel Lubezki, visual effects supervisor Tim Webber and a crew that numbered in the hundreds, spent four years conquering the forces of gravity in the Sandra Bullock-George Clooney drama due out Oct. 4.

The film stars Bullock as an engineer with little space experience who must rely on veteran astronaut Matt Kowalsky (Clooney) when satellite debris cuts them loose in space.

Though traditional visual effects have often sufficed for movies set beyond Earth’s atmosphere, Cuarón’s vision would find the astronauts outside their shuttle for most of the film that coupled with his penchant for the long, multi-minute camera shot would require some new technology to realistically simulate zero gravity for such extended periods.

“The biggest miscalculation was that I thought everything was going to be more straightforward,” said Cuarón. Turns out nothing was.

The team tested several technologies for the job, Cuarón said, so different in nature that all they had in common was that they all used “computers and robots, and they were all very painful for the actors.”

The solution they settled on was to put the actors into contraptions that would spin them around just enough to make it look like they are floating while the cameras and lights did the majority of the visual work.

To do so, Webber invented “The Lightbox” after Lubezki, who for the first time pre-lighted an entire film inside a computer, was inspired by the lighting design at a Peter Gabriel concert at the Hollywood Bowl.

In a studio outside London, Webber and his staff created a roughly 20-foot-tall, 10-foot-wide cube filled with LED lights in segmented panels that act as projection screens lining the walls, ceiling and floor.

Each actor in turn stood in a “tilt-a-rig” — something akin to a cherry picker bucket that could tilt Bullock and Clooney forward and backward and spin them in various ways to create the illusion of tumbling head over heels, adrift in the stars.

The camera, mounted on a giant robotic arm, was capable of swooping in and out, shooting from above and below, all adding to the effect of untethered movemen

Hollywood’s latest blockbuster Gravity was filmed by robots.

Four giant industrial robots whisked props, lights and even actors around the set in a ballet of split-second precision, as well as doing the camerawork.

They call it cinematic automation.

“We are taking a movie set and thinking about it like a manufacturing facility,” says Jeff Linnell, co-founder of Bot&Dolly.

Bot&Dolly bought three second-hand industrial robots back in 2008. “I had been wondering for years why people weren’t using them to move cameras around,” says Linnell. He ran a small advertising and video production company in San Francisco and had spent his career doing motion graphics and animation. “The first robot found its way into a Louis Vuitton TV commercial a week later.”

Some time later, Linnell got a call from a Warner Brothers executive who said the studio was shooting a new movie called Gravity.

“It has a lot of impossible shots which you would not be able to do with traditional wire work and is massively ambitious technically,” explains Linnell. It took a year and a half to write a new control system for the robots that could be used on the set.

Robots were used in film-making as far back as Star Wars, but they were always custom-built and required proprietary software and a highly-specialized human operator. In the 1980s the computer conquered Hollywood, movies went digital, and robots faded into the background.

Bot & Dolly’s founders felt that everything that could be done on a computer had already been done and that it was time to get film-makers back into the real world.

So they took Autodesk’s Maya animation software (the industry standard) and wrote tools to allow non-robotics exeprts — like animators — to run robots using the software they were already familiar with.

“Animators were flying cameras around in the virtual world doing Avatar or whatever but they never had the power to be film-makers,” says Linnell. “Now the same animators can move a camera around, or an actor or a prop. Anyone from Pixar can pick up the tool that they use every day, hit an export button and animate a robot.”

Bot&Dolly’s software system controls some standard robots like Scout and Iris, which weigh from 6 up to 500 kilograms, but users can also control their own robots by adding a new model to the software.

Robots can achieve a level of precision, speed and coordination of movement which cannot be matched by humans. “If you want to move a coffee cup six inches across a table at two meters per second and have it stop on a dime, we want to give you a tool to do that without hiring a developer.” Lights, props, explosions, special effects, and even the positions of the actors — all can be synchronized to the millisecond and coordinated with sound and playback.

Industrial robots don’t usually work in such close proximity with people, so safety was a critical issue, especially when those people are expensive movie stars like George Clooney. The system contains checks and safeguards to ensure the robots are on the programmed flight path and uses laser tripwires, pressure mats and other technology to keep track of the humans. High-risk shots are rehearsed at various speeds, building up to real-time.

Bot&Dolly’s robots have also developed showbiz careers of their own. They have appeared in advertisments for Google and star in a Las Vegas show where they act and play music with the Blue Man group.

“People are pretty fascinated by large robots,” says Linnell. “When they move in a highly coordinated way where all the axes are moving at the same time, the movement is incredibly organic and snake-like. It’s a bit disconcerting and amazing even to myself having watched these things for years now. In a theatrical production, we are trying to give them a sense of character, purposely making them sad or proud or scared. You can convey emotion quite easily.”

The next generation of stars may be built, not born.