< > ↑

Storyboard VR

A multi-user storyboard tool for VR

Image of miniature dioramas that were created with the Storyboard VR program
Storyboard VR is a prototype tool that allows artists and directors to collaborate in real-time to create storyboards for VR productions.
Storyboard VR is a prototype tool that enables the creation of storyboards for virtual reality productions at similar speeds to traditional film. It allows artists and directors to collaborate and discuss concepts unique to VR, and create digital assets while planning in the medium itself.
Problem
Creating a movie for VR is not as simple as taking a regular script and using the well-developed film making process. Unique properties of VR require directors to consider concepts such as presence and peripheral vision, and use them effectively. There’s an added challenge of guiding audiences through a narrative, while leaving them free in a fully immersive environment, to look or move in any direction, and even trigger events within the environment. No tool exists to properly let film teams plan for this environment in real-time during early discussions.
Solution

To help address this unique set of problems, the workflow specific to the needs of professionals creating storyboards for VR and the related challenges were examined, and a prototype was created, with the purpose of enabling teams to conduct initial explorations and collaborate at/near the speed currently possible with traditional film.

A multi-device interactive storyboard system was created for tablet and VR: that leverages both the speed and skills of a storyboard artist, and allowed a director to experience the results in VR simultaneously.

My Role
Project Designer & Manager
Researcher
Developer
Tool
C++
OpenGL
Adobe Photoshop
Adobe Premiere
Hardware
Microsoft Surface 2
Oculus Rift (Gen. 2)
Process
Discovery
Interviews
Ideation
User Stories
Design
Prototyping
Usability Testing
Reflections & Synthesis
examples of some storyboard images as seen in VR
step 1
Discovery
step 2
Ideation
step 3
Design
step 4
Prototype
step 5
Reflection

Discovery

Since there is no standard way to plan for stories in VR, the discovery process commenced by interviewing a variety of professionals (producers, directors, creative & technical directors, and storyboard artists) who each had at least 10 years of experience working with film, television, and VR. Discussions were led with open-ended questions encouraging the individuals to explain their process, the challenges, and the different factors they viewed as important while planning and working with VR versus traditional or Stereo 3D film.

Differences

There were numerous differences between traditional film and VR stories that were highlighted during our discussions. Some of the major ones include:

  • Presence – Presence is the feeling of being inside a scene rather than looking at it from the outside. VR evokes a sense of being there that is not captured by traditional or stereo3D film
  • Active Experience – Unlike movies - which are traditionally passive, static experiences - VR experiences allow audience participation, where viewers can actively look and potentially move around to explore the immersive setting, while following a narrative
  • No Framing – Due to the border of the screen, traditional film allows the director to control the layout and frame a shot. Since VR typically has no border and no guarantee where a user will look, directors need to change how they approaches planning a shot
  • Blocking – the concept of blocking is the process of deciding what will be where. It is vastly different with VR since the space all around the viewer can be used
  • Peripheral Vision – Since VR creates the sense of actually being on location, the director needs to think about the viewers’ lines of sight. In the past, things that would have been off-camera can now be seen peripherally
  • Audience Attention – Keeping the audience focused on the narrative is an added directorial burden in VR. Sound or staging cues, lighting and movement can all be used to draw the user’s attention
  • Limited Transitions – With traditional movies, directors rely on cuts, fade in/out, wipes and other shot transitions to keep the story engaging and to move the narrative from one moment to the next. In VR, transitions are more jarring and potentially disorienting, as they involuntarily “teleport” the audience
  • Screen Format – A director has a wide choice of spatial staging formats in VR, from a virtual screen to a completely immersive environment
  • Stitching – Current fully immersive VR video capture rigs mount multiple cameras omni-directionally, to immersively capture a live scene. Stitching these multiple camera images together often creates faint distortions and discontinuities along the seams. While future imagery may be seamless, current directors need to plan their shots to minimize viewer focus in the vicinity of seams
  • Optics – A director’s choice to present their movie mono- or stereoscopically has a different dimension in VR. With stereoscopy, the left and right eye images are slightly different creating the sense of depth; monoscopically, a director relies on depth cues like retinal size, overlap, and aerial perspective
  • Duration – For the near future, VR films will be fairly short in length, both due to the complexity of authoring content and viewer fatigue from sustained viewing in VR
  • “You” as the Camera – with the “camera” direction being controlled by the viewer’s head, the audience is often treated to an experience where they act as the physical camera within the scene
  • Layer Alignment – With VR, content can be in the environment, locked to the viewer’s perspective, or locked in the environment in relation to the user
  • Scene in Audience Space – with stereoscopic film, content can be on, in front of, or behind the physical screen. With VR, however, the scene converges as it recedes into the distance. Due to this fact, all of the action is in audience space
Types of VR

Virtual reality actually belongs to a spectrum of mixed realities, and there are many different styles of VR. VR can be fully immersive (e.g. Oculus), 360˚ surrounding environments (e.g. Google Cardboard), or using head tracking and a computer monitor in what is known as fish tank VR. Virtual reality can also take a full room (or cave), a small mobile device, or a viewing surface attached to an arm mechanism, that can be moved around a specified area.

As a consequence, the viewing volume is different depending on the medium. Standard shapes are semi-spherical (often experienced with fully immersive spaces), cylindrical (often seen in 360˚ surround) or cubic (common in cave and fish tank VR).

the top half of a sphere

Semi-sphere
(Fully immersive environments)

a cylinder

Cylindrical
(360˚ surround environments)

a cube

Cubic
(Cave environments)

Existing Drawing Programs

At the beginning of this project, there were only a handful of standard drawing applications in VR. Programs such as Tilt Brush, Quill, and Gravity Sketch. All of these are more 3D modelling tool than sketching applications, and therefore, not appropriate for early stage storyboards.

Tilt Brush Splash Screen Quill Splash Screen Gravity Sketch Splash Screen
No Standards

Currently, there are no standards for planning VR stories, as there is no preferred or even sufficient process for discussing evolving VR cinematic concepts. Consequently, groups scramble to cobble together a 3D visual mock-up (using images, projected 2D storyboard panels, 3D objects, toys, cardboard cut-outs, previs and other proxies), so presence in VR can be experienced and discussed.

To make matters more challenging, to this day, no one really knows or agrees upon what a storyboard is for VR. However, everyone agrees that planning needs to be done!

Ideation

User Stories
As a director, I need to quickly explore different ideas. I need help to refine my vision of scenes. I need to think about presence, blocking, and layout. I need to experience the ideas in VR, and I need this process to be fast enough to see results in real-time, during a discussion, having artifacts to work with within seconds. These ideas need to be iterative and disposable, and in a style that I feel reflects my aesthetic.
As an artist, I need to be able to get ideas down as quickly as possible, spending as little time as possible on a panel. I need to be able to capture what the director wants and translate that into a sketch that’s based on my style of drawing. I need to think about blocking, foreground / middle / background, peripheral vision, layout, actors positions, and scenery. I need to be able to come back and clean up my images later.
User Requirements

Requirements for storyboarding for VR are very similar to those needed for traditional storyboards. In particular, artists need to be able to work quickly, capture the director’s ideas, and think about blocking and layers during discussions. Directors still need help exploring and refining their vision. However, VR has many unique requirements, as outlined above, requiring the director to work in 3D space.

Since none of the existing drawing tools worked fast enough in VR to allow for a storyboard artist's speeds, and since drawing in VR is, quite frankly, very tiring after long periods of time, ultimately artist need to stay on their tablets - a tool that works well and has been proven. The question was "how should the artist use the tablet"?

Possible Approaches

A few ideas to allow artist to draw on a tablet that I toyed with are:

  1. Draw images, then paste them into the VR space
  2. Setup 3D infrastructure (i.e. blocks) beforehand, then draw on those
  3. Mirror the physical tablet in VR space, allowing the artist draw on the tablet while in VR
image of a character sketching on a tablet then copying it into the VR space image of a character sketching on a tablet on 3D objects that also appear in the VR space image of a character sketching on a tablet while in the VR space themselves

One of the major drawbacks to all of these options is that in the end, they all end up being a 2-step process, which slows down the artists.

Synchronicity

Another challenge to consider is that all productions are different, and at times the director and artist work together (synchronously), at other times they work at different times (asynchronously). If the artist is working on a tablet, how do they know what the director is referring to while in VR space. They could both work on the tablet, but the director needs to be in VR for the uniquely VR considerations (e.g. presence)...things that aren’t easily captured on a tablet.

image showing different combination of artist and director in the same space, or different spaces; working at the same time or different times
The artist and director may be co-located and use the system simultaneously, or apart and use it asynchronously

Design

While interviewing the different professionals, one of the tools I noticed a few of them use was a radial map of the space, to help them understand how things were being laid out. I felt this was a good tool to include in the design.

an overhead map with concentric rings going out from the middle, each ring labeled with the distance from the middle (i.e. 1m, 2m, etc.); as well, there are several objects within the space

A radial map is sometimes used to help plan the VR space

The final solution was partly inspired from this radial map. Having created a stereoscopic storyboard tool that allowed artists to select the depth of a plane, I realized that I could combine these two solutions. If I extended the stereoscopic drawing layers so they were long enough to represent a 360˚ panorama, I could treat the drawing volume as a set of concentric rings that could be unwrapped and displayed as flat panoramic panels on a drawing tablet, allowing the artist to draw on the tablet at their usual speed, while the director could experience the content in the VR space.

an image showing a set of concentric cylinders with arrows to and from a series of flat rectangles stacked one on-top of another

By treating the VR space as a stack of concentric cylinders, one can unwrap them and display them as panoramic panels on the tablet

This solution allows the artist to draw on the tablet and view the panels as a series of storyboards, while the director experiences the content as it’s being created in VR. This still leaves the question of what does the storyboard look like for the director? Should they simply see a 2D representation of the panels like the artists sees, or is it possible to do something more engaging?

For the final design I proposed a scene that represented the panels as a series of miniature dioramas. This allows the director to quickly view and manipulate each scene, while still experience it in VR, with spatial cues and layout.

Collaboration

Yet another challenge that presents itself when the director is within a VR headset, and the artist is in the “real world” on their tablet, is how do they easily communicate? How do they know where the other is looking or what they’re referring to? This challenged was addressed through several measures.

First, on the artist’s sketch surface, there is a bar at the bottom of the sketch area that indicates the view range of the person in VR. As well, with the radial map, in the center of the circles is a pie shaped wedge that presents the direction that the director is looking.

an image of the sketch surface on the tablet, and a red bar along the bottom indicating where the director is looking a radial map showing the lines for the objects that were drawn in the space, and a red 90 degree wedge in the middle, showing the direction the director is looking
A red line at the bottom of the sketch surface shows the viewing range of the person in VR (left) as well as a red wedge in the middle of the radial map (right)
a picture of the moon lander as sketched in VR, and a rectangular widget surrounding the area that the artist is currently looking on the tablet./> <div class=
A “widget” in VR shows the director where the artist is focused

In the VR space, there is a visual widget that shows the direction and area where the artist is currently looking.

Prototype

A number of issues were addressed with this collaborative multi-device storyboard prototype: ensuring homogeneity between viewing environments, quick authoring, fast navigation, and a proper story overview. The prototype was presented to VR film professionals for feedback on the design, its ability to address the planning needs in VR, and to assess the overall workflow.

Basic UI

The sketch surface has similar functionality to the  Storeoboard app, with several additions:

  1. a line across the sketch surface to help show the artist at what depth they are drawing
  2. a floor grid to help with visualizing the depth
  3. directors notes
  4. a VR view range indicator that moves according to where the director is looking
the sketch surface as seen by the artist, image highlights the VR range indicator, the depth line indicastor, and the director's notes. The grid on the floor can also be seen
an image of the radial map as seen on the tablet by the artist. A 90 degree grey wedge indicates the direction that the tablet is currently showing

The radial map (centered around the user) provides:

  1. information about where content is laid out
  2. the ability to select and move items
  3. information about where the director is looking - red wedge in the middle of the map
  4. ability to adjust current sketch surface viewing direction (gray wedge)

The storyboard itself is presented as a series of panoramic strips - one for each panel. The degrees offset is shown underneath (from -180˚ to 180˚), and there are boxes showing the intended initial direction for the user to look. These can be adjusted by dragging the boxes.

a series of panaramic sketched strips, each showing the 360 degrees of the sketched VR scene. Under each strip is a number line showing the number of degrees, and over each panel is a green rectangle highlighting the intended initial direction for the user to look.
an image showing the miniature diaramas that are shown within the VR space as an alternative to the traditional storyboard view

Within the VR space, the director can view and manipulate an alternative to a storyboard through a series of miniature dioramas.

Usability Testing

To evaluate the system, interviews with professionals from the VR cinematic industry were conducted. The interviews focused on high-level feedback to assess the workflow and its ability to support the creative process.

Each professional was invited to experiment with the system following a ten-minute demonstration of its functionality. They experimented with each context freely without any time constraint (i.e. using the tablet and HMD simultaneously, using just the tablet and then looking at the results in VR afterwards, etc.). The sessions were concluded with a semi-structured interview, where their feedback was elicited on the system as a whole as well as exploring specific points.

The feedback was largely positive, including some of the following responses:

“from a prep side, it’s a huge step forward”

“the simultaneous view in the [HMD] is amazing… a hit hands down”

“I loved it… hugely valuable”

“good way to preview without using the [HMD]”

“If you had a [consumer] version ready tomorrow, we’d use it”

Reflection

Since this project was based on an exploratory idea, I was uncertain about what would work, and how users would react. This was a great project to explore and learn more about interactions in VR, as well as learn about cross network communications with applications. Overall, this 2-device approach was positively received, and hints at many possibilities. With the application itself, there were many insights and potential to add or adjust components for future development.

The design decision to treat the environment as a stack of concentric cylinders results in a setup that is more ideal for 360° video. Though this still can be used for other types of layouts (i.e. stacked semi-spheres), the ability to change the shape of the environment would be more ideal for actual production.

As well, with this system using sketch strokes to add content, though being lightweight and effective, they lack explicit mass and volume. Combining the sketches with previs quality 3D assets would be interesting for future additions.

Rorik Henrikson

Product Designer

© 2024 Rorik Henrikson