technical

Skyspheres Released: 360 Panos in WebVR on all platforms

1107 312 flowadmin

I’ve been working on getting WebVR projects working across all platforms and in all modes, and I’m declaring cautious success!

You can see the result here: skysphere.flow.gl.

skysphere screenshot

This is a simple app to show 360 photos that I’ve taken around California. There are four aspects that I needed to figure out:

  1. 360 image capture and editing
  2. WebVR across all platforms
  3. Control panel interface
  4. Text labeling

360 Image capture and editing

Photography is a serious hobby of mine, and in the last year I’ve used my Samsung Note 4 to capture dozens of 360 photos on my various travels and hikes around California.

I used the built-in 360 Panorama feature in my Samsung Note 4 using the Samsung Camera photosphere extension. I found this interface superior to the Google camera app, because of the little sphere icon that fills in as you take the 40 or so images required. (Note that this is possible with any Android phone, with the built-in Android 360 capture.) The result is certainly not perfect, and even though I did some serious tweaking to the original files in Photoshop to fix some stitching errors, there are still some issues. But overall, it is exciting to think that this is possible with any phone, today.

Sierra Buttes Ridge 2015-07-13x1024

Photoshop editing is pretty easy, except for the view when looking directly down and up. Getting the stairs below me at Sierra Buttes to line up through photo editing was impossible using straight Photoshop without jumping though complicated hoops… oh well, good enough!

Sierra Buttes Steps 2015-07-13x1024

WebVR across all platforms

I used Boris Smus’s WebVR boilerplate: https://github.com/borismus/webvr-boilerplate.

When I started the project a few months ago, getting it working across all platforms was a huge pain, but it kept getting better, and just as soon as I had it all working pretty well, Boris (at Google) released this boilerplate and I adopted his. (What’s a little bit of wasted effort when living life on the bleeding edge!)

To enter stereo VR mode, just click the little “cardboard” icon in the lower corner.

Control panel interface

I spent way too much time thinking about and experimenting with this control panel.

The design expectations included:

  • no obstructions (control panel or text) of the 360 content
  • ability to navigate to the next photo without going ‘back’ to a menu screen
  • ability to see what the next photo is… avoid typical blind forward/back mechanisms. Since these photos are big, slower connections will take a while, so only pull in the big 360 photos with intention.

I liked the idea of finding the control panel by looking down, so I experimented with finding the right pitch of the head to have it appear.

skysphere screenshot control panel

I wanted it to always appear in front of the camera, but I wanted to avoid doing complex math, so my trick here was to create an invisible object as a child of the camera so that it will always be in front of the camera at a certain z distance. Then I use that object’s location as the location to pop up the control panel. Easy, no math!

Once the control panel pops up, then it locks into that position, so that additional gaze interactions will move a cursor over the particular photo to be selected.

I haven’t seen a lot of similar control panel techniques, so I’m interested in getting your feedback on Twitter @jmarshworks.

Text labelling

The typical text labeling mechanisms for 360 photos seems to lock the text front and center of the camera, or up in the center of the view, with the text always directly facing the camera. Although this is highly readable, it also can be distracting until it fades. So I adopted a different design, just to see what it felt like (VR interaction design requires experimentation at this point!) I’m obsessed with beautiful letter forms in 3D space, so maybe I’ve gone overboard.

skysphere text example

I actually put the text plane in space, and this plane responds as though it were a real 3D object, particularly in that it’s not locked into facing the camera. While still clearly an artificial addition to the content, I felt like the sense of presence was stronger because it moves naturally. Do you?

Note: I haven’t seen people do this! It seems to make so much sense to me, and I enjoy it much better.

Conclusion

I will next experiment with adding audio to the experience. For each of the photos taken, I had recorded several minutes of ambient audio in stereo on my phone. (Simple, I know, not binaural, nor with a stereo array to spatialize correctly.) My intent was to just see if I could enhance the experience with the simplest audio that any user could capture with the phone. But I lost the files due to a phone glitch!

So I’ll be more careful with the audio files, and next time will see if a streaming audio loop will provide a significant benefit in presence to still photos.

 

Building re.flow Part 4: WebGL visuals

600 100 Jason Marsh

This is the fourth part of a four-part set of posts about re.flow. If you missed part 1, go here.

Hear and see the end result here: http://dolby.flow.gl.

This post is about programming the WebGL visuals.

The connections between all the parts of the program objects are shown in this diagram:

CodeDiagramFor the visuals, I’m creating WebGL objects using ThreeJS, and GLAM is providing some handy utilities for tracking your scene graph, handling interaction with Pickers, easy key-frame Animation, and more.

Animation

Let’s start with the animation before we get to the textures. When I create an AudioClip, the animation is parameterized and included with it.

[code language=”JavaScript”]
new AudioClip( {name: "spaceshwoosh1", file: "spaceshwoosh1", track: "spaceshwoosh", label:"Space Shwoosh 1",
animation : [{//duration in musical measures , each step is 4 measures
duration:18,
object: "parent",
target: "rotation",
keyBeats : [ 1, 2.75, 4.5, 7.5, 9, 18], //keys should be in beats
values: [ //values should be in radians around center
degreesToYradians( 56, 1),
degreesToYradians( 37, 1),
degreesToYradians( 71, 1),
degreesToYradians( -171, 1),
degreesToYradians( -73, 1),
degreesToYradians( 56, 1),
] },

] }),

[/code]

AuditionDegreesI cheat to do the animation: Instead of building a series of spline curves through space (I tried this too, but it was too hard synchronize with the audio panning), I just add an extra invisible object to each 3D Object, and spin the pair around their mutual center point. That way I can move the objects in a plane around the Y axis in front of the user with nothing other than the degrees, which happens to be exactly what Audition shows in the interface!

As a good lazy programmer, I wanted to make it as easy as possible to synchronize the audio panning to the animations. So I created math to translate the same numbers in Audition directly into the key frame animator, hence the degreesToYradians function:

[code language=”JavaScript”]
/*
This function is specific to taking the values directly out of Adobe Audition’s 5.1 track panning and converting them into a horizontal plane around the user.
It inverts the values, and subtracts the 180 degrees in order to match those values.

The animator needs to be sweeping across the circle, not just animating between points. Otherwise anything that is moving relatively fast will not work right.
*/
function degreesToYradians(degrees, radius) {
var y = (Math.PI * -degrees /180) * radius ; //radians = Math.Pi * degrees /180
return {y:y}
}
[/code]

auditionKeyframespngAnd the same is true for the keyframe timing: I create a keyBeats structure so I can just type in the measure numbers that match the keyframes from Audition.

Here’s the code for creating the geometry plus the invisible partner to create the offset for the rotation:

[code language=”JavaScript”]
function addTexturedEtherPlane(trackName, texture, position, rotation, spinRadius) {
var parentObj = new glam.Object;//we use a parent object to create an offset
var obj = new glam.Object;
parentObj.addChild(obj);

var visual = addTexturedEtherPlaneVisual(trackName, texture);
var picker = addPicker(trackName);
obj.addComponent(visual);
obj.addComponent(picker);
spinRadius = spinRadius ? spinRadius : 1;

//sets the position of the pair in space
position = position? position : {x:0, y:0, z:0};
parentObj.transform.position.set( position.x, position.y, position.z + spinRadius );
obj.transform.position.set(0, 0, -spinRadius*2);

if (rotation) {
obj.transform.rotation.set(rotation.x, rotation.y, rotation.z);
}

//connects this visual object up to the AudioVisualizer object so when the
// track starts and stops, they stay synced, as well as to send the amplitude
// changes to drive the visualizer
var audioVisualizer = findAudioVisualizer(trackName);
audioVisualizer.trackObj = obj;

app.addObject(parentObj);
}
[/code]

On to the dynamic textures!

 

GLSL Textures

I had never done GLSL before this project. But, no fear, this is the age of Google. Specifically, the awesome ShaderToy. I hunted all around for a shader that I thought would give me an interesting rippling effect that wasn’t too computationally expensive. I found Ether, by nimitz, on ShaderToy. and the audio amplitude as an input to brighten the colors as the volume increased.

I wanted the textures to be ‘beautiful’ as defined by my own subjective criteria, which is like observity obscenity: I’ll know it when I see it. Actually I did want to use natural textures, hinting at hypnotic complex repetitive forms such as campfires and ocean waves. So I started with these original images I photographed and photoshopped:

tahoesolanabeachsentinelstackedGrasses

I combined the Ether shader with the animating texture maps based on code examples from the Shader Fireball by stemkoski.

Without much explanation, here is the fragment shader for the nerdiest readers:

[code language=”JavaScript”]
uniform sampler2D baseTexture;
uniform float baseSpeed;
uniform float alpha;
uniform float time;
uniform float amplitude; //audio volume as an input into the math
uniform vec3 color;

varying vec2 vUv;
varying vec3 vPosition;

float map(vec3 p){
vec3 q = p * 2.0+ time * 1.0;
return length(p+vec3(sin( time * 0.7)))*log(length(p)+1.0) + sin(q.x+sin(q.z+sin(q.y)))*0.5 – 1.;
}

void main( ){
//texture
vec2 uvTimeShift = vUv + vec2( -0.7, 1.5 ) * time * baseSpeed;
vec4 noiseGeneratorTimeShift = texture2D( baseTexture, uvTimeShift );

//ether
float resolutionY = 1.; //smaller the value, the more detailed the image
vec2 p = vec2(vPosition.xy) / resolutionY – vec2(0.5, 0.2);

vec3 color = vec3( amplitude) * 2.0;
float crispness = 4.5; //values between 0.5 and 5.5 work nicely
for(int i=0; i<=2; i++) { //iteration values between 2 and 10 work nicely
vec3 p = vec3(0, 0, 5.0) + normalize(vec3(p, -1.0)) * crispness /2.0;
float rz = map(p);
float f = clamp((rz – map(p+0.1))*0.5, -0.1, 1.0 );
vec3 colorRange = vec3(noiseGeneratorTimeShift.r, noiseGeneratorTimeShift.g, noiseGeneratorTimeShift.b)*0.8+ vec3(5.0, 4.5, 0.5) * f;
float amplitudeClamped =clamp(amplitude, 0.1, 20.0) *2.0;
color = color * colorRange + (1.0 – smoothstep(0.0 , 2.5, rz)) * 0.7 * colorRange *amplitudeClamped ;
crispness += rz; //min(rz, 1.0); //filters out some complexity
}
float calculatedAlpha = (color.b + color.g + color.r > 0.02)? alpha : 0.1 ;
gl_FragColor = vec4(color, calculatedAlpha);
}

[/code]

And here is the ThreeJS code to tie an AudioVisualizer track to the shader material:

[code language=”JavaScript”]
function addTexturedEtherPlaneVisual(trackName, texture ){
var audioVisualizer = findAudioVisualizer( trackName );
audioVisualizer.setShaders({
vertexScript:"texturedEtherVertexShader", fragmentScript:"texturedEtherFragmentShader"
} );

var waterTexture = new THREE.ImageUtils.loadTexture( texture );
waterTexture.wrapS = waterTexture.wrapT = THREE.RepeatWrapping;

audioVisualizer.shader.uniforms = new ShaderUniforms( {
color: {type: "c", value: new THREE.Color( 0xff0000 )},
baseTexture: { type: "t", value: waterTexture },
baseSpeed: { type: "f", value: 0.1 },
noiseTexture: { type: "t", value: noiseTexture },
noiseScale: { type: "f", value: 0.2 },
alpha: { type: "f", value: 0.8 },
time: { type: "f", value: 1.0 },
amplitude: {type: ‘f’, value: 1}
});

var planeShaderMaterial = new THREE.ShaderMaterial({
fragmentShader: audioVisualizer.shader.fragmentScript,
vertexShader: audioVisualizer.shader.vertexScript,
uniforms: audioVisualizer.shader.uniforms.params,
side: THREE.DoubleSide,
transparent: true,
depthWrite:true
});

var geometry = new THREE.PlaneGeometry( 2,2 );

var visual = new glam.Visual(
{ geometry: geometry,
material: planeShaderMaterial
});

return visual;
}

[/code]

Particles

Finally, while I like the clean black background, I wanted to create a sense of an even larger space, so particlesI added particles in the distance. Some are fixed particles randomly dispersed, and some are based on the “fireflies” particle engine created by Lee Stemkoski: https://stemkoski.github.io/Three.js/Particle-Engine.html . I tweaked it a bit, including a more textural texture map, but they are almost straight from his open-source code.

 

Pickers

ThreeJS doesn’t have great built-in selection mechanisms for the 3D Objects. IT is not all that trivial to select an object in 3D space. So GLAM solves this for me with a few lines of code:

[code language=”JavaScript”]
var picker = new glam.Picker;
picker.addEventListener("mouseup", selectFunction);
picker.addEventListener("touchend", selectFunction);
obj.addComponent(picker);
[/code]

That’s it!

re.flow for Dolby Laboratories, 3D object view

So, a quick review: I wrote a piece of music, found some images, and through Google searches and a simple matter of programming, created “art”. You can do it too.

 

Huge Thanks to Andrew Vaughan at Dolby Laboratories, Tony Parisi for GLAM, and also to Dustin Butler for getting the servers set up and the DynamoDB database on Amazon.

 

Follow me @jmarshworks on Twitter, and stay tuned for the Virtual Reality version.

Building re.flow Part 3: Web AudioContext and Sequencer

600 100 Jason Marsh

This is the third part of a four-part set of posts about re.flow. If you missed part 1, go here.

Hear and see the end result here: http://dolby.flow.gl.

This post is about programming the Web AudioContext and the sequencer.

How the AudioContext works together with Dolby Digital Plus

For this project, the audio motion is hard-wired into the sound files themselves, so each file can be created as a 5.1 mix. At this stage of using Dolby tech, you can’t real-time move sounds in 3D space. To be clear, it is not doing real-time re-positioning of each sound: instead it is processing the 5.1 mixes into appropriate positioning based on the audio hardware attached. If the hardware is an HDMI out to a surround system, then you get 5.1 surround sound. If you are using headphones, it uses Head Related Transfer Functions to put a sound in a precise location.   That’s why getting beyond the built-in laptop speakers are so important; Dolby’s real-time processing gives truly amazing spatialization.

I’ll touch on a the project-specific topics regarding the Web AudioContext API, but Boris Smus has laid out the full documentation, so check out his  (free) book.

The AudioContext has built-in positioning of 3D objects, described well here: Mixing Positional Audio and WebGL. It sounds to me like this is just doing basic panning within a stereo mix, without volume changes for distance. But, if you are running on browser without Dolby Digital Plus enabled, this is a good fallback, so re.flow does so.

Here’s code for querying the browser’s capabilities, both the AudioContext and the EC-3  ( the codec for Dolby Digital Plus) capability:

[code language=”javascript”]
var isAudioContextSupported;
var isEC3supported;

function checkAudioCompatibility() {
var contextClass = (window.AudioContext ||
window.webkitAudioContext ||
window.oAudioContext ||
window.msAudioContext);
if (! contextClass) {
// Web Audio API is not available.
alert("This browser won’t work! Multichannel audio is not available with this browser. \n\n" +
"Try the following browsers: \n" +
‘ Desktop: Edge, Chrome, Firefox, Opera, Safari \n ‘ +
‘ Mobile: Firefox Mobile, Firefox OS, Chrome for Android. ‘);
isAudioContextSupported = false;
return;
}
isAudioContextSupported = true;

var myAudio = document.createElement(‘audio’);
myAudio.id = "audioTest";
if (myAudio.canPlayType) {
// CanPlayType returns maybe, probably, or an empty string.

playMsg = myAudio.canPlayType(‘audio/mp4; codecs="ec-3"’);
if ( "" != playMsg){
isEC3supported = true;
console.log("ec-3 is "+playMsg+" supported");
} else {
isEC3supported = false;
}
}
}

[/code]

And here’s the sweet part, which tells the browser that we are dealing with 5.1 surround (6 channels):

[code language=”javascript”]
var finalMixNode;

function initializeAudio(){
audioContext = new AudioContext();

finalMixNode = audioContext.destination;
console.log("maxChannelCount:" + audioContext.destination.maxChannelCount);

if (audioContext.destination.maxChannelCount >=6) {
audioContext.destination.channelCount = 6;
}
[/code]

The above code will report 2 channels in Chrome (or any non-EC3 browser), and 6 channels in Microsoft Edge. If there are not 6 channels available, then we leave the channelCount to its default of 2.

[code language=”javascript”]
//if not using EC3, then use the AudioContext’s positional panner
if (!isEC3supported) {
audioContext.listener.setPosition({x:0, y:0, z:0});
}
}
[/code]

Just to complete the code as to the soft-switch based the browser capabilities, here is the code for preparing a clip to play, which I’m managing by creating an “AudioTrack” object. An AudioTrack is a track that I can swap different clips into. It has functions like prepareClip, playClip, and stopClip. Here is the prepareClip code:

[code language=”javascript”]
AudioTrack.prototype.prepareClip = function ( buffer) {
if (this.isPlaying && this.bufferSource) {
this.bufferSource.stop();
}
this.isPlaying = true;
this.bufferSource = audioContext.createBufferSource();
this.bufferSource.buffer = buffer;
this.bufferSource.loop = false;

if (isEC3supported) {
this.bufferSource.connect(masterGainNode);
} else { //Since we are not using Dolby Digital, fall back to the built-in Panner
this.volume = audioContext.createGain();
// Connect the sound source to the volume control.
this.bufferSource.connect(this.volume);
this.panner = audioContext.createPanner();
this.panner.setPosition(0,0,0);
// Instead of hooking up the volume to the main volume, hook it up to the panner.
this.volume.connect(this.panner);
// And hook up the panner to the main volume.
this.panner.connect(finalMixNode);
}
}
[/code]

There is quite a bit of underlying object structure I created to manage the clips, the tracks, the 3D objects, the visualizers, and the animation. Instead of walking through all that code, here’s a diagram to give you a sense of it:

CodeDiagram

There are 16 AudioClips, each with a sound file. Based on the isEC3Supported flag, the file with be either an mp4/ec-3 file or an mp3 file.

There are 8 AudioTracks, each of which can have a currently playing AudioClip.

An AudioVisualizer is the Audio track plus the connections to the visualization for that track. It has the currently playing audioClip on that track, the 3D Object, the Shader on the 3D Object, and the key-framed Animation for the 3DObject. This will be explained in more detail in Part 4, describing the visualization aspects of the project. For now, think of it as a visual track, and we’ll have 8 of them.

The ClipKit manages a kit (think drumkit) of clips available to swap in and out of an AudioTrack. This is a singleton. It manages loading the audio files, and playing and starting a particular clip or track.

The Sequencer manages which clips are to play on which tracks, and when. It has an internal data structure called “measures” which is used to time out when to play which clips. Keeping all the tracks carefully synchronized is well described here: A Tale of Two Clocks – Scheduling Web Audio with Precision.

The sequencer interface is a bit different than a typical drumkit, such as WebAudio Drum Machine – Chromium. Each column represents 4 measures, instead of maybe a sixteenth note.

re.flow_Sequencer_on_white

It is also different than the usual looping interface: no clips are looped. Instead each clip is started and plays its full length, and then stops. Different clips have different lengths, as shown above. This keeps the audio flowing and remixing in much more complex ways than a bunch of short clips. And that complexity makes can make the experience more interesting over long sessions.

The clip start times can be edited in this interface, or by clicking directly on the 3D Objects. Since each track can have multiple objects, successive clicks toggle through the clips available on that track.

Once the user has built their own sequence, they might want to save it for later, or share it with their friends! So my friend Dustin Butler created a DynamoDB instance on the Amazon Web Services, and saves the measures structure as a JSON, attached to a unique key. Hopefully this simple save and share will promote some viral sharing!

So now we have audio clips, panning around in space, organized via a sequencer, savable in the cloud. But now we need some visuals! Move on to Part 4.

 

Building re.flow Part 2: Music and Sound Production

600 100 Jason Marsh

This is the second part of a four-part set of posts about re.flow. If you missed part 1, go here.

Hear and see the end result here: http://dolby.flow.gl.

This post is about the music and surround sound production for the project.

Musical Goals and Creation

An interesting aspect of this project was how different it is than just a typical music sequencer. Sequencers usually provide per-note access to the music, which is great for musicians and hobbyists, but not so good for wider populations. Instead I wanted users to feel like they were inside the music, but couldn’t go wrong, like a ‘super DJ.’ So this is a musical sequencer, where there are no loops, rather each track plays through until it ends. This gives a basis for maintaining musical form and structure that wouldn’t be possible if I gave complete, note-by-note or bar-by-bar control to the user.

Tools Used: Ableton Live, MOTU UltraLite-mk3 Hybrid

Before starting this project, I had created a song just for fun, after getting inspired by my teenage son and his appreciation for DeadMou5 Strobe, which I called A House Built on Sand.

Desiree_small
I had found that these tracks were fairly diatonic and could be rearranged many ways, and with some tweaking could get most of them to sound good together in almost any configuration. But it was a bit too sterile, and although I had a very abstract project in mind, I wanted textures and feel of the work to have a natural, organic feel. So I hired a wonderful vocalist, Desiree Goyette Bogas and her amazing vocalist daughter Lily to humanize the piece.

LilyI’m musically inspired by Imogen Heap, who often uses her voice as a wordless instrument. I wrote out a few parts, but in the end used almost entirely the improvised wordless melodies they created on the spot. For example, the resulting “vocal melody” clip is Lily improvising a line, and then Desiree improvising a harmony part. I added a third part where Desiree sang a vocal line I wanted in tight unison with a synth line I wrote, so she was imitating the synth phrasing and scoops. But I wanted to make it tighter, so I used Ableton Live’s ability to convert her melody to MIDI, so in the end had Desiree imitating a synth which then imitated Desiree.

AbletonClipsI also wanted a variety of whispers and mouth rhythms, and they brilliantly improvised these as they listened to rough tracks. These are highly spatializable, and I loved the idea of a surprising whisper in the user’s ear.
AbletonPremixesBy the time we were done, I had dozens of instrumental parts, and maybe 50 vocal parts, and whittled them down to clips in Ableton and just experimented until I had the few that felt best together.

I premixed these vocal bits into four premixed sequences: Vocal Melody, Vocal DooDops, Whispers, and Mouth Rhythms. Check them
out one at a time at dolby.flow.gl.

I ended up with 16 clips mixed out of Ableton in stereo.

Getting to 5.1 Surround

When we get to the Web AudioContext, we’ll describe in detail that there are several ways to do spatial audio. For this project, the audio motion could be hard-wired into the sound files themselves, so each file could be created as a 5.1 mix. At this stage of using Dolby tech, this is as far as the browser can go; to be clear, it is not doing real-time positioning of each sound. This is somewhat limiting, but if your project fits within this limitation, Dolby’s real-time processing gives truly amazing spatialization.

I hadn’t done any 5.1 surround production before. So I thought I needed to upgrade some studio hardware. Let me be clear, this project was not about lots of hardware or an amazing studio. With the exception of the vocal tracks, I did the project on about $1200 of equipment (hardware) in a home studio little bigger than my typical desk space. (Software used was more expensive, unfortunately!)

Now, in fact, if you trust visual mixing interface in Adobe Audition, you don’t really need a 5.1 studio. I thought that to do a good job, I should hear it as I worked. But, that depends on your budget and quality expectations.

Tools Used: MOTU UltraLite-mk3 Hybrid, JBL LSR305 5″ Powered Studio Monitors, Home surround A/V system, Adobe Audition CC 2015

I needed to upgrade my computer audio interface to get 6 simultaneous outs. The MOTU UltraLite-mk3 Hybrid fit the bill, for about $500. And I needed matching speakers. I bought 5 JBL LSR305 5″ Powered Studio Monitors for about $135 each. And then I needed a subwoofer, so I just reused the one from my home surround A/V system. Plus wires, that is literally it, about $1200 in new outlay.

So now I have the audio hardware and a bunch of stereo audio files to bring into Adobe Audition, I’m ready to play. Note: Adobe Audition CC 2015 is the only version so far to have the ability to mix to Dolby Digital Plus (ec-3).

AuditionNewSessionI started the mixing by creating a new 5.1 multitrack session.

Audition has this cool panning interface, within which you can drag sounds around the 5 speakers. I tried that, but decided to go with the key-framing interface so that I could have better editing capability after the fact.

AuditionDegrees

I put multiple clips in a single session for convenience.

AuditionPanning

I exported them out one by one using a right click on the clip -> Export Session to New File -> Selected Clips. This gave me a 5.1 file.

Audition5.1Mix

 

Then I could save it out as an ec3 file.

AudiitonSaveAs

Wrapping the ec3 files into mp4 files

The browser cannot play this ec3 file directly, but needs it to be wrapped inside of an mp4 file. Encoding.com can do this, but I had the Dolby team help me do this. Contact the developer team at developer@dolby.com for advice on file processing, and soon there will be some good options for this. I’ll update this article for more information.

[update: Dolby Developer is now providing free encoding for registered Dolby Developer members in a lightweight, easy-to-use tool that’s entirely cloud based. http://developer.dolby.com/News/Dolby_Audio_File_Encoding__Now_Free!.aspx]

Once you’ve got the mp4 files, check them out by using the typical Microsoft media player application. Plug your computer’s HDMI into your home A/V system where you’ll be able to hear the surround sound (you may need to select the option on the A/V system to use the Dolby Surround setting). Or just use headphones, which work great too.

Now we need to get it to play in the browser, so moving on to Part 3: Web AudioContext and Sequencer.

Building re.flow Introduction: Why use Dolby Digital Plus for web-based projects?

150 150 Jason Marsh

Dolby Digital Plus gives you spatial audio for media projects beyond just the classic video use case. This is the first in a series of posts describing the creation of “re.flow” by Jason Marsh at the request of Dolby Laboratories to show off the possibilities of spatial audio with nothing other than a modern browser. As of this writing, Dolby Digital Plus is available in Microsoft Edge on Windows 10, and hopefully more browsers soon.

DolbyAudio_Vert_white_on_blackDolby Digital Plus in the browser is significant not only because it affords playback of multi-channel content when connected to a home theater or headphones on systems that provide surround virtualization, but because it provides a premium consistent audio experience to the browser by incorporating a rich set of metadata about the audio that is part of the Dolby Digital Plus codec. This way, Dolby encoded content sounds great regardless of the system it’s being played back on.

The guidelines for the project were quite simple: create a WebGL project that features Dolby Digital Plus 5.1 surround audio. WebGL is used to create highly dynamic, smoothly animating graphics like you would expect from gaming consoles.

It took me under five minutes to realize that this was the perfect project for me: I have a Masters in Music composition, I’ve been programming forever, years ago I produced binaural projects for Bose, and I’ve been doing WebGL programming full-time for the past year working on Virtual Reality (WebVR). A project made in heaven, right?

The original pitch I made was a silly sketch of planes floating in space, each synchronized to an audio track, and each with a custom visualizer for that track. Luckily, to their credit and not mine, for some reason the Dolby folks trusted me.

re.flow for Dolby Laboratories, 3D object view
re.flow for Dolby Laboratories, 3D object view

Before reading further, you have to try the end result. The actual experience is hard to describe, and since it is a web project, why bother? Use the Microsoft Edge browser on Windows 10, if you have it, or the other modern browsers including Chrome if you don’t. The audio won’t be as good without the Dolby processing in Microsoft Edge, but it will all work.

Go here now if you haven’t already done so: http://dolby.flow.gl

Did you like it? I can hope so!

This post is the first of four posts describing various aspects of the project.

 

The main aspects of this project, and tools used were:

Audio Production:

  • Producing the music, 16 clips in all, in Ableton Live and hiring the vocalists.
  • Obtaining 5.1 surround studio hardware.
  • Taking stereo mixes exported from Ableton Live into Adobe Audition CC 2015 to create the 5.1 panned ec3 files.
  • Wrapping the ec3 files into mp4 files that the browser will be able to use.

Audio Programming

  • Building the music sequencer using the Web Audio Context (all in JavaScript) to simultaneous play 8 tracks, synchronized to the millisecond
  • Building the sequencer user interface
  • Saving the sequencer to a database in the cloud

WebGL design and programming

  • Creation of the 3D scene using ThreeJS and GLAM
  • Implementation of animated textures in GLSL shaders
  • Animation engine to sync up the spatial audio mixes to the graphics
  • Pickers and interface elements within the 3D scene
re.flow for Dolby Laboratories, sequencer view

re.flow for Dolby Laboratories, sequencer view

Finally, though not within the scope of the original project, I went ahead and created a WebVR version so that I could experience it all in Virtual Reality. While this is not ready for prime time and only works on pre-beta browsers, it is a great experience and ultimately a powerful application of all these technologies.

So into the details we go:

Building re.flow Part 2: Music and Surround Sound Production

Building re.flow Part 3: Web AudioContext and Sequencer

Building re.flow Part 4: WebGL Visuals