Posts By :

Jason Marsh

Architecture of Ideas: FLOW, by Jason Marsh and Iker Jamardo

150 150 Jason Marsh

We presented at the California Academy of Sciences as part of Mediate Art Group’s SoundWave Festival in July, 2016.

“Architecture of Ideas” is an experience of symbolic information in Virtual Reality. The source content is quotes from the book “Flow: The Psychology of Optimal Experience” by Mihaly Csikszentmihalyi. The quotes gain a new double meaning: his ideas about a psychological subjective mental experience take on a new perspective when seen applied to a technological immersive experience. We see ideas floating around and through us visually and sonically using both Virtual Reality headsets and computer screens linked together in this short multi-user experience.

The multi-user experience included three Samsung GearVR headsets, synced with a server so that each user could see the avatars of two other users. The three users needed to work together to gaze in the same direction in order to proceed with each scene of the experience.

 

 

The Hall of Africa, right next to live penguins, was a great space for the event, and plenty of room for the unfortunately long line. We ran approximately one hundred users, mostly first-time VR users, through the experience.

 

 

As users gazed at panels, the sea of letters would animate in sync with a narration track. Since the goal was to engage deeply with the concepts, the narration wasn’t a single voice, but a multiple takes artistically timed and mixed, so phrases were repeated and mixed on top of each other, in tone from serious to silly. This unusual technique was surprising at first, but users quickly understood that it was by design and helped to enhance the meaning of the content by repetition with variation.

At the end, the letters swirl around the user and a giant “FLOW” fades in, larger than can fit in the field of view, so users can only read by turning to see the letters two at a time.

 

I probably spoke to about 20 people coming out of the experience. Comments included:

“At first, the multiple voices were distracting, even sounded like a bug, but then I got used to it and liked it.”

“I liked the repetition in the voice so you could really think about the words.”

“I was really engaged with the words and towards the end I realized how I was taking it in so much that I wondered if this was what it would feel like to be brainwashed.”

A life coach/corporate motivational consultant had a lot to say about intrinsic motivation and has contacted me afterward.

“The big ‘FLOW’ at the end felt mind-expanding.”

“Enjoyed the flowing letters at the end.” Most positive comments were about this.

Some users were “amazed, unlike anything” they’d ever seen.

 

My ideas for future events:

Throughput. This is and will continue to be a huge challenge in live events. The lines are just going to be long, which might in itself serve as an attractor on a trade-show floor. Perhaps we can develop media so that all that time waiting in line could be used to familiarize users with a product or service. Some good conversations could/should be had at that point.

I would like to think more about how to build expectations but not necessarily show the whole presentation on the preview monitors. Something to be said for having people experience the unexpected in VR.

Interaction with the presentation for VR virgins should be kept to an absolute minimum. Guided experiences are going to be most effective. But ways that users can pause and ask questions of the guide would be very valuable. It will be challenging to balance the competing interests of “the message is the medium” and “the message is the message.”

User testing follow-up. Get emails and phone numbers to conduct follow-up interviews. I want to see if people remember the content the next day.

A huge week of demos for re.flow

600 100 Jason Marsh

I’ve found so many avenues to present re.flow to the public, and I’ve been getting great responses to the experience in Microsoft Edge and the beta version in VR. So if you are near SF, check it out!

Two weeks ago, I showed re.flow to attendees of the HTML5 DevConf. Many “VR Virgins” got their first taste of VR in re.flow. I had people come up to me and say “I hear you have the best demo at the show.” And when finished, agree that it was indeed the best.

Last week I showed it to the Programming Club at my daughters high school. Seriously cool enthusiasm!

On Wednesday (Nov 11) I’ll be doing a lightning talk at the SFHTML5 meetup at Google. This will be very short, but a large audience.

On Thursday (Nov 12) I’m the featured speaker at the SF WebGL meetup  at Dolby Laboratories theater. I’ll describe my motivations, process, some code, and of course demo on the big screen and in VR.

Kinetech ArtsFriday and Saturday (Nov 13 and 14), Kinetech Arts presents its annual holiday dance performance at the Dance Mission (3316 24th St, San Francisco near the 24th St Mission BART station). Kinetech Arts is pushing the boundaries of dance and technology. I’ll be demoing re.flow as an installation on laptop with headphones and in VR for 45 minutes before and after each performance.

And finally, on Sunday, Nov 15, I’ll be participating in Dance Hack Day 2015 (http://dancehack.org/) at Safehouse Arts (1 Grove Center, San Francisco near the Civic Center BART station). This  is an opportunity for dancers, artists or technologists to be part of a collaborative effort to create a dance performance. Contributors start at 10 am. Doors open for the day’s performances at 7pm. An example of Kinetech Arts live production efforts (https://vimeo.com/143439488) was shown on Talk Bubble TV two weeks ago. Advance tickets available for the performances.

This is going to be crazy fun!

Building re.flow Part 4: WebGL visuals

600 100 Jason Marsh

This is the fourth part of a four-part set of posts about re.flow. If you missed part 1, go here.

Hear and see the end result here: http://dolby.flow.gl.

This post is about programming the WebGL visuals.

The connections between all the parts of the program objects are shown in this diagram:

CodeDiagramFor the visuals, I’m creating WebGL objects using ThreeJS, and GLAM is providing some handy utilities for tracking your scene graph, handling interaction with Pickers, easy key-frame Animation, and more.

Animation

Let’s start with the animation before we get to the textures. When I create an AudioClip, the animation is parameterized and included with it.

[code language=”JavaScript”]
new AudioClip( {name: "spaceshwoosh1", file: "spaceshwoosh1", track: "spaceshwoosh", label:"Space Shwoosh 1",
animation : [{//duration in musical measures , each step is 4 measures
duration:18,
object: "parent",
target: "rotation",
keyBeats : [ 1, 2.75, 4.5, 7.5, 9, 18], //keys should be in beats
values: [ //values should be in radians around center
degreesToYradians( 56, 1),
degreesToYradians( 37, 1),
degreesToYradians( 71, 1),
degreesToYradians( -171, 1),
degreesToYradians( -73, 1),
degreesToYradians( 56, 1),
] },

] }),

[/code]

AuditionDegreesI cheat to do the animation: Instead of building a series of spline curves through space (I tried this too, but it was too hard synchronize with the audio panning), I just add an extra invisible object to each 3D Object, and spin the pair around their mutual center point. That way I can move the objects in a plane around the Y axis in front of the user with nothing other than the degrees, which happens to be exactly what Audition shows in the interface!

As a good lazy programmer, I wanted to make it as easy as possible to synchronize the audio panning to the animations. So I created math to translate the same numbers in Audition directly into the key frame animator, hence the degreesToYradians function:

[code language=”JavaScript”]
/*
This function is specific to taking the values directly out of Adobe Audition’s 5.1 track panning and converting them into a horizontal plane around the user.
It inverts the values, and subtracts the 180 degrees in order to match those values.

The animator needs to be sweeping across the circle, not just animating between points. Otherwise anything that is moving relatively fast will not work right.
*/
function degreesToYradians(degrees, radius) {
var y = (Math.PI * -degrees /180) * radius ; //radians = Math.Pi * degrees /180
return {y:y}
}
[/code]

auditionKeyframespngAnd the same is true for the keyframe timing: I create a keyBeats structure so I can just type in the measure numbers that match the keyframes from Audition.

Here’s the code for creating the geometry plus the invisible partner to create the offset for the rotation:

[code language=”JavaScript”]
function addTexturedEtherPlane(trackName, texture, position, rotation, spinRadius) {
var parentObj = new glam.Object;//we use a parent object to create an offset
var obj = new glam.Object;
parentObj.addChild(obj);

var visual = addTexturedEtherPlaneVisual(trackName, texture);
var picker = addPicker(trackName);
obj.addComponent(visual);
obj.addComponent(picker);
spinRadius = spinRadius ? spinRadius : 1;

//sets the position of the pair in space
position = position? position : {x:0, y:0, z:0};
parentObj.transform.position.set( position.x, position.y, position.z + spinRadius );
obj.transform.position.set(0, 0, -spinRadius*2);

if (rotation) {
obj.transform.rotation.set(rotation.x, rotation.y, rotation.z);
}

//connects this visual object up to the AudioVisualizer object so when the
// track starts and stops, they stay synced, as well as to send the amplitude
// changes to drive the visualizer
var audioVisualizer = findAudioVisualizer(trackName);
audioVisualizer.trackObj = obj;

app.addObject(parentObj);
}
[/code]

On to the dynamic textures!

 

GLSL Textures

I had never done GLSL before this project. But, no fear, this is the age of Google. Specifically, the awesome ShaderToy. I hunted all around for a shader that I thought would give me an interesting rippling effect that wasn’t too computationally expensive. I found Ether, by nimitz, on ShaderToy. and the audio amplitude as an input to brighten the colors as the volume increased.

I wanted the textures to be ‘beautiful’ as defined by my own subjective criteria, which is like observity obscenity: I’ll know it when I see it. Actually I did want to use natural textures, hinting at hypnotic complex repetitive forms such as campfires and ocean waves. So I started with these original images I photographed and photoshopped:

tahoesolanabeachsentinelstackedGrasses

I combined the Ether shader with the animating texture maps based on code examples from the Shader Fireball by stemkoski.

Without much explanation, here is the fragment shader for the nerdiest readers:

[code language=”JavaScript”]
uniform sampler2D baseTexture;
uniform float baseSpeed;
uniform float alpha;
uniform float time;
uniform float amplitude; //audio volume as an input into the math
uniform vec3 color;

varying vec2 vUv;
varying vec3 vPosition;

float map(vec3 p){
vec3 q = p * 2.0+ time * 1.0;
return length(p+vec3(sin( time * 0.7)))*log(length(p)+1.0) + sin(q.x+sin(q.z+sin(q.y)))*0.5 – 1.;
}

void main( ){
//texture
vec2 uvTimeShift = vUv + vec2( -0.7, 1.5 ) * time * baseSpeed;
vec4 noiseGeneratorTimeShift = texture2D( baseTexture, uvTimeShift );

//ether
float resolutionY = 1.; //smaller the value, the more detailed the image
vec2 p = vec2(vPosition.xy) / resolutionY – vec2(0.5, 0.2);

vec3 color = vec3( amplitude) * 2.0;
float crispness = 4.5; //values between 0.5 and 5.5 work nicely
for(int i=0; i<=2; i++) { //iteration values between 2 and 10 work nicely
vec3 p = vec3(0, 0, 5.0) + normalize(vec3(p, -1.0)) * crispness /2.0;
float rz = map(p);
float f = clamp((rz – map(p+0.1))*0.5, -0.1, 1.0 );
vec3 colorRange = vec3(noiseGeneratorTimeShift.r, noiseGeneratorTimeShift.g, noiseGeneratorTimeShift.b)*0.8+ vec3(5.0, 4.5, 0.5) * f;
float amplitudeClamped =clamp(amplitude, 0.1, 20.0) *2.0;
color = color * colorRange + (1.0 – smoothstep(0.0 , 2.5, rz)) * 0.7 * colorRange *amplitudeClamped ;
crispness += rz; //min(rz, 1.0); //filters out some complexity
}
float calculatedAlpha = (color.b + color.g + color.r > 0.02)? alpha : 0.1 ;
gl_FragColor = vec4(color, calculatedAlpha);
}

[/code]

And here is the ThreeJS code to tie an AudioVisualizer track to the shader material:

[code language=”JavaScript”]
function addTexturedEtherPlaneVisual(trackName, texture ){
var audioVisualizer = findAudioVisualizer( trackName );
audioVisualizer.setShaders({
vertexScript:"texturedEtherVertexShader", fragmentScript:"texturedEtherFragmentShader"
} );

var waterTexture = new THREE.ImageUtils.loadTexture( texture );
waterTexture.wrapS = waterTexture.wrapT = THREE.RepeatWrapping;

audioVisualizer.shader.uniforms = new ShaderUniforms( {
color: {type: "c", value: new THREE.Color( 0xff0000 )},
baseTexture: { type: "t", value: waterTexture },
baseSpeed: { type: "f", value: 0.1 },
noiseTexture: { type: "t", value: noiseTexture },
noiseScale: { type: "f", value: 0.2 },
alpha: { type: "f", value: 0.8 },
time: { type: "f", value: 1.0 },
amplitude: {type: ‘f’, value: 1}
});

var planeShaderMaterial = new THREE.ShaderMaterial({
fragmentShader: audioVisualizer.shader.fragmentScript,
vertexShader: audioVisualizer.shader.vertexScript,
uniforms: audioVisualizer.shader.uniforms.params,
side: THREE.DoubleSide,
transparent: true,
depthWrite:true
});

var geometry = new THREE.PlaneGeometry( 2,2 );

var visual = new glam.Visual(
{ geometry: geometry,
material: planeShaderMaterial
});

return visual;
}

[/code]

Particles

Finally, while I like the clean black background, I wanted to create a sense of an even larger space, so particlesI added particles in the distance. Some are fixed particles randomly dispersed, and some are based on the “fireflies” particle engine created by Lee Stemkoski: https://stemkoski.github.io/Three.js/Particle-Engine.html . I tweaked it a bit, including a more textural texture map, but they are almost straight from his open-source code.

 

Pickers

ThreeJS doesn’t have great built-in selection mechanisms for the 3D Objects. IT is not all that trivial to select an object in 3D space. So GLAM solves this for me with a few lines of code:

[code language=”JavaScript”]
var picker = new glam.Picker;
picker.addEventListener("mouseup", selectFunction);
picker.addEventListener("touchend", selectFunction);
obj.addComponent(picker);
[/code]

That’s it!

re.flow for Dolby Laboratories, 3D object view

So, a quick review: I wrote a piece of music, found some images, and through Google searches and a simple matter of programming, created “art”. You can do it too.

 

Huge Thanks to Andrew Vaughan at Dolby Laboratories, Tony Parisi for GLAM, and also to Dustin Butler for getting the servers set up and the DynamoDB database on Amazon.

 

Follow me @jmarshworks on Twitter, and stay tuned for the Virtual Reality version.

Building re.flow Part 3: Web AudioContext and Sequencer

600 100 Jason Marsh

This is the third part of a four-part set of posts about re.flow. If you missed part 1, go here.

Hear and see the end result here: http://dolby.flow.gl.

This post is about programming the Web AudioContext and the sequencer.

How the AudioContext works together with Dolby Digital Plus

For this project, the audio motion is hard-wired into the sound files themselves, so each file can be created as a 5.1 mix. At this stage of using Dolby tech, you can’t real-time move sounds in 3D space. To be clear, it is not doing real-time re-positioning of each sound: instead it is processing the 5.1 mixes into appropriate positioning based on the audio hardware attached. If the hardware is an HDMI out to a surround system, then you get 5.1 surround sound. If you are using headphones, it uses Head Related Transfer Functions to put a sound in a precise location.   That’s why getting beyond the built-in laptop speakers are so important; Dolby’s real-time processing gives truly amazing spatialization.

I’ll touch on a the project-specific topics regarding the Web AudioContext API, but Boris Smus has laid out the full documentation, so check out his  (free) book.

The AudioContext has built-in positioning of 3D objects, described well here: Mixing Positional Audio and WebGL. It sounds to me like this is just doing basic panning within a stereo mix, without volume changes for distance. But, if you are running on browser without Dolby Digital Plus enabled, this is a good fallback, so re.flow does so.

Here’s code for querying the browser’s capabilities, both the AudioContext and the EC-3  ( the codec for Dolby Digital Plus) capability:

[code language=”javascript”]
var isAudioContextSupported;
var isEC3supported;

function checkAudioCompatibility() {
var contextClass = (window.AudioContext ||
window.webkitAudioContext ||
window.oAudioContext ||
window.msAudioContext);
if (! contextClass) {
// Web Audio API is not available.
alert("This browser won’t work! Multichannel audio is not available with this browser. \n\n" +
"Try the following browsers: \n" +
‘ Desktop: Edge, Chrome, Firefox, Opera, Safari \n ‘ +
‘ Mobile: Firefox Mobile, Firefox OS, Chrome for Android. ‘);
isAudioContextSupported = false;
return;
}
isAudioContextSupported = true;

var myAudio = document.createElement(‘audio’);
myAudio.id = "audioTest";
if (myAudio.canPlayType) {
// CanPlayType returns maybe, probably, or an empty string.

playMsg = myAudio.canPlayType(‘audio/mp4; codecs="ec-3"’);
if ( "" != playMsg){
isEC3supported = true;
console.log("ec-3 is "+playMsg+" supported");
} else {
isEC3supported = false;
}
}
}

[/code]

And here’s the sweet part, which tells the browser that we are dealing with 5.1 surround (6 channels):

[code language=”javascript”]
var finalMixNode;

function initializeAudio(){
audioContext = new AudioContext();

finalMixNode = audioContext.destination;
console.log("maxChannelCount:" + audioContext.destination.maxChannelCount);

if (audioContext.destination.maxChannelCount >=6) {
audioContext.destination.channelCount = 6;
}
[/code]

The above code will report 2 channels in Chrome (or any non-EC3 browser), and 6 channels in Microsoft Edge. If there are not 6 channels available, then we leave the channelCount to its default of 2.

[code language=”javascript”]
//if not using EC3, then use the AudioContext’s positional panner
if (!isEC3supported) {
audioContext.listener.setPosition({x:0, y:0, z:0});
}
}
[/code]

Just to complete the code as to the soft-switch based the browser capabilities, here is the code for preparing a clip to play, which I’m managing by creating an “AudioTrack” object. An AudioTrack is a track that I can swap different clips into. It has functions like prepareClip, playClip, and stopClip. Here is the prepareClip code:

[code language=”javascript”]
AudioTrack.prototype.prepareClip = function ( buffer) {
if (this.isPlaying && this.bufferSource) {
this.bufferSource.stop();
}
this.isPlaying = true;
this.bufferSource = audioContext.createBufferSource();
this.bufferSource.buffer = buffer;
this.bufferSource.loop = false;

if (isEC3supported) {
this.bufferSource.connect(masterGainNode);
} else { //Since we are not using Dolby Digital, fall back to the built-in Panner
this.volume = audioContext.createGain();
// Connect the sound source to the volume control.
this.bufferSource.connect(this.volume);
this.panner = audioContext.createPanner();
this.panner.setPosition(0,0,0);
// Instead of hooking up the volume to the main volume, hook it up to the panner.
this.volume.connect(this.panner);
// And hook up the panner to the main volume.
this.panner.connect(finalMixNode);
}
}
[/code]

There is quite a bit of underlying object structure I created to manage the clips, the tracks, the 3D objects, the visualizers, and the animation. Instead of walking through all that code, here’s a diagram to give you a sense of it:

CodeDiagram

There are 16 AudioClips, each with a sound file. Based on the isEC3Supported flag, the file with be either an mp4/ec-3 file or an mp3 file.

There are 8 AudioTracks, each of which can have a currently playing AudioClip.

An AudioVisualizer is the Audio track plus the connections to the visualization for that track. It has the currently playing audioClip on that track, the 3D Object, the Shader on the 3D Object, and the key-framed Animation for the 3DObject. This will be explained in more detail in Part 4, describing the visualization aspects of the project. For now, think of it as a visual track, and we’ll have 8 of them.

The ClipKit manages a kit (think drumkit) of clips available to swap in and out of an AudioTrack. This is a singleton. It manages loading the audio files, and playing and starting a particular clip or track.

The Sequencer manages which clips are to play on which tracks, and when. It has an internal data structure called “measures” which is used to time out when to play which clips. Keeping all the tracks carefully synchronized is well described here: A Tale of Two Clocks – Scheduling Web Audio with Precision.

The sequencer interface is a bit different than a typical drumkit, such as WebAudio Drum Machine – Chromium. Each column represents 4 measures, instead of maybe a sixteenth note.

re.flow_Sequencer_on_white

It is also different than the usual looping interface: no clips are looped. Instead each clip is started and plays its full length, and then stops. Different clips have different lengths, as shown above. This keeps the audio flowing and remixing in much more complex ways than a bunch of short clips. And that complexity makes can make the experience more interesting over long sessions.

The clip start times can be edited in this interface, or by clicking directly on the 3D Objects. Since each track can have multiple objects, successive clicks toggle through the clips available on that track.

Once the user has built their own sequence, they might want to save it for later, or share it with their friends! So my friend Dustin Butler created a DynamoDB instance on the Amazon Web Services, and saves the measures structure as a JSON, attached to a unique key. Hopefully this simple save and share will promote some viral sharing!

So now we have audio clips, panning around in space, organized via a sequencer, savable in the cloud. But now we need some visuals! Move on to Part 4.

 

Building re.flow Part 2: Music and Sound Production

600 100 Jason Marsh

This is the second part of a four-part set of posts about re.flow. If you missed part 1, go here.

Hear and see the end result here: http://dolby.flow.gl.

This post is about the music and surround sound production for the project.

Musical Goals and Creation

An interesting aspect of this project was how different it is than just a typical music sequencer. Sequencers usually provide per-note access to the music, which is great for musicians and hobbyists, but not so good for wider populations. Instead I wanted users to feel like they were inside the music, but couldn’t go wrong, like a ‘super DJ.’ So this is a musical sequencer, where there are no loops, rather each track plays through until it ends. This gives a basis for maintaining musical form and structure that wouldn’t be possible if I gave complete, note-by-note or bar-by-bar control to the user.

Tools Used: Ableton Live, MOTU UltraLite-mk3 Hybrid

Before starting this project, I had created a song just for fun, after getting inspired by my teenage son and his appreciation for DeadMou5 Strobe, which I called A House Built on Sand.

Desiree_small
I had found that these tracks were fairly diatonic and could be rearranged many ways, and with some tweaking could get most of them to sound good together in almost any configuration. But it was a bit too sterile, and although I had a very abstract project in mind, I wanted textures and feel of the work to have a natural, organic feel. So I hired a wonderful vocalist, Desiree Goyette Bogas and her amazing vocalist daughter Lily to humanize the piece.

LilyI’m musically inspired by Imogen Heap, who often uses her voice as a wordless instrument. I wrote out a few parts, but in the end used almost entirely the improvised wordless melodies they created on the spot. For example, the resulting “vocal melody” clip is Lily improvising a line, and then Desiree improvising a harmony part. I added a third part where Desiree sang a vocal line I wanted in tight unison with a synth line I wrote, so she was imitating the synth phrasing and scoops. But I wanted to make it tighter, so I used Ableton Live’s ability to convert her melody to MIDI, so in the end had Desiree imitating a synth which then imitated Desiree.

AbletonClipsI also wanted a variety of whispers and mouth rhythms, and they brilliantly improvised these as they listened to rough tracks. These are highly spatializable, and I loved the idea of a surprising whisper in the user’s ear.
AbletonPremixesBy the time we were done, I had dozens of instrumental parts, and maybe 50 vocal parts, and whittled them down to clips in Ableton and just experimented until I had the few that felt best together.

I premixed these vocal bits into four premixed sequences: Vocal Melody, Vocal DooDops, Whispers, and Mouth Rhythms. Check them
out one at a time at dolby.flow.gl.

I ended up with 16 clips mixed out of Ableton in stereo.

Getting to 5.1 Surround

When we get to the Web AudioContext, we’ll describe in detail that there are several ways to do spatial audio. For this project, the audio motion could be hard-wired into the sound files themselves, so each file could be created as a 5.1 mix. At this stage of using Dolby tech, this is as far as the browser can go; to be clear, it is not doing real-time positioning of each sound. This is somewhat limiting, but if your project fits within this limitation, Dolby’s real-time processing gives truly amazing spatialization.

I hadn’t done any 5.1 surround production before. So I thought I needed to upgrade some studio hardware. Let me be clear, this project was not about lots of hardware or an amazing studio. With the exception of the vocal tracks, I did the project on about $1200 of equipment (hardware) in a home studio little bigger than my typical desk space. (Software used was more expensive, unfortunately!)

Now, in fact, if you trust visual mixing interface in Adobe Audition, you don’t really need a 5.1 studio. I thought that to do a good job, I should hear it as I worked. But, that depends on your budget and quality expectations.

Tools Used: MOTU UltraLite-mk3 Hybrid, JBL LSR305 5″ Powered Studio Monitors, Home surround A/V system, Adobe Audition CC 2015

I needed to upgrade my computer audio interface to get 6 simultaneous outs. The MOTU UltraLite-mk3 Hybrid fit the bill, for about $500. And I needed matching speakers. I bought 5 JBL LSR305 5″ Powered Studio Monitors for about $135 each. And then I needed a subwoofer, so I just reused the one from my home surround A/V system. Plus wires, that is literally it, about $1200 in new outlay.

So now I have the audio hardware and a bunch of stereo audio files to bring into Adobe Audition, I’m ready to play. Note: Adobe Audition CC 2015 is the only version so far to have the ability to mix to Dolby Digital Plus (ec-3).

AuditionNewSessionI started the mixing by creating a new 5.1 multitrack session.

Audition has this cool panning interface, within which you can drag sounds around the 5 speakers. I tried that, but decided to go with the key-framing interface so that I could have better editing capability after the fact.

AuditionDegrees

I put multiple clips in a single session for convenience.

AuditionPanning

I exported them out one by one using a right click on the clip -> Export Session to New File -> Selected Clips. This gave me a 5.1 file.

Audition5.1Mix

 

Then I could save it out as an ec3 file.

AudiitonSaveAs

Wrapping the ec3 files into mp4 files

The browser cannot play this ec3 file directly, but needs it to be wrapped inside of an mp4 file. Encoding.com can do this, but I had the Dolby team help me do this. Contact the developer team at developer@dolby.com for advice on file processing, and soon there will be some good options for this. I’ll update this article for more information.

[update: Dolby Developer is now providing free encoding for registered Dolby Developer members in a lightweight, easy-to-use tool that’s entirely cloud based. http://developer.dolby.com/News/Dolby_Audio_File_Encoding__Now_Free!.aspx]

Once you’ve got the mp4 files, check them out by using the typical Microsoft media player application. Plug your computer’s HDMI into your home A/V system where you’ll be able to hear the surround sound (you may need to select the option on the A/V system to use the Dolby Surround setting). Or just use headphones, which work great too.

Now we need to get it to play in the browser, so moving on to Part 3: Web AudioContext and Sequencer.

Building re.flow Introduction: Why use Dolby Digital Plus for web-based projects?

150 150 Jason Marsh

Dolby Digital Plus gives you spatial audio for media projects beyond just the classic video use case. This is the first in a series of posts describing the creation of “re.flow” by Jason Marsh at the request of Dolby Laboratories to show off the possibilities of spatial audio with nothing other than a modern browser. As of this writing, Dolby Digital Plus is available in Microsoft Edge on Windows 10, and hopefully more browsers soon.

DolbyAudio_Vert_white_on_blackDolby Digital Plus in the browser is significant not only because it affords playback of multi-channel content when connected to a home theater or headphones on systems that provide surround virtualization, but because it provides a premium consistent audio experience to the browser by incorporating a rich set of metadata about the audio that is part of the Dolby Digital Plus codec. This way, Dolby encoded content sounds great regardless of the system it’s being played back on.

The guidelines for the project were quite simple: create a WebGL project that features Dolby Digital Plus 5.1 surround audio. WebGL is used to create highly dynamic, smoothly animating graphics like you would expect from gaming consoles.

It took me under five minutes to realize that this was the perfect project for me: I have a Masters in Music composition, I’ve been programming forever, years ago I produced binaural projects for Bose, and I’ve been doing WebGL programming full-time for the past year working on Virtual Reality (WebVR). A project made in heaven, right?

The original pitch I made was a silly sketch of planes floating in space, each synchronized to an audio track, and each with a custom visualizer for that track. Luckily, to their credit and not mine, for some reason the Dolby folks trusted me.

re.flow for Dolby Laboratories, 3D object view
re.flow for Dolby Laboratories, 3D object view

Before reading further, you have to try the end result. The actual experience is hard to describe, and since it is a web project, why bother? Use the Microsoft Edge browser on Windows 10, if you have it, or the other modern browsers including Chrome if you don’t. The audio won’t be as good without the Dolby processing in Microsoft Edge, but it will all work.

Go here now if you haven’t already done so: http://dolby.flow.gl

Did you like it? I can hope so!

This post is the first of four posts describing various aspects of the project.

 

The main aspects of this project, and tools used were:

Audio Production:

  • Producing the music, 16 clips in all, in Ableton Live and hiring the vocalists.
  • Obtaining 5.1 surround studio hardware.
  • Taking stereo mixes exported from Ableton Live into Adobe Audition CC 2015 to create the 5.1 panned ec3 files.
  • Wrapping the ec3 files into mp4 files that the browser will be able to use.

Audio Programming

  • Building the music sequencer using the Web Audio Context (all in JavaScript) to simultaneous play 8 tracks, synchronized to the millisecond
  • Building the sequencer user interface
  • Saving the sequencer to a database in the cloud

WebGL design and programming

  • Creation of the 3D scene using ThreeJS and GLAM
  • Implementation of animated textures in GLSL shaders
  • Animation engine to sync up the spatial audio mixes to the graphics
  • Pickers and interface elements within the 3D scene
re.flow for Dolby Laboratories, sequencer view

re.flow for Dolby Laboratories, sequencer view

Finally, though not within the scope of the original project, I went ahead and created a WebVR version so that I could experience it all in Virtual Reality. While this is not ready for prime time and only works on pre-beta browsers, it is a great experience and ultimately a powerful application of all these technologies.

So into the details we go:

Building re.flow Part 2: Music and Surround Sound Production

Building re.flow Part 3: Web AudioContext and Sequencer

Building re.flow Part 4: WebGL Visuals

Announcing: Flow!

600 100 Jason Marsh

We have a public presence on the web!

Flow has been working on Virtual Reality since January, 2015. We’ve been a small team creating VR experiments with rapid iterations to learn what was the best/most powerful way to present information in a fully interactive 3D space.

We originally were named Third Eye, and in October we renamed the company to “Flow.”

FLOW_logo_white-large

Tony Parisi and Jason Marsh started Third Eye to explore a new category of VR. Not VR gaming, not cinematic VR, not Social (avatar-based) VR, and not medical or enterprise. There are no road maps to follow, just a vision of what VR/AR is going to look like day-to-day. What will it VR like once we get past the ‘demo phase.’

Jason is steering the ship, and Tony is still involved as a board member and close advisor. Dustin is cranking on the web stack.

We’ve been doing demos to potential investors, VR professionals we meet, and most anyone who asks! Let us know what your interest is.

VR as a new way to see the world

1245 208 Jason Marsh

Republished from MarshWorks blog, by Jason Marsh:

My friend Issac Cohen (also goes by cabbi.bo), the brilliant digital interactive artist, spoke at a WebVR meetup this Spring, and while he waved his arms (with a LeapMotion) to interactively ripple a gorgeous metallic face-like blob, he said something like:

[themify_quote]”I want people to have an amazing experience in VR, and come out, look around at the real world, and be astounded at the Ultra-HiDef  detail and wonderfulness of being in real reality. I want VR to change the way they see the world in real life and notice things they always used to overlook.”[/themify_quote]

So I gotta say he succeeded, at least with my family, and he did it in a browser, without VR.

I spent an hour going through his new storybook, Enough, first with my wife, and then the next day with my son and his girlfriend (college-aged). We were all stunned.

And then we went camping and hiking in the high Sierra for the weekend. All weekend long, the kids would say: look at how those ripply clouds are like the sandy ocean floor in Enough.

enough

And, look at the color of the California Fuschia, and how it reminds me of the color of “Sol” a creature from the story.

enough2

These still shots do not do it justice. And watching it as a video doesn’t do it justice either. The interactive motion of the screen elements is such an important part of the experience: each individual letter is animated based on mouse movement.

I think that Issac is establishing a new art form. He calls it a storybook, and it is that. But it is unlike any storybook.

Instead of NetFlix tonight, put your family in front of a big monitor, or hook your laptop up to your TV, and simply go to the website: http://cabbi.bo/enough/ (it’s all WebGL and runs in the browser). We swapped off reading the text aloud and playing with the mouse.

What he’ll be able to do with translating it to VR is crazy… I hope he’s working on it.

And it changed our weekend in the mountains.

What is an information architect?

600 100 Jason Marsh

Republished from MarshWorks.com blog, by Jason Marsh:

I call myself an information architect. I don’t use it exactly the same way as Wikipedia defines it, but since there are similarities, I’ll start there:

Information architecture (IA) is the structural design of shared information environments; the art and science of organizing and labelling websites, intranets, online communities and software to support usability and findability; and an emerging community of practice focused on bringing principles of design and architecture to the digital landscape. Typically, it involves a model or concept of information which is used and applied to activities that require explicit details of complex information systems. These activities include library systems and database development. –Wikipedia

I read this definition as more of a software systems designer, kind of similar to a software architect.

This doesn’t quite fit my definition, because I’m not a hard-core programmer (although I did start programming 45 years ago at the age of 6 on a 6502 Jolt kit computer: pre-Apple I.)  I didn’t study Computer Science in school. I’ve been programming my entire career, but I’ve always focused on the user experience, and have focused exclusively on client-side software. This has guided me more toward script-based platforms, and since the Internet, I’ve focused on JavaScript, Flash, Adobe Flex, CSS, XML, XSLT… you get the idea. I just believe that focusing on the user experience is my unique value-add.

But my definition of “information architecture” transcends the realm of software.

Here is my definition:

Ideas have shape. A mental model of an idea can be created in a single consciousness, and transferred to others, though a process of discovery, diagramming, analyzing, and discussing. Sometimes an idea feels like it can be fully documented, and sometimes big areas of the idea are left to other parties to figure out. But architected ideas include the ‘why’: the big picture of where this concept fits into the next larger system of thought. And they include the concrete: the details of that make this realizable in the world, and therefore give it credibility. For years, I’ve described ideas like a blob, floating above the table we share when we discuss and deliberate, listen and argue. We can spin the blob, seeing it from different perspectives, flipping the axis, or discovering the N-dimensions that may be required to see it in adequate clarity.

I would agree with the Wikipedia definition in these aspects:

  • These ideas are usually complex (and real-world.)
  • Labeling is key: the process of categorizing, organizing aspects of an idea into mental buckets, is vitally important.

Ideas that need architecting can be seen as ‘problems’ needing solutions, in which case the both the problem and the solution may need to be analyzed and ‘architected’, and then the solution needs to fit the problem like a 3D Tetris.

Ideas that need architecting transcend a single consciousness. The first goal is to build a shared mental model, and that involves the concretizing of the idea into documentation.

Ideas that need architecting can’t be merely described in prose. Diagrams are key. Ideally diagrams should be able to ‘zoom’ from the macro view to the micro view, the why and the concrete, and then back to the macro view quickly to re-affirm the big picture. I use Prezi for this: not as a slideshow replacement, but as an interactive, sharable whiteboard with infinite zoom capabilities (I should do another post about this!) I also use colored pens to draw words and lines on pure white sheets of loose paper. The colors convey multiple dimensions, as well as provide a more memorable visualization. Words are key, even more important than little stick-figures or other visual representation. I say that a word is worth a thousand pictures. Put a word on a piece of paper in the middle of the table, and soon everyone is pointing to it and saying “this,” and everyone sees how “this” is connected to “that.” A new shorthand language appears spontaneously, and the team is actively developing a shared mental model.

Every discussion of consequence that uses language can be a candidate for information architecture, including politics, religion, history, and the arts. As an anti-example, pure art transcends language: for example, words only hint at the emotive expression of dance, and Taoism explicitly transcends words. Love transcends language so it is also not a good candidate, although that is an interesting idea: maybe if more of us had a clear mutual understanding our deepest relationship, many practical long-term benefits are possible.

Sidebar: I’ve realized I’ve violated my rule of information architecture! I described it in prose without diagrams and examples! Most of my best info design has been with medical customers with proprietary information, so I can’t share them as-is. I’ll have to rectify this in my next post!

Here is one trivial example of the micro/macro view: a photograph I took this morning:

P1270937_edit

Lupine on Table Mountain

What do I like about this photo? The crisp detail of a single flower is a micro view, very real and concrete. But it has its place in a vast field of flowers, which would merely be a wash of color without the micro view. And it has a relationship to the macro view (not as in “macro photographic lens”, but as in big-picture view) which is the moment of sunrise, which gives a sense of place and meaning to the flower. Your eye naturally flows around the photo: back and forth between the flower in focus, and the sun, and the tree, and the other close-up flowers, and the sun, circling around and around.

The micro view gives credibility to all the other flowers: you trust that you can understand each one of them without needing to zoom to each element. A manager will only look at one detail in a big picture: when she finds that the detail is adequate, then she assumes that all the other details are in good order and leaves them to her subordinates because she can trust them on the basis of the single well-documented detail.

The macro view becomes the story within which I can mentally file away the detail. In a Montessori classroom, a timeline rings the room’s walls right up against the ceiling. And every history lesson starts with students finding its time period on the timeline. This gives a mental filing-place for the lesson, and relationships are seen and remembered. An abstract history lesson without a proper context would be confused and forgotten, similar to what happens in many less-inspired classrooms.

 

By this point, I’ve spent many years doing information architecture. I do aspects of this at every meeting: if something is worth discussing, it is worth documenting and understanding. I try to see every meeting as an opportunity to create a shared, concrete, meaningful mental model. And that is how I define Information Architecture.