Signing off now. Thanks again for joining my AMA and asking great questions. If you want to keep in touch - I'm @PhilipRosedale on Twitter, and my company is @HighFidelityXR.

_______________________________________________________________________________________________

Hi Reddit!

I am the founder of the virtual civilization Second Life, populated by one million active users, and am now CEO and co-founder of High Fidelity — which has just released a real-time spatial audio API for apps, games, and websites. If you want to check it out, I’d love to hear what you think: highfidelity.com/api

High Fidelity’s Spatial Audio was initially built for our VR platform — we have been obsessive about audio quality from day one, spending our resources lowering latency and nailing spatialization.

Ask me about immersive spatial audio, VR, virtual worlds and spaces, avatars, and … anything.

(With me today I have /u/MaiaHighFidelity and /u/Valefox to answer technical questions about the API, too.)

Proof: https://twitter.com/philiprosedale/status/1362453056223285251?s=20

Comments: 298 • Responses: 54  • Date: 

Marcus_Llewellyn152 karma

Would you like to return to investing your time with virtual reality and virtual worlds again? What would you do differently?

PhilipRosedale312 karma

I'll be working on virtual worlds in one form or another until I die.

I'd like to see a physics/simulation model that creates some sort of low-level digital 'atoms' that are used to build everything, using a decentralized compute model.

DrNarf64 karma

Philip, I have tried your High Fidelity 3D sound many times, and it's stunning. I wonder, however, if you still have dreams/plans for another metaverse? Thanks. DrFran

PhilipRosedale45 karma

I love virtual worlds and plan to keep working on them in one form or another until I die!

mrvladnov63 karma

Hi Philip,

What are your thoughts on the dapp economy right now? From your experience in Second Life, do you see a world in Earth's future where people sustain themselves by making income as worker-consumers in a virtual world fulltime?

PhilipRosedale91 karma

Yes: A few thousand people make their living today in Second Life, and that is a place that typically has around 50,000 people online. So absolutely people will sustain themselves increasingly as time goes by from work in virtual worlds.

Grimegang_Cody47 karma

Whats going to be the next big step in vr?

PhilipRosedale145 karma

Typing at normal speed (probably by using see-through camera to show you a real keyboard) is the biggest change that would make VR devices usable for general computing.

Next would be reduced weight and greater comfort for longer sessions.

The 'big step' is simply to get to something that everyone is comfortable using.

Trigit_VR46 karma

Do you think a shift in consciousness (social or spiritual) could be triggered through VR? Did you see any signs of something like that yet?

PhilipRosedale103 karma

The internet (and VR specifically) has the potential to deeply connect people. But it can also disconnect and separate people. The difference is in the choices we all make about the products we build. But sometimes maximizing profit does not maximize public good. We are going to have to make the decision to use it for good, at a cost.

Marcus_Llewellyn30 karma

I see that you have a client library in JavaScript available. Do you plan on making it just as easy for integration with tools such as Unity?

PhilipRosedale30 karma

Yep, we are working right now on Unity and iOS. Coming soon...

wrapped_in_clingfilm30 karma

Hi Philip - we met many years ago on SL, I was a fairly well-known content creator and we chatted on a few occasions. Whatever happened to Starax? And what is the future of SL? (I left many years ago, but notice its still going).

PhilipRosedale56 karma

I don't know what happened to Starax, but I can certainly say that his Wand was one of the coolest things I had ever seen, and it gave me real hope that Second Life was going to make it. His wand was something like $50USD (if I remember), and it was the gift everyone was dreaming of getting in SL one Holiday season. Who wouldn't want a magic wand that makes a 3D locomotive appear and drive through you and your friend if you happened to type "train".

XRfuture21 karma

Always respected HiFi's user-focused stance on privacy vs. data-mining. Wonder where you think the future in that area is heading, and do you ever see working together with other cool companies like Mozilla, DDDGo, Proton, etc. to create any alternative Social Media platforms?

BTW: High Fidelity sounds GREAT using Audeze Mobius headphones w/head-tracking!

PhilipRosedale17 karma

Thank you. Yes I bet a number of companies will need to keep working together on standards and approaches for privacy as well as new protocols that might create new kinds of social networks. Mozilla has done lots of great work.

Thx re Audeze - will try to find a pair!

PhilipRosedale20 karma

Signing off now. Thanks again for joining my AMA and asking great questions. If you want to keep in touch - I'm @PhilipRosedale on Twitter, and my company is @HighFidelityXR.

CleanAcorn7716 karma

Hi Philip! Thanks for doing this AMA.

In terms of future game development, where do you see VR heading in the next few years? At what point will AAA games become mainstream on VR?

Thanks again!

PhilipRosedale18 karma

VR HMD's have to get mainstream first, which I think means comfortable for long-term use and in use by a diverse range of age/sex/race/global people - and they aren't there yet. I think it is going to take another 5 years or so.

SenatusSPQR16 karma

Hey Philip. Cool to see you doing an AMA. If you were rebuilding a virtual world today, would you be including cryptocurrencies in there? What's your take on the timeline to making virtual worlds so realistic that people start being unable to distinguish them from reality?

PhilipRosedale52 karma

Yes I would use cryptocurrencies, but NOT using proof-of-work as the consensus mechanism, since this puts the environment at risk. The consensus mechanism cannot trade units of the currency for electrical power - that is a recipe for global disaster. Fortunately there are many other ways of maintaining a distributed record-keeping system.

DaemonBlackflag15 karma

Hey Philip! I'm an entrepreneur since 18 (33 now), and most of my successful business ventures reside within Second Life (Fennux, Fawns, Kreatures.)

Do you ever have a vision for something that no one else seems to really grasp? Maybe you find it hard locating others that share that same vision?

If so, how do you deal with that? (Clearly you still make things happen.)

If that's not an issue, what's your secret to attracting those that share your vision? Thanks!

PhilipRosedale22 karma

Don't let it get you down. Sometimes you can see something (or an opportunity) that others can't see. When I was younger I would blame myself for being unable to communicate it, or get mad at people for not hearing me, or funding me, or whatever.

Now I relax and realize that we all have gifts to offer, we're all different, my gift is (sometimes) to have these strange ideas, and it's OK if those gifts are not always accepted.

DrNarf14 karma

Why not do some of this AMA in High Fidelity? It would be a great way to demo the tool for others.

PhilipRosedale16 karma

Great idea - We will do that toward the end so it's not distraction to typing fast!

Ifitirondick12 karma

Do you have any interest in augmented reality projects or purely virtual?

PhilipRosedale41 karma

I think AR and VR are very different. Mostly I think about VR.

AR has some tremendous technical and privacy challenges. For example, I can't tell exactly (better than GPS) where I am standing unless my always-on-AR-camera is uploading my 3D surroundings to the cloud to match up the surfaces. Privacy implications there are chilling - we are not going to want to live in that world unless things are designed very safely.

AR is also further out than VR because of mobile device battery life and CPU/GPU.

twoworldsin112 karma

Second Life seemed to follow a definite hype cycle in 2006-2007. After a lot of it died down, "influencers" and brands seemed to pull out and bad press replaced the hype. What lessons have you and Linden Lab learned about that experience?

PhilipRosedale19 karma

Certain ideas (like the chance that people might leave earth for a digital world) seem to evoke a degree of FOMO that greatly exceeds the baseline utility of the idea. I think that was the case with Second Life. We didn't spend anything on marketing before 2008 or so, and never have spent that much.

I see the same thing happening right now with Clubhouse - everyone is afraid of missing out on something without knowing yet what the something is.

I don't think there is any action item here, just an observation.

jsimoncalle9 karma

Hello Phillip,

Taking into account that current audio production is done thinking about descrete channels, what are the steps do you believe the industry has to take to make spatial audio a standard in audio production? New formats like MPEG-H across the whole chain? Better tools?

Thanks!

PhilipRosedale13 karma

Sources of sound should be mono, unprocessed. The spatialization needs to happen on the server, so that everyone can essentially 'sample' the experience at a given point. Existing low-latency compression formats like Opus are adequate for compression.

donkeylipswhenshaven9 karma

Hi Phillip, Perhaps this is not the place for politics, but would you vote for The Boy Mayor of Second Life in a Second Term? Why or why not?

PhilipRosedale11 karma

"Don't hate the game, hate the player." :)

MrWeirdoFace8 karma

Philip. A couple open source projects like Vircadia, and Tivoli Cloud are attempting to preserve and expand upon what was started with the High Fidelity 3d engine. What are your hopes for these efforts, and do you have any advice for the teams working on these projects?

PhilipRosedale9 karma

I'm excited by their work! There are so many experiments needing to happen in shared 3D spaces with live building, and it is great to see the original High Fidelity code being useful for some of those experiments.

bubblesort6 karma

I love virtual worlds, and I've been an SL resident since the early years. My work study in college was in SL.

This probably sounds like a dumb question, but... one thing I don't get is why HMD adoption is such a big deal.

I've used HMDs, I think they are cool, but I can't think of any content I really want to use an HMD for. Maybe a flight simulator, where my head can be another axis of control for guns as my hands control the space ship, but other than that... I just don't think it's necessary.

Why should I make the jump from developing content in SecondLife to develop for HMD platforms?

PhilipRosedale6 karma

You should wait until there is a large diverse set of people using HMDs. There isn't yet. HMDs need to be accessible and comfortable for everyone: across age, gender, and race. They aren't yet.

Humming_Pera5 karma

Hi Philip (and company),

I've been sporadically keeping up with High Fidelity for a few months, and attended a short chat with Philip. I deeply appreciate your work with foregrounding an audio experience!

I am working with colleagues on creating, rehearsing and presenting music/soundart works through separate direct url connected via audio vst plugins into DAWs, simultaneously connecting on Skype or Zoom for communication. (We turn off the video when playing - much better for sonic intimacy.)

Our next step is using spatialization software with this configuration. I wonder if it is possible to experiment in High Fidelity - that is, could an ensemble spread themselves in a HF location, each broadcasting our sound, and have the listeners move in and around us? It would be interesting to create things specifically for the technology, rather than present existing things, to be honest. (I also work in Second Life, and gave you some links via a Twitter convo about the Avatar Orchestra.)

Thanks for your work.

PhilipRosedale4 karma

Yes - an ensemble can play into a High Fidelity space and let people move around among the performers. It is a fascinating experiment and I'd encourage you to try it (and invite me please!).

PhilipRosedale4 karma

Here is our demo space where you can move around and hear multiple people and music... https://map.highfidelity.com/x28jmHHCob/?mapJSON=https%3A%2F%2Fbackgrounds.highfidelity.com%2Fzaru6.json&x=21.72&y=33.68&o=1.41

MagicSPA4 karma

If we haven't already done so, will we ever reach a point where some humans spend so much time in VR - including working and socialising - that they will come to perceive the virtual world as their seat of reality, and only grudgingly interact with the real world as little as they can get away with?

I mean, like the human/VR equivalent of a whale, that needs to surface now and then to breathe, but which doesn't consider the realm above the water to be its home.

PhilipRosedale12 karma

My answer when I was younger would have been unreservedly "Yes, of course!". And there are so many ways that Virtual Worlds can extend our lives - new jobs, new friends, new kids of artistic expression. The list goes on and on, and VR devices (once they become mainstream) only make it bigger.

As I've gotten older and wiser, I'm struck by how deeply connected we are to the physical world we exist in, as well as to our precious planet. Our brains and bodies have evolved in a way that connects us perfectly to the world around us - things like touch and proprioception are incredibly important to most of us and may always lie beyond the reach of VR. Also - as we are waking up to most recently - the game called 'Planet Earth' is a special game we all have to play together, with a finite set of resources. I think there is something very powerful and affirming about that idea. If we are to build worlds (perhaps the vaunted 'metaverse') beyond this one, I bet that the idea of sharing a finite set of things (as opposed to just building into infinity with infinite resources) is part of what will make them 'real'.

AnotherDrunkCanadian4 karma

I own a vr arcade in Tahiti. The majority of people here can't afford / don't have room for a vr setup at home. How can we create a world for them to jump into for a while, leave and come back a few weeks later?

I mean shooting zombies and crap is cool for a first timer but if I want them to keep coming back, they should have a virtual home to revisit over and over again.

PhilipRosedale9 karma

VRChat or RecRoom or Tivoli or Vircadia (latter two are based on High Fidelity) are some possibilities for creating stable spaces like you suggest. I agree it seems like a great direction for an arcade.

SirNicksAlong4 karma

What do you think of Decentraland?

PhilipRosedale3 karma

It's an amazing experiment to watch! For simulation, I definitely think that long-term there will be a distributed compute model that makes the physics of the virtual world some sort of inviolable consensus. I'd like to see Decentraland add our spatial audio API!

dale_glass4 karma

Hey there! Big fan of your work.

There's one thing I wondered for a while: how on earth did you manage to make High Fidelity (the original, 3D one) happen? I mean, it was a platform that was fully open both on the client and server side, heavily distributed, with a cryptocurrency and made in such a way that made central control difficult. I have a very hard time imagining how one makes a business pitch for that. How do you convince people to invest in something you're allowing people to just take and do whatever they please with?

But however it happened though I'm very happy it did -- there's some excellent work in there, and we hope to keep it going.

PhilipRosedale7 karma

Thanks! For a while there (2013-2019) it seemed like we would have hundreds of millions of people with immersive VR devices by 2020. That was a sound thesis for investing in a project like High Fidelity - especially given the success of Second Life in delivering many of those things to the desktop.

The failure of the VR HMD to reach mass-market (I think it is going to take about another 5 years) made it too long a project to keep pushing forward. And this kind of open / social experience would require mass adoption of the headsets.

theanine3D3 karma

Godot Engine, a fully open source 3D game engine, has really matured in the last several years and even now supports VR. Do you see any viability in an engine like this for developing a social VR platform?

PhilipRosedale3 karma

I haven't looked at Godot. What I think matters most in the 'engine' for Social VR is the low-level simulation rules, the communication affordances, the support for great scale, and the ability to build live. Graphical detail seems much less important.

VRMeg3 karma

Hello Philip,

Years ago I saw you do an interview where you created an avatar in AOL and jumped into the Sims I think. But your goal at the time was to make it so people could have avatars to jump from site to site. In second life, people jumped from unique created world like the sims to sim. And in high fidelity it was more jumping specifically to a URL. Are your creations getting you closer to that end goal?

Also, in High fidelity with spatial audio, how do you make a grid with entities horizontal and vertical where depending on where an avatar stands, they would sound different in reference to another avatar. Or the sound of a fountain as you get closer to it gets louder. These are all perceptions to make things seem more real as we know it. What will be the new reality we will experience in a virtual world. i.e. what is next?

PhilipRosedale7 karma

The hard question, I think, is why we will want to jump between these worlds or share a larger world. Most games are intentionally holistic - it doesn't really make sense to jump between them. You don't want to drive a car from GTA into Among Us. So the big question about "what is the metaverse" is what sort of space(s) we want to share, and why? I don't think anyone has very good answers to this, myself included!

Usernombreuno3 karma

Thanks for taking questions, Philip. I’m curious what you think about the potential for using VR spaces in education, particularly in support of students with exceptional needs (e.g. non verbal, students with autism)? If a teacher and student(s) can have avatars interacting in a virtual world, I presume we could eliminate a number of limitations we have in the real world. Do you have any insight into who might be already doing this well or where the technology is going in terms of serving students with special needs?

PhilipRosedale5 karma

There were remarkable studies done in SL with adults with autism that you might want to look up. A guy named John Lester, aka Pathfinder Linden did some of the early work, but you will find numerous academic papers on the subject.

zenzealot3 karma

What surprised you about how people acted in second life?

PhilipRosedale15 karma

Lots! But I was deeply moved by how people came to know each other through their avatars, despite being unable to hear or see each other. I built Second Life with a focus on the 'lego kit' / Minecraft dream of building a simulated world and seeing what people would make, but I came to regard the connections between people it enabled as the most important thing.

Badger_Druid3 karma

What good things do you do with your wealth, power and influence in the real world? Or are you just another member of the self preservation society?

PhilipRosedale8 karma

I give to local charities to help people near San Francisco get by during and beyond the pandemic.

I am also devoted to using both money (funding) and my time and skills to make virtual spaces that are of a benefit to people, particularly in addressing racism and polarization.

I think that it is vital that those who have any degree of influence or power use them now and use them carefully to help steer the world toward greater compassion and connection.

goblix3 karma

Have you ever watched Sword Art Online?

PhilipRosedale3 karma

Yep. The idea of being 'trapped in the simulation' is resonant for me. I think the 'boundary value problem' of how to make a consistent virtual world that nevertheless is acted upon by outside agents may turn out to be very problematic.

Look for example at things like Stephenson's 'Fall' - where a lot of how the world works is contingent on there being no way to influence the world. Important to think about.

patrapp3 karma

What's a good entry point for VR gear that will work with High Fidelity?

PhilipRosedale5 karma

The current version of High Fidelity is audio-only, so you only need a browser and (hopefully) some good wired headphones. We work on mobile and desktop, too.

Flowerwind222 karma

What’s your view on the future of VR? And which company/companies will be leading this industry? Thank you.

PhilipRosedale3 karma

VR devices need to get much more comfortable and enable typing at a normal speed for them to become mainstream. I think that will be about 5 years from now.

I don't think VR and AR will merge anytime soon.

I hope that many companies rather than a few compete to drive the future of VR.

westrajc2 karma

Greetings Philip! There is a ton of 'buzz' lately about NFTs and Virtual Worlds. As someone who has created virtual economies from scratch and understands the desire of creators to control and derive value from their work, do you see a future where there is a standard, secure, privacy-focused #Metaverse currency? If yes, how can we avoid the negative ecological effects of current Crypto offerings?

PhilipRosedale5 karma

Love the idea of NTFs as a durable way for art to move around and create more income for artists. Have seen similar things already in Second Life, which had transferrable/resellable secure digital assets from day one and a really big GDP in part because of that.

But... we can't use cryptocurrencies with Proof-of-work move them around because of the ecological impact. But of course there are alternatives to POW so that's great. Strictly speaking, a provable NFT doesn't strictly require a blockchain. Web of trust + signed proofs would suffice. But I love the overall progress so far and idea.

Voicebotai2 karma

Hey Philip. I had the chance to try out an HF demo last summer and think it is a very interesting concept for developers to build immersive, interactive spaces. I have two questions, you can decide if they are related.

- You oversaw the creation of one of the most immersive interactive spaces to date. Why do you think someone else will come up with something better than you as opposed to just building it yourself using your technology?

- We are seeing a rapid rise of virtual humans along with easier methods for people to create their own avatars and an increased interest in spending time in virtual spaces. When should we expect a new space where humans via avatars and virtual humans congregate and interact on a regular basis and when do you expect that space to reach some significant scale?

PhilipRosedale5 karma

I think these are different questions:

There are 100+ apps/sites right now that are trying to create social spaces of one kind or another with a ton of highly specific features for different verticals, like customized buildable objects or shared whiteboards, or avatar pickers. That's a ton of people working on new worlds - hard to compete with them all. But none of them have audio that is even close to doing what we do with spatial audio. So enabling all of them to move forward faster (with our API) seems like the right move overall if we want to see more spaces out there.

As to virtual humans (powered by AI) - I think that AI is emerging as the most important and potentially dangerous area of human progress. We've done work on virtual humans and there are a ton of big problems still - we are very early. Visual representations can still be very uncanny. Another huge problem is bias, racism, and polarization coming through in their behavior and communication. As much as with AI as with ourselves, this is something we must address before moving forward. So I don't think we want to congregate with our digital children, yet.

roccnet2 karma

What is High Fidelity? And what is SL doing to compete with VRChat?

PhilipRosedale4 karma

VRChat is amazing to watch develop! But we are still early - there isn't yet a mass market for VR headsets.

villagedesvaleurs2 karma

Hey Philip,

I was involved in academic research on SL as an undergrad anthropology student. Have you read any of the (surprisingly prolific) academic work which has been done on your virtual world like "Coming of Age in Second Life"?

PhilipRosedale7 karma

That's a great one - Tom Boellstorff is amazing - I try to collect/read all the books, yes. So many great ones. Many amazing collections of SL art too.

I remember having this "now we've really arrived" moment when I was flying into NYC and heard some super cool art gallery on prince street (I think) was having a gallery opening of these huge landscape images from inside Second Life. Not even avatars - just avant garde landscapes.

But yes, still lots more anthropology and sociology and economic to do in virtual worlds. For example - we should be using them to prove the value of UBI!

TRIZnik2 karma

Why do you think Facebook is keeping Horizon so closed in testing?

Could they be worried about anti-trust action if/when social VR starts dominating the VR software market?

(I've been bullish on social VR engines "swallowing" games since at least 2012)

PhilipRosedale5 karma

I don't know.

Social VR needs to be larger than one app or company or we are all at great risk.

DarthBuzzard2 karma

What is the physical limit of latency for avatar communication? Can we get to a point that is genuinely imperceptible from two people talking in real life, where no one overlaps the other talking due to latency, the way many people do in today's video calls?

PhilipRosedale5 karma

For people speaking to each other, the one-way latency needs to be below about 180 milliseconds to speak comfortably. There is a famous ITU paper from 2003 with a graph, if I remember... don't have the URL handy.

Old-school land-lines have a latency of 60 milliseconds (great!), cell phones typically are about 350 milliseconds (why we hate them).

A good Zoom / WebRTC call can be at 150msecs or even lower, which is why our company can exist! :)

MatterShim2 karma

Do you think a new virtual world will ever come along with a large focus on creating in the same way SL does? Why hasn't anything come along yet that overtakes SL in that way?

PhilipRosedale5 karma

Creating programmable objects that you can edit live in a big world where things can both move and teleport around at will is a really hard problem. We wrote a lot more code to do it than people think - and as SL people will tell you there are still lots of bugs and edge cases. I think this is why we haven't seen more things like it - we were really ahead of our time. I bet we'll see some amazing new projects soon (but I guess I've said that before and been wrong).

cilren2 karma

Hey

I recently discovered data sonifications of astronomical images, specifically by SYSTEM Sounds and since then hoped to hear these images in 3D audio. Wouldn't they be a perfect match?

PhilipRosedale4 karma

I'd LOVE to hear those sounds in High Fidelity. Put up a space? I'll come.

rangers13122 karma

What effect do you feel 5G and lower lag times will have in these areas? Would you see a Surrogates type of existence?

PhilipRosedale2 karma

5G can drop one-way latency by about 40 milliseconds, but 5G coverage isn't very broad yet. Once it gets there it will be a big help with smoother communication, yes.

blanketyblank12 karma

When will we have VR tech that I can wear without being teased by my wife?

PhilipRosedale6 karma

I guess maybe that depends on you. :)

But seriously - the fact that different people have such different responses to VR devices (both wearing them themselves as well as their perspective on other's using them) is evidence of the failure thus-far of VR to be a general-purpose computing device. No one felt this way about the iPhone.

VRMeg2 karma

You have been working on sound, curious what other senses might be added in future to make a fuller experience? Many times I wish I could smell a candle, or a campfire or rain on pavement. Do you think some day we will be able to smell things, because as someone create a candle they could add the smell of watermelon to it in creation?

PhilipRosedale3 karma

I love the idea, but I think the physics of simulating smell may never be possible.

But an even bigger problem is touch: We can't use our body's largest sense in virtual worlds. We can't 'feel' our bodies there. Our body is basically a part of our brain - pretty inseparable - when it comes to feeling things and moving around. I think there are some ideas yet to try (we built one called 'the rig' that was actually the first project at Linden Lab, before Second Life), but this is what we really need to make work - touch.

SleepIsForChumps1 karma

I haven't been back in hifi recently, have you gone to an open world yet? that's the one thing I LOVE LOVE about SecondLife that there is a whole world to explore, not just individual rooms. Part of why I love Bellisseria so much inside SL.

PhilipRosedale3 karma

Come and use our new API to build an open world using High Fidelity!

OryxTempel1 karma

Where’s my stipend?

Just kidding! What’s the plan for SL in the future? Are you going to go full VR or will it remain essentially as-is?

PhilipRosedale5 karma

I'm not running SL. :)

And, yes, Stipends in SL are a great example of the potential for UBI.

timf57581 karma

Hello Phillip,

My question is: 1. Apple is developing AR technologies. Do you think VR will eventually replace AR or do you think they can coexist in harmony.

  1. Do you think VR is essential to virtual worlds? And how far away are we in terms of technologies to develop a fully rendered real-life-like cities that players can join.

Thanks in advance

PhilipRosedale3 karma

I think AR and VR are very different things. AR requires approaches that blend information with the real world and balance attention. VR is the opposite - total immersion. Both the hardware, software, and economic/social/moderation implications are very different for the two.

I don't think VR (if by that you mean HMDs) are essential to virtual worlds. We interact with virtual worlds with whatever interfaces we care to. We will probably never be 'natives' to those worlds through our interfaces, however good, btw.

coryrenton1 karma

Have you seen any interesting peripherals to encode spatial audio for hearing impaired or deaf audiences (perhaps in a spatial vibration apparatus)?

PhilipRosedale3 karma

I haven't, but I would love to see more. Really great text captioning (which has gotten so much better with latest AI) seems to be one big tool for the hearing impaired - we are passionate about accessibility and are working on that.

croninsiglos1 karma

Does it work with the new airpods max?

PhilipRosedale2 karma

Answering for both Airpods Max and regular Airpods: Not quite, but it's fixable by Apple and other hardware vendors. Airpods and other bluetooth devices shift to mono/16Khz when the microphone is turned ON. So this means you can use spatial audio only if you are NOT using the microphone. This is fixable! We're bugging Apple and others about it.

Lewinga1 karma

I led my school's VR club in 360 Film and Animation. This was a particular topic of interest to me, so tons of questions incoming:

  1. What are some papers you'd recommend reading to understand how spatialization of audio works?
  • If 360 cameras record image in all directions, what is the equivalent in sound?
  • Assuming a six mono microphone rig in a cube-format, how do you reconstruct the sound from real life to virtual? Are you interpolating changes in headspace positions? What's going on there?
  • Is a stereo recording sufficient to extrapolate 360 sound? Or is it easiest to transform a single mono sound?
  • Using cameras as analogy: are more microphones better? Or is two enough to differentiate objects in space?

  1. How does your API know which waves to transform?
  • Your website cites (1) shifting time delay and (2) adjusting frequencies as part of the transformation process. Assuming a static position, (1) seems straightforward by adding a "shift" amount to two audio sources to simulate the delay in ear placements. (2) seems to imply that sound is transformed to simulate passing through the anatomical structure of the ear. Is this correct?
  • For (1), how do you add the ability to move your head in XYZ space? Suppose two instruments exist with different timbres. Relative to a listener facing the instruments, the listener's left cello plays A-440hz and a right trumpet plays A-440hz. How does the transformation function know how much of A-440hz to "tweak" for the cello vs the trumpet as the listener moves their head? How does this principle extend for the entire sonic-spectrum?
  • For (2), if my understanding is correct, what process do you use to "reconstruct" the absorbent/reflective profile of the ear? Did you have to measure the absorbent/reflective profile per wavelength to create a model?
  • For (2), I imagine the angle of attack to be important in hearing the source clearly (eg. same XYZ position, different pitch/yaw). How do you model hearing sound through the ear (as if it were behind you, parallel to source) vs hearing it directly from one ear only (and hearing less from the other ear, perpendicular to source) as the listener's position changes with respect to the source? I think I'm generally confused about how spatial audio is able to reconstruct 360 audio without an object-oriented representation of the world.

  1. What is the simplest rig/way to capture 360 sound?

  1. In your eyes, what are the biggest limitations to 360 sound recording right now, and what would it take for it to become commonly adopted?
  • What are the biggest technical challenges you've had to overcome, and what are you still facing?

  1. Have you gone to any Oculus Connect conventions (or any other VR conventions)? Can I connect my university's VR club to you?

PhilipRosedale3 karma

Some answers:

Binaural recording (with a rubber head with ears and two mics) is the other interesting way to capture a space from the point of one observer. As you say, some sort of spherical array of mics is the other.

Yes, an HRTF applies a frequency convolution to a source as a function of direction to the source. It simulates (roughly) the passage of sound through the Pinna and head.

This transformation must be applied independently to each source. This is computationally expensive, and is where a lot of our R&D went - doing spatialization for many (hundreds) of sources for hundreds of receivers on a server.

It turns out that most people are pretty insensitive to the pitch angle. You have to be trained to have a good sense of it.

I've gone to most of the Oculus Connect events. Sure - connect me to your club.

financeandfucks1 karma

What are three core features / capabilities that you would like to see built out in Second Lifw over the next few years, and why?

PhilipRosedale4 karma

I'm not running Second Life, so I don't have specific answers there. Some good recent changes have made it faster at the simulation layer (things like faster region crossing) - I know that is something people value and that I'd like to see get faster and faster.

Brianiac801 karma

What is the theoretical and realistic minimum audio latency for High Fidelity? Could it be used to organize a virtual choir?

PhilipRosedale2 karma

Within a large city you can use special tools like JackTrip to get latency lower than 20 msecs to enable musicians to perform together live. But browser-based WebRTC isn't fast enough yet to make this work for everyone. Typical 'real-world' latency is more like 100-200 msecs.

thereidenator1 karma

I see you answered a question about cryptocurrencies, have you had chance to check out Upland? I think it's got potential to be a big game like second life, what do you think about it?

PhilipRosedale1 karma

I have heard but have not checked it out yet. Will do, thanks for mentioning.

More broadly: Is the future of the real world to be sold parcel-by-parcel to the highest bidder? We should be very careful what we wish for. I, for one, don't think that would move us forward.

day7seven0 karma

When is Third Life going to be released?

PhilipRosedale3 karma

Not in my life. :)