Virtuoso: musical instruments in VR

[NothanUmber]

A recommendation to people who have access to a VR headset (Quest1/2, anything SteamVR compatible): Virtuoso is a collection of instruments that can be played in the air with VR controllers. It has a looper, an arpegggiator, various sounds to choose from and quantization (who wants that ). One can also have one hand at one of the instruments and one hand in one of the effect boxes to add feedback, echo etc.

Instruments include:

• Board: A 3d grid with 14x4x4 cells. The scale of the cells can be chosen. There are three buttons per hand that can be used to sustain up to six notes of choice - the space where one presses a button defines the pitch of a new note and that note plays (and keeps it’s initial pitch) until one releases the button. Twisting the hands controls timbre. (In some demos they control timbre with the left twist and pitch bend with the right twist axis - couldn’t get this to work yet though)

• Oorgan: A 3d grid like blocks, with the difference that the played note changes when entering a new cell

• empads: a 4x2 grid with round cells that acts as a drum kit. Pads can also be dragged out. Instread of triggering samples one can also trigger already recorded looper-sequences.

• wHarp: 30-or-so vertical bars. Works like chimes - no button has to be pressed, just touching a bar makes it play

• Wavemin. Kind of a Theremin where pitch is vertical and timbre change front/back. Twisting the hand is volume control. Instead of playing it polyphonically one hand can also go into one of the effect boxes e.g. for feedback control or echo.

• a microphone. As all VR headsets have a physical mic this was probably a no-brainer

Apparently it supports MIDI, so one can also use it as controller for own synths!

Here some demo/tutorial videos of most instruments and a few features: https://www.youtube.com/channel/UCorPaMn_yiWgwoQBkQFS66A/videos

Here their homepage: https://virtuoso-vr.com/

[NothanUmber]

Chatted with the devs they consider to add pitch bending between cells with auto-rounding-as-soon-as-movement-stops. Sounds promising! Also dropped the term MPE (might be nice to have pitch bending and modulation for both hands independently also when using external MIDI).
External MIDI relies on a companion application that is currently only available for Windows. They mentioned that they plan to port this to Mac though.
As Steam isn’t supported on Mac, the Quest 2 (=Facebook standalone headset) is probably the only option for Apple fans atm. Or to wait for the Apple MR headset that is rumored to come out later this/early next year…
Let’s see…

[Kai]

Very cool. I will definitively try this.

I tried SynthVR on Quest some time ago. It was very basic (at least back when I tried it), but I remember thinking that patching a modular synth in VR made more sense than I had initially expected.

I’m generally very enthusiastic when it comes to VR. In a similar way to how the early days of TV apparently often were some men behind a desk talking as if it was a radio broadcast, VR is now still mostly “stuck” in the video game mindset. I get why, but I look forward to seeing VR figuring out what its unique strengths are and become its own thing. Hopefully, music creation will be a part of that.

[NothanUmber]

Think with VR it will stay more niche. But when AR glasses start to replace (or at least complement) smartphones, having virtual musical instruments that one doesn’t have to carry around is rather obvious. Who wouldn’t like an air guitar that actually works But more interesting than real-instrument conversions will imho be new kinds of instruments that make use of the new spatial possibilities.
Then people can also meet face to face again and play together without latency. Or (assuming 5/6/7g) project the band members into their living rooms.
This certainly won’t replace physical, haptic instruments. But I could imagine it could complement them and create a new category of accepted instruments.

[Kai]

I want to use VR for work. “Office work” (software development in my case) in VR isn’t quite there yet. Although some people work like that already, I don’t like it. Yet. I see a lot of potential, though, and if the experience would be acceptable for most people", perhaps that would give VR a big boost. Being able to remote work from anywhere, having unlimited screen space, have better digital meetings and even the ability to concentrate better in a busy office environment. In my opinion, higher resolution, better comfort and better AR functionality is what is missing, but it might not take long for any of this to be good enough.

Now I started thinking of how I would want a VR DAW to be. Perhaps I should try doing some Cubase work in VR just to see how it feels.

[NothanUmber]

Agreed. The HP Reverb G2 is borderline sharp enough that text work would be possible. But why should I? All work related programs I need are optimized for monitor - and I have a much sharper monitor where I don’t have to wear a bulky thing that blocks my view out of the window.
Think for office work getting viable we also have to wait for AR. But not only hardware but also new user interface paradigms that make more use of the new possibilities.

The “killer app” when it comes to office would probably be the ability to project other people into your home office, so one could remotely talk “face to face” even more natural than via webcams. For that one would need cameras that film and track the full body as well as the environment. Which opens a lot of possibilities. But also to-be-answered questions regarding data privacy…

[greaterthanzero]

I think AR for musicians has a lot of potential, augmenting physical controls instead of replacing them.

The one-handed FX controls listed above would work great in this scenario, leaving your other hand free to interact with familiar music hardware.

But also, we could…

• overlay dynamic visual feedback onto your MIDI control surfaces, be they a keyboard, soundplane, sensel morph, or whatever. (The display would be attached to whatever functionality an app gives that controller, independent of manufacturer participation)

• integrate and expand on traditionally screen-based utilities (like Camelot Pro and Divisimate, which quickly select patches and reconfigure your setup)

• display lyrics, chord charts, sheet music, etc in a virtual teleprompter, which is synchronized between band members as the set list evolves.

• add a readable text display to whichever Eurorack module your hand is closest to, to clarify obtuse designs

• add a readable text display to whichever pad your index finger is closest to, on versatile grid controls like Deluge or Zoia, or your favorite monome apps.

…and so forth.

[greaterthanzero]

@NothanUmber

The “killer app” when it comes to office would probably be the ability to project other people into your home office, so one could remotely talk “face to face” even more natural than via webcams.

For that, one would need cameras that film and track the full body as well as the environment. Which opens a lot of possibilities. But also to-be-answered questions regarding data privacy…

There’s several layers of uncanny valley to overcome, as well.

I turn my camera off when I feel that having it on will create distractions. The immediacy of body language is rarely desired. But imagine how disruptive that would be if it feels like I’m in the room with you, popping in and out of existence.

Likewise, I want everyone’s mics to be muted until they’re actively participating in the conversation. Which is creepy AF, if they’re passively staring at each other.

And physical space becomes a concern, if meeting participants are meant to seem present. Your cubicle is a terrible conference room.

All of this can be solved in time. But I’d personally rather have webcam chat, with 2D windows floating around me.

[NothanUmber]

Fully agreed, the way how this would have to be set up would probably need many iterations to feel natural and more helpful than irritating.
It could be good if one could have the vibes of being part of a group of people who sit in a workshop room iterating on some task together. Perhaps even interacting with some kind of digital twin of an otherwise physical machine or something.
Like with a phone where one has to accept a call it would be mandatory that you invite people to your room and give them watch, listen and talk rights - they shouldn’t just pop up as they please behind your back

[Kai]

I haven’t tried a headset with eye and face tracking, but I hope that will make online communication feel much more natural. The issue of latency still remains, obviously, but still, having a meeting in MS Teams is anything but great.

I would already prefer meetings in VR on current tech to Teams. Avatars, voice, head tracking and hand gestures in a shared VR space is in my opinion already way better than Teams. Facial responses and the ability to look directly at the person you talk to might be what is missing to make this a significant enough upgrade from Teams that companies want to start using it. But I’m not sold on AR for this, though. I would prefer a full VR experience for meetings.

[teknico]

@Kai

I want to use VR for work.

@NothanUmber

Agreed.

You might want to have a look at SimulaVR (website is working in Chromium but not in Firefox right now, for some reason).

[thetechnobear]

recently Ive been doing a lot flying with an FPV drone, great fun… and I guess is some kind or AR.

this has meant, Im quite interested in trying some more modern VR/AR gear.

however, my general feeling is, the tech is not quite there yet…
theres a few reasons;
a) quality… are the displays as comfortable to use for long periods, are they sharp enough not to create eye strain when reading text (etc)
b) price … too expensive, and simply it limits market potential - its no surprise that gaming is a leader in this field , as some gamers spend a lot on gaming rigs already.
c) functionality … I just think without good haptics its a pretty limited experience/usefulness.

VR has been around a LONG time, and frankly the above ‘issues’ have been holding it back ever since its inception… of course, each decade we get that bit closer to solving… and the fact its now having success in gaming is proof its making progress…

but for usage outside gaming, Im not so convinced (yet)

virtual space is great , you don’t need 4 high res monitors, but only if the headset is comfortable to wear

physical relation… this is where AR wins, without tactile feedback, and probably objects to manipulate its easy to lose orientation… and this means you are losing a kind of ‘human sense’ , so whilst augmenting some sense we are losing others.
and for music, that tactile nature is something thats always been important, and if you see the resurgence in hardware synths… we are seeing again… VSTs dont have that feel/involvement.

exclusion… phones are distracting enough these days, and remote working is creating exclusion - without office workers not only be in a separate cube, but virtual world… thats almost nightmare-ish. humans need real social interactions.

overall, unfortunately, it still a bit feels like the tech industry looking for a problem to market their solution too, and I get that… a lot of tech evolves this way… but it takes a bit of time…

… and for now the likes of virtuoso-vr, unfortunately, seem to be doing this… I dont see it solves a real problem we have, sure… a great ‘tech demo’, but not something you cannot already do outside VR.

online music collaboration I think is more hampered by mundane tech issues like latency … and things like tool support- than sharing a virtual space. and even where its been ‘solved’, the take up is quite minimal really.

do I think AR/VR will influence this all long term…
hell, yeah… once price/functionality/quality hit a certain point, where a product can be released that has a decent market. imho, this is why AR has overtaken VR (for now) … more accessible, more relevant.

as for now…
I’ll keep flying my drone, and I might yet buy a VR set… or just wait till it improves again

[NothanUmber]

yepp, still a long way to go until it feels as if it weren’t there. Things are improving though

a) comfort is getting better, even if not optimal yet, each generation is a little lighter than the one before. And the pixel per degree value of the better headsets is >30 now (with the exception of high res micro center screens for the very expensive headsets which are > 60 ppd). 30 ppd is still below visus 0.7 (=60*0.7 = 42 ppd) that one would need to e.g. be allowed to drive a car in the real world - but day and night compared to first gen VR.

b) on the one side prices are decreasing - mainly because Facebook is subsidizing the Quest massively because they need the high numbers for their more data centered midterm plans. On the other side it’s going through the roof because like GPU manufacturers some headset makers have found out that there is a certain group of people who is willing to pay almost any price for their hobbies.

c) generally right, vision and hearing is missing out senses. One thing that surprised me though is how well feedback via rumble motors works. E.g. when playing VR table tennis the brain is mocked that the bat just hit the ball quite convincingly. A keypress on a virtual keyboard wouldn’t of course work as well, as the rumble motor cannot prevent the hand from going through the keybed E.g. sticks on drums where one naturally retracts the hand for the next beat feels not totally unconvincing again.

Long story short - it’s getting more convincing with every iteration, but waiting is still a valid strategy

[NothanUmber]

@thetechnobear

exclusion… phones are distracting enough these days, and remote working is creating exclusion - without office workers not only be in a separate cube, but virtual world… thats almost nightmare-ish. humans need real social interactions.

That is another interesting aspect. Gabe Newell (the Valve boss) hypothized that in some not too distant future (he thinks within his lifetime) the (not only) visual fidelity of virtual worlds will be better than reality. By bypassing eyes and other senses, using direct brain-computer interfaces, claiming to make virtual worlds more realistic than reality.
His assumption is that people will be disappointed by the limitations of the real world soon.

Which has potential to make both the best utopies and the worst dystrophies become real at the same time. But hard to see the future is

[thetechnobear]

yeah, as games have kind of proven, technically VR can work and be immersive.
they prove it can be fun

but the issue is fun is not the same as productive…
this I think is where AR has proven itself, because its premise was … take the great things we have in reality and enhance them… you don’t really have to ‘trade’ benefits , pros/cons.

and thats the issue with VR, you always are trading things off…

so in @Kai , development environment…
sure Id love a virtual multiscreen setup. and to some extent I’d love to do away with the mouse…(its far from ideal)

BUT for text editing
a) display
I need it to be PIN sharp…because modern monitors are very sharp, and easy on your eyes.
b) keyboard
again, keyboards are not perfect, however they are REALLY fast … you cannot do without one (in the work I do at least) , and I need that to be physical, so has has a feel/resistance… developers have preferences for a particular keyboard for good reason.
sure, I could ‘overlay’ this in VR… but now we are moving more to AR.

c) wave hand in the air is tiring
try keeping your arms in the air horizontal in one position, without resting on something (like a desk) , its tiring … and some things like typing would require this.
funny, this is why Steve Jobs was a bit against touchscreen monitors , but so much for tablets, because he said holding your hands up to a monitor is not really comfortable. (a tablet you bring down to lap etc) - and I agree ,
though I still like touchscreen monitors for some purpose, and you can place them horizontal… though that had ergonomic issues too, and is not really relevant to VR

and I think music making suffers very similar issues…

people LIKE playing physical instruments, the tactile response and how it affects sound…
even in our world of expressive controllers… playing a virtual sound through say a continuum surface vs a linnstrument is very different.
and this is also an emotional response… and that (I believe) is very important in music making, and whilst in one sense instruments are tools, I think most believe they are more than that (partly due to the time investment?)

this is where VR really ‘fails’ in demos Ive seen…
sure we have nicely crafted ‘3d joysticks’, but have you seen any musician(*) reach for a joystick as their preferred way to play a VST/DAW?.. this would be possible, even outside or VR.
instead, even in DAWs, ‘most’ musicians use some kind of keyboard (or grid) controller. (**)

again, not hating on VR… I think it will evolve, and Ive no doubt the tech is getting leaps n’ bounds better…

the issue is perhaps that gap - I think for mainstream, we are need something like :

• something as small/light as glasses… or at least, a bit smaller than size of Orqa fpv goggles
• glasses as good as modern monitor, no eyestrain, no/low latency
• some kind of haptic solution - perhaps gloves
• budget… glasses, around $300-400? price of a decent monitor, haptics $100?
• compatibility… needs to be compatible across the board, no select brands (unless its apple )

we are close, and (excluding current chip/component shortage) tech prices will tumble…
and I do think, they’ll be a tipping point… where its a fortuitous circle… mass market buys, developers support → more buy.

and, importantly, I think we need to understand better the ‘goal’ of VR, in terms of its relationship to the physical world and objects… perhaps, VR never really exists in terms of its totally ‘virtual’, but perhaps its more than we think of as AR currently.
thats an area I think will be really exciting to explore.

anyway, better start brushing up on my openGL skills, if I want to participate in this world

---

(*) ok, there are some, but thats not the point
(**) ok, not 100% true, a lot of ‘producers’ create music in daw with a mouse/computer keyboard … but for those, Id argue my developers use-case against VR is perhaps rather valid

[Kai]

Hmm, yes, my current opinions on dev work in VR:
a) display - agree, needs to be very sharp. Quest 2 is not good enough and I have no good sense of how much better it needs to get. If the rumoured “Quest Pro”, the Apple headset or even PSVR2 will be good enough. In any case, this is why I’m not currently working in VR at all.

b) keyboard - this is technically solved already, but is not great. Either by mapping keyboard to VR or use the cameras on the headset to see your desk/keyboard. Both solutions needs to be improved to be really convenient, but the current state is acceptable for me. I can not imagine ditching the keyboard anytime soon, so some (reasonably simple) improvements on what already works would be good enough imo.

c) Agree, I really don’t want to be waving my arms around in the air. Unsure about what the need for a mouse will be on headsets with eye tracking, though. Perhaps a keyboard and eye movement will be enough soon-ish? Idk, but for now, I want a traditional mouse and keyboard, even if in VR.

Generally, VR vs AR is not that big a deal to me. Quest have some AR functionality (because of the inside-out tracking) and this seems very likely to be improved upon in new headsets. For several reasons. Safety when gaming (e.g. my cat steps into the play area to let me know how disappointed he is with today’s menu), office work (where did that coffee cup go!), etc. I think VR and AR will have a lot of overlapping use. And that VR headsets with decent AR capabilities will be leading the way. I hope we are as good as there now, hardware wise. I don’t think it is unlikely for me to be doing some of my work in VR in a year or two. Good AR glasses seems further away tech-wise to me. (But I don’t actually follow AR tech as much as I do with VR, so what do I know…)

@thetechnobear

and, importantly, I think we need to understand better the ‘goal’ of VR, in terms of its relationship to the physical world and objects… perhaps, VR never really exists in terms of its totally ‘virtual’, but perhaps its more than we think of as AR currently.

Yes, I feel the current state of VR is hugely impressive, but at the same time almost a novelty or toy. As if there is a lot of untapped potential, but no-one knows exactly what do with it, in a way. What keeps blowing me away in VR is how strong the sense of being present in the VR world is. That is probably the unique, core strength for VR as a new medium.

No idea what direction it will take, but I like paying close attention to what devs are doing. Perhaps more “experiences” of various kinds, rather than games or movies in the traditional sense. Also for social, human interactions in our modern, digital world. I don’t see VR as a threat to real-world interactions, but a big step in making our digital lives more “human-friendly” again. I notice I have a much stronger sense of internetters being actually present in VR. And it is funny to notice how much more polite people tend to be in online games in VR.

[thetechnobear]

@Kai

Either by mapping keyboard to VR or use the cameras on the headset to see your desk/keyboard.

agreed it works, but by definition a physical object now restricts your move movement on some axis.

this is not a huge deal, but implies restrictions
imagine being surrounded by ‘screens’ , you cannot now just turn your head/body to use them, since this would not be ergonomic to twist yourself and not your keyboard.
so we start to get into ‘swiping thru views’ , but how different is that to using a decent large monitor with multi-desktop support?
perhaps this could be solved by swivel chairs with a rotating keyboard stand attached, but that might required a different physical space.

as I say… not the end of the world, but really it still begs the question of …
how much benefit does this provide, over what I currently have … and much cheaper?

K, so to be positive, I do see ONE potential ergonomic benefit (thats also a budget consideration) in this area for VR.
desks/monitors are prone to ergonomic issues too… and we are trying to work around these with things like standing desks, better seating etc… and these solutions are not cheap.
so there is definitely an area of interest here, if we could become a bit more ‘physical mobile’ for desk work, this could really provide health benefits.
BUT we have to be careful to not to swap one ergonomic issue for another

@Kai

Generally, VR vs AR is not that big a deal to me.

I agree… I actually thought AR, would be the ‘new VR’ … and I think to some point there is no real distinction… its more a continuum Physical Reality → AR → VR.

@Kai

Also for social, human interactions in our modern, digital world. I don’t see VR as a threat to real-world interactions, but a big step in making our digital lives more “human-friendly” again. I notice I have a much stronger sense of internetters being actually present in VR. And it is funny to notice how much more polite people tend to be in online games in VR.

thats an interesting observation…
I do think that even voice (in games) helps this as well, its about breaking down ‘anonymity’ and the sense that there are no consequences to actions.

however, in fairness I think VR wont solve that… we can see in all social media, this is a much broader problem (bullying, hate speech etc is rife on SOME platforms) … unfortunately, avatars, no physical presence still gives that ‘freedom’. that was seen really badly in places like Second Life (and even Minecraft/WOW ) … this is more about platforms have consequences for “bad” behaviour.

I’ve hope in this area… whilst its still a problem, it is now much better understood, and is much less tolerated… zero tolerance, will (hopefully) lead to consequences, and in turn that will change social attitudes. but it’ll take a while, as the issue here is not tech, but social attitudes.

[teknico]

@thetechnobear

anyway, better start brushing up on my openGL skills, if I want to participate in this world

OpenGL is old tech, WebGPU is where it’s at nowadays. See also (skip to “Truly cross-platform graphics API”; but afterwards maybe go back and read about Zig anyway ).

[thetechnobear]

I cannot see also how you can say WebGPU is where its at… given their own website says it’s in trial, and not launched yet - that kind of implies, there are no commercial products built on it yet. is it proven at scale yet? if not Id rather put it in the experimental/interesting category, rather than proven tech. (*)

anyway, doesn’t really matter… I find with most tech, once I get a bit of experience with one (in the field), I usually find it very easy to switch to another.

and my interest in OpenGL is in products that exist today in commercial products that people have in their hands, things like Percussa SSP and ETC

but hey, its cool we have some alternatives brewing, and of course, as we see wide spread adoption - I’ll pick them up and use them

---

(*) Ive also little interest in many of these web based techs… pinning our hopes on chrome/chromium is not a future Id savour much

[teknico]

@thetechnobear

I cannot see also how you can say WebGPU is where its at… given their own website says it’s in trial, and not launched yet - that kind of implies, there are no commercial products built on it yet. is it proven at scale yet? if not Id rather put it in the experimental/interesting category, rather than proven tech. (*)

Its value proposition is better explained in the article I linked under “See also” (admittedly not the clearest description).

It allows you to use a single API instead of having to learn Vulkan (the actual OpenGL successor), DirectX (Microsoft’s thing) and Metal (Apple’s thing).

Here’s its implementation status in the three main web browsers.

@thetechnobear

(*) Ive also little interest in many of these web based techs… pinning our hopes on chrome/chromium is not a future Id savour much

WebGPU is not limited to the web stack. The linked article describes WebGPU usage in a forthcoming Zig game engine that builds on Dawn, Google’s WebGPU implementation.

@thetechnobear

but hey, its cool we have some alternatives brewing, and of course, as we see wide spread adoption - I’ll pick them up and use them

Fair enough.