MuSA_RT 3.1 is out! This latest version features RealityKit-based graphics, support for audio file input, and the ability to stream result analysis to other MuSA_RT clients for local rendering.

 

RealityKit graphics

MuSA_RT 3.1 features slicker and more efficient visuals, courtesy of entirely new RealityKit-based models and rendering.

The resulting app is now compatible with VisionOS, although untested beyond the simulator. It is likely that the app running on the Vision Pro device does not have access to suitable live microphone input, which is partial motivation for the other two features.

File input

MuSA_RT 3.1 can take input from any local audio file.

File input is useful when live input of suitable quality (or live musicians) are not readily available. Unfortunately, as a third party app, MuSA_RT cannot integrate with music streaming APIs, as it requires sample-level access to the music data, which DRM protection precludes.

For scientific study, file input allows to run experiments consistently and repeatably. In this context, it is also desirable to have access to the raw analysis data, which is also partial motivation for the streaming feature.

Streaming

MuSA_RT 3.1 can stream analysis results to other MuSA_RT instances which only perform rendering of the streamed analysis results (no local analysys). Server and clients must be connected to the same local network.

A typical use-case involves a dedicated machine or device that performs the audio analysis, maybe from quality sound input and/or a library of audio files.

In a scientific setting, a very simple Websocket client can collect analysis results to save them in any format suitable for further analysis and comparison. This is especially convenient when used in conjunction with file input.

In a shared experience setting, the machine running the analysis also acts as a server to MuSA_RT instances running on other device, e.g. iPhone or iPad (maybe even Vision Pro?), set to remote input. Connecting to the server is as easy as scanning a QR code. The client devices render the same analysis results computed on the server.

A Glimpse of MuSA_RT in Mixed Reality

Finally, here is a preview of what a Mixed Reality version of MuSA_RT could look like. This was captured in a Quest 3 headset. The MR client, developed in Unity, renders the model from audio analysis data streamed from another device (Mac or iPhone/iPad).