PianoVision is one of the best mixed-reality experiences I’ve tried so far on Meta Quest 3. It helps you to learn the piano by placing a digital overlay on your real piano, showing you what notes to play, providing feedback on your performance, and has lots of features to assist your learning. Whilst it works best with a digital MIDI piano, you can also use it with an acoustic piano, or even use a virtual piano.
I’ve been learning to play the piano very slowly over the last two years, mainly due to my daughter starting piano lessons, which has ignited my interest. Until now, I’ve used traditional books and sheet music, as well as two popular piano learning apps, Simply Piano, and FlowKey.
I’ve tried using PianoVision with an acoustic piano and as a virtual piano on my desk, but you will have the best experience if you have a digital piano that connects to your headset via a MIDI cable.
PianoVision is compatible with Meta Quest 3, Quest 2, and Quest Pro. The passthrough cameras on Quest 2 are low quality and grayscale, but it is perfectly usable. You will have the best experience using PianoVision with Quest 3 due to the high-resolution dual RGB passthrough cameras and depth sensor, which is better able to track your hand movements and detect the notes you play.
PianoVision Video Review
In addition to this article, I’ve also made a video of my experience using PianoVision On Meta Quest 3. I cover all the features, pros, and cons, and share why I think this is one of the best MR apps I’ve tried so far.
Is PianoVision Any Good?
PianoVision feels like a killer app for mixed reality, and it has genuinely been one of the biggest WOW moments I have had since I got my Meta Quest 3. It makes it easier to learn and play the piano than any other method I have tried in the past. It’s also loads of fun and works wonderfully well with my electric piano with a MIDI connection.
I’m really just a beginner on the piano, and I struggle to keep up with my daughter, who is learning Grade 2 at present, but playing songs using PianoVision is so much fun, that I know I’m going to improve a lot over the next few months.
PianoVision has quite a few rough edges, but in its current state, it’s an app that I would highly recommend to anyone who has a MIDI keyboard or piano at home. It’s probably not worth getting if you just plan to use the virtual keyboard, or even if you only have an acoustic piano.
You can play songs using PianoVision even if you have limited musical experience, as the app uses visual note indicators to show you what notes to play and when. There are more advanced settings to help you improve your technique and learn the pieces you are playing. You can also display the sheet music for each piece if you like.
The hardware and software to create this type of experience have only been possible for a short period, and as a result, there are quite a few rough edges that you will encounter when using PianoVision.
You must carefully adjust the virtual piano each time you use the app, ensuring that it lines up with the physical keys on your piano. The virtual keyboard can drift a little during use, requiring you to readjust from time to time. In addition, the MIDI connection to your headset uses the USB-C connector, so you can’t use an external battery while using PianoVision.
There are a huge number of songs included, but the song library lacks some popular favorites for people learning the piano. However, there are more than enough songs to keep you busy for a long time.
Despite the negatives, PianoVision is a terrific mixed reality app to help people learn to play the piano. It is fairly inexpensive to buy and if you have a MIDI piano or keyboard, and want to experience a fantastic implementation of mixed reality, you should 100% give it a shot.
Setting Up PianoVision
Let’s look at the process for getting started with PianoVision. You should sit at your piano first and then launch the app. It will immediately tell you to put your controllers down and you will use hand tracking to control everything within the app.
If you are using a digital piano, this is the time to connect it to your headset via a MIDI cable. Your digital piano or keyboard should have at least one of the two common MIDI ports on it. Either a round, MIDI port, or the more square USB-B port. As a result, you will either need a USB-B to USB-C cable, or a MIDI to USB-C cable to connect your piano and VR headset. If your cable only has the larger USB-A connector in place of the USB-C connector, you can get a USB-A to USB-C adapter.
When you connect the cable to your headset, you should see a prompt asking you if you permit the headset to use the MIDI connection. Use your hands as controllers and pinch to confirm that you are happy to do this.
Next, you need to calibrate your piano, and this is done by selecting the number of keys on your piano and then marking the left and rightmost edges. Select the leftmost key and then the rightmost key, and you should now see pink markers at the edges of all of the white and black keys.
If your experience is anything like mine, you will now need to use the control panel on the left side to fine-tune the position of the keys. These markers must align as closely as possible with the real keys as this will determine where the virtual note indicators appear.
Using PianoVision
The central menu screen is where you will choose what song to play. These are split up into different difficulties, from Level 0 to Level 12. You can preview each song before playing it by toggling the autoplay button at the top right corner. You can also press the heart icon on the right-hand side to mark your favorite songs, and these can be accessed from the favorites tab, for easy access to your favorite songs.
Once you’ve chosen a song you like, click play to get started. You will see colored blocks coming towards you, aligned with the keys that you need to press. The notes are color-coded, with left-hand notes being pink and right-hand notes being blue. White key notes are lighter in color and black key notes are darker, so you always get a great visual indicator of what notes to play.
The letter of each note appears on each note as it moves towards you, and for the simpler tunes, you will see an indicator of what finger to use for each note.
As the notes move forward, you position your hands and play the notes when they meet the real keys. The app will wait for you to play the correct notes, and the blocks will turn green as you play them. As a result, if you are playing a song for the first time, and want to start to learn it, you can take your time ensuring you get each note correct, rather than having to keep up with the song.
Under the keyboard, you will see a control panel that can be used to adjust a lot of settings and customize your experience.
You can choose to play the left-hand or right-hand parts individually by toggling the hand selection button. You can then concentrate on learning one part and the notes from the other hand will play automatically.
You can also adjust the tempo to speed up or slow down each song. This is helpful if you are playing a more difficult tune and are struggling to master it. Slowing the song down until you can play it perfectly, and then slowly increasing the tempo with each attempt until you can play it at full speed is a great way to improve and master specific songs.
You can also toggle a metronome on and off if you would like some help with keeping the tempo of the piece you are playing.
Monitoring Your Performance
At the end of each song, you are presented with a performance summary that gives you a percentage based on how well you played the song, indicating how many incorrect notes you played, and giving you feedback for each hand. You can click the “See More” button to get a breakdown of your performance for each section of the song. This helps you see which area of the song you need to focus on to master it.
Tools To Improve Your Playing With PianoVision
While you are playing a song, you can select the performance tab on the right to see your performance in each section of the song. You can select any part of the song to practice it individually, and there is an option to enable infinite looping to play a section, or the entire song on repeat until you have mastered it.
This is my favorite way to improve my playing. I usually play each song through from start to finish a few times to get a general feel for it and then pick a section to play on a loop until I have mastered it, before moving on to try a different, or a longer section to practice.
Also located in the performance tab are two other features, the performance engine and the memory engine.
- The performance engine automatically chooses what section of the current song you need to practice and will adjust the tempo as needed to help you improve.
- The memory engine aims to wean you off your reliance on the colored note indicators. As you play each song in memory engine mode, the indicator notes gradually fade, helping you learn the song so you can play it from memory.
Using Sheet Music With PianoVision
In addition to getting visual indicators of what notes to play, you can display the sheet music for every song in the app. You can toggle this on by pressing the book icon in the lower central settings panel.
The sheet music will appear in front of you and you can simply use this to play each song. A visual indicator will appear on the sheet music to show you where you are, but you can read the music as you would normally to play the correct notes.
If you get stuck, you can still glance down at the colored indicators to show you what notes to play. I think this combination is quite powerful for beginners as it helps you learn to read sheet music in a fairly frictionless way. The downside is that it’s easy to get lazy and just go back to looking at the indicator blocks.
Uploading Additional Songs To PianoVision
You aren’t restricted to playing the songs that are included in PianoVision as you can upload MIDI files to the app, allowing you to play any song for which you have a suitable MIDI file. You’ll need to download the PianoVision desktop app to your computer and connect your headset and keyboard to your computer by USB cable. You can then upload additional MIDI files to the app.
I haven’t been able to try this functionality as my piano is downstairs and my desktop computer is upstairs, so I can’t easily connect the two. I would love to see the developer add a much larger range of songs that are popular with people learning to play. Unfortunately, there were many songs that I wanted to play that were not available in the app at present.
Can PianoVision Help You To Learn To Play The Piano?
Although PianoVision is a great app and has helped me to play much more challenging pieces of music than I normally would on piano, it sometimes feels like you’re just reacting to the colored blocks approaching the keys, rather than actually learning to play.
I have found this to be the case with some other piano-learning apps, so it isn’t unique to PianoVision. I’m glad that PianoVision can display sheet music and has features such as the memory engine to help you learn pieces, rather than relying overly on the colored indicators.
I think that PianoVision is a tremendous tool to help people learn to play the piano, and is one of the most effective tools I have ever used. I remember trying piano-learning apps for the first time, and immediately being struck by how much easier they made it to play and maintain my motivation to learn. Essentially, PianoVision works like a piano-learning app that is overlaid on your real piano, merging the two and making the experience more seamless.
I have been able to play songs that are much more difficult than what I would attempt to learn from sheet music, and I’m able to play through them slowly at first, and slowly improving with each attempt.
What Needs To Be Improved
Although PianoVision is a great app, there is a long list of things that need to be improved. Some of these issues can be fixed by the developer, but some highlight the limitations of current mixed reality technology.
Firstly, PianoVision works well with a keyboard or Piano that is capable of making a MIDI connection with your headset. The limitations of hand tracking accuracy mean that using the virtual piano is largely a waste of time. At best, it will be an amusing novelty, but at worst, it is highly frustrating to use, with poor note recognition, largely as a result of the limited hand-tracking accuracy.
Secondly, you will need to readjust the virtual piano overlay every time you play, as it always drifts out of position in between uses. This adds some friction to using the app, as you have to spend a minute or two ensuring that the keys on the virtual keyboard align perfectly with the real keys.
The virtual keyboard can sometimes move a little when you move your head while you are playing, and can drift out of position if you are using the app for a while, necessitating you to readjust the position.
Once again, this is largely a limitation of the technology in my Quest 3 headset and is unlikely to be fixed any time soon. I can imagine a situation in the not-too-distant future where the app would automatically detect your keyboard and align the virtual keyboard on top of your real one without it having to be manually adjusted. I think this will only be achieved with upgraded hardware, but it will be awesome when it happens.
Another downside is that because you will use the USB-C port on your headset to connect it to your keyboard, you can’t use an external battery when using PianoVision. This limits you to using PianoVision for about two hours at a time before your headset will need to be charged. Once again, I don’t think there is anything that can be done about this, but as someone who has multiple external batteries, and battery straps for my headsets, it is a shame that I have to take a break so soon.