HoloLens HMD

Homebuilt Aircraft & Kit Plane Forum

Help Support Homebuilt Aircraft & Kit Plane Forum:

athilenius

Member
Joined
Sep 25, 2022
Messages
5
Location
San Diego, CA, USA
Has anyone played with a Microsoft HoloLens as a HMD? I found this Using augmented reality to reduce workload in offshore environments - CEAS Aeronautical Journal but I'm really surprised to find no mention of the HoloLens in the EAB community.

The idea is pretty simple: take a HoloLens (augmented reality headset that paints images onto the back of your eyeball using lasers, no joke), attach high-precision head tracking within the cockpit (there are several affordable ways to do this thanks to VR tech advances), then piggyback avionics data with sufficient IMU resolution and you have yourself a functional HMD. The information is invaluable for situational awareness... artificial horizon, traffic diamonds and ranging (ADS-B), altitude, airspeed, VSI, AoA/alpha, prograde vector, ISL glideslope/alignment and so on.

The software is not particularly difficult, the spatial transforms are easy-peasy (just a few matrix transforms), and the HUD symbology is easy to paint once you have the data. I have little experience with modern avionics hardware, but it looks like they are generally CAN these days, so easy enough to piggyback. You could also have an entirely separate system if need be. The only really "critical" component would be a robust health-check loop that cuts power to the entire headset the instant things go sideways. It wouldn't be a PFD, so hard failure-modes are a must, as opposed to for example painting an incorrect artificial horizon (which is exactly what happens when an antiquated vacuum system craps out). I have the knowledge to take a crack at the project, but (for now) lack my own E/AB to install it in 😭 Still eyeballing an RV-8 though, so someday soon hopefully!

This seems like a no-brainer given that most of the leading causes of GA fatalities are rooted in a loss of situational awareness. So what am I missing?

Plus... the kid in me that wants to be a fighter pilot gets giddy imagining things like mounting a 365 camera to 'look through' the plane, or an NVG on gimbals for night flying. All this tech is available to consumers now, measured in hundreds "to" thousands instead of hundreds "of" thousands or millions. I think a 20KW phased-array projector is still out of reach though...
 

proppastie

Well-Known Member
Supporting Member
Joined
Feb 19, 2012
Messages
5,824
Location
NJ
sounds like an opportunity for you however pilots are very conservative so do not give up your day job.....looking at the visuals in the paper did not impress me.....We mostly fly for fun so looking out the window is very important. If you could also paint navigational data and traffic display on the cockpit glass ( HUD) I think you would might have a winner though.
 

Vigilant1

Well-Known Member
Supporting Member
Joined
Jan 24, 2011
Messages
8,448
Location
US
1. The HoloLens product itself seems to be an shaky ground, I wouldn't want to put a lot of effort into developing a product for hardware that MicroSoft might kill at any time.
2. Pilots may be crazy, but I think the sensible ones won't be crazy about shooting lasers onto the back of their eyes. Maybe it is 100% safe, but you should anticipate resistance.
3. It might sound easy, but I'd bet it isn't once into the details. Possible spatial disorientation, possible rod/cone temporary desensitization due to laser effects (so now the dimly illuminated traffic that would have been above the threshold for visual detection is now below that threshold), etc, etc.
 

addaon

Well-Known Member
Supporting Member
Joined
Feb 24, 2008
Messages
2,340
Location
Kanab, UT
I’ve played with the HoloLens — more time on the first gen than the second. I would be very concerned about ability to see the HUD against a daylight-bright background.

The system I’ve wanted to build is a laser projector sync’d with prop rotation and lighting up the back of the blades. Far enough to keep eye focus bear infinity.
 

athilenius

Member
Joined
Sep 25, 2022
Messages
5
Location
San Diego, CA, USA
"I would be very concerned about ability to see the HUD against a daylight-bright background" Nooo. That's not the news I was hoping for! Can you tint the visor? I'm also surprised you can't crank up the brightness, as it's direct light projection; there isn't really an (engineering) limit to brightness like there is for an LCD.

"laser projector sync’d with prop rotation and lighting up the back of the blades" That's clever! But then you need some serious projection power to see it in daytime, assuming diffuse light scattering by the blades. Even more so because you can only project while the blade is sweeping past. I think I would just go a traditional HUD route in that case.
 

addaon

Well-Known Member
Supporting Member
Joined
Feb 24, 2008
Messages
2,340
Location
Kanab, UT
Scanning laser projectors only light one point at a time anyway, so not a big difference there — if your “screen” is 90 degrees of the prop arc and you have three blades, you have a point lot up 3/4 of the time. But yeah, it would need to be a stupid bright projector.
 

bifft

Well-Known Member
Joined
Apr 17, 2011
Messages
394
Location
Utah
I looked into it a couple of years ago. Augmented Reality in Aviation My conclusion was that the hardware wasn't good enough yet. Looking at the Hololens specs it still has the tiny field of view and isn't described as focused at infinity. Still waiting for "good enough" hardware before I'd start a project.
 

athilenius

Member
Joined
Sep 25, 2022
Messages
5
Location
San Diego, CA, USA
I've been pinging everyone I know to see if anyone has a HoloLens V2 I can borrow. 3.5k is too much to drop on an experiment. I've read that the field of view on the V2 is pretty good (54 degrees). It doesn't need to be full vision, you just 'clamp' out-of-frame symbols to the edge of the FOV.

Microsoft recently released an update for the tracking software to allow them to work on a "moving platform" (car, plane, elevator... anything where 'gravity' disagrees with the visuals).

I would be immensely curious to see what kind of accuracy you can get form nothing but a HoloLens and a Sentry/Stratus/Stratux. My gut says that it wouldn't be enough to paint things on the ground, but probably enough to draw a diamond around traffic and almost certainly enough for an artificial horizon. Any symbology that follows your head doesn't require any tracking, so that's certainly doable. If the HoloLens is 'good enough', then all the software can run on their embedded hardware.
 

cluttonfred

Well-Known Member
Supporting Member
Joined
Feb 13, 2010
Messages
9,732
Location
World traveler
I think a less immersive solution is a much better idea for flying. You need something that does not distract the pilot from looking outside, a heads up display in the behavioral sense. I think smart glasses that project key instrument data outside the center of the field of view, just a flick of the eyes to check airspeed/altitude/course, would be great.
 

bifft

Well-Known Member
Joined
Apr 17, 2011
Messages
394
Location
Utah
I would be immensely curious to see what kind of accuracy you can get form nothing but a HoloLens and a Sentry/Stratus/Stratux. My gut says that it wouldn't be enough to paint things on the ground, but probably enough to draw a diamond around traffic and almost certainly enough for an artificial horizon. Any symbology that follows your head doesn't require any tracking, so that's certainly doable. If the HoloLens is 'good enough', then all the software can run on their embedded hardware.

This was my thought. I don't think you care where the helmet is inside the plane, just the location of the plane relative to ADSB traffic and the orientation of the glasses in space. Everything you care about is far enough outside the plane that a pilot with the seatbelt still on can't move enough to matter.

In addition to the artificial horizon, I think a projection of the current motion vector would be nice. Once at stable speed and direction it would show your touchdown point in a glide, if you have the right wind correction to go to that pass, etc.
 

Aesquire

Well-Known Member
Joined
Jul 28, 2014
Messages
3,353
Location
Rochester, NY, USA
I was on approach, everything was almost lined up, and suddenly all I could see was an ad for erectile dysfunction drugs with no visit to your doctor. Luckily, I was on autopilot. When that ended, I was just a few hundred yards from the threshold, when the ad for the new phone game....
 

athilenius

Member
Joined
Sep 25, 2022
Messages
5
Location
San Diego, CA, USA
I don't think you care where the helmet is inside the plane

Unfortunately this is only true for a very small subset of symbology.

There are 3 coordinate spaces you'll need to work in:
  • Head/headset space - symbols always floating directly in front of your vision (if any)
  • Aircraft space - symbols that are 'anchored' to the airframe, like a virtual HUD when you look forward, or 'virtual' avionics like a checklist floating next to your console, a power readout hovering over your throttle... the possibilities are endless...
  • World space - symbols that are anchored to the outside world, like artificial horizon, the prograde vector, and traffic diamonds
So you have two transformations, headset to aircraft space (this is generally called the "view" matrix in computer graphics) and a transformation from aircraft to world space (generally called a "model" matrix in computer graphics). There is are a few additional spaces that we can ignore, like projection space and offset spaces. All this sounds complicated, but it's really not. You create a 4X4 matrix for each transform, then it's pretty simple linear algebra. I've been doing computer graphics for a decade so this stuff is second nature at this point.

Now the shitty part: to get to world space, you need to multiply the view and model matrices together. Meaning... any error you have will be compounded. So if the headset tracking is 'meh' and the Stratus updates/tracking is 'meh' then the world-space symbology will be garbage. Fighter jets track the helmet to within a milliradian, and their IMUs are exceptional (laser gyros, et. al). But... I don't think you need that level of precision for our use case.

Also... I doubt you'll actually want any symbols permanently affixed to your head. Better would be to have a 'virtual HUD' that is anchored to the forward part of the airframe, something like this screenshot (from my favorite VR game, VTOL VR). It's modeled after an F-35, the HUD you see is purely virtual, the headset is painting it even though it looks like it's part of the plane (it stays in place as you turn your head).1664403897562.png

When you look far enough away from that HUD though, then a few symbols will be anchored to your head, like airspeed, altitude, and heading. It's too much for your brain to try and translate attitude information when you're looking sideways.

I was on approach, everything was almost lined up, and suddenly all I could see was an ad for erectile dysfunction drugs with no visit to your doctor. Luckily, I was on autopilot. When that ended, I was just a few hundred yards from the threshold, when the ad for the new phone game....
😂 That's just the freemium version! For as little as 12 easy payments of $19.99 a month you can have the add free version!

All joking aside, any effort I put into this will be 100% open-source and free. Zero interest in the liability that would come with trying to monetize a HMD for GA.
 
Top