Augmented Reality in Aviation

HomeBuiltAirplanes.com

Help Support HomeBuiltAirplanes.com:

bifft

Well-Known Member
Joined
Apr 17, 2011
Messages
275
Location
Utah
So, this little aside in the Rutan Catbird thread totally nerd-sniped me (https://xkcd.com/356/).

I actually meant the "flight instrument" display - an augmented-reality overlay laser-projected on the canopy, showing a "highway in the sky", positions of conflicting traffic, etc.. We'd be able to do that with AR goggles these days (and I'm amazed that only the R/C crowd has done it), but for the early '80's when the article was written, the entire concept was amazingly ahead of its time for light-airplane usage.
Spent quite a bit of time over the last couple of days researching, wanted to share what I found. Seems like more of a new thread thing.

Even though such a thing wouldn't really be all that useful with the kind of flying I do (recreational in good weather, aerobatics), it seems like it wouldn't be hard to do. At least from the software side. Also, it's actually something I'm qualified to create. Could get a beta version running in a couple of months of my free time.

Almost all of the data you would want would be available from an ADSB-in box. Combine that with orientation from the goggles (which they need to have to work at all) and the only thing missing is your flight plan for the "highway in the sky" part.

From orientation, display current heading, virtual horizon. From GPS position and history display current track, both horizontal and vertical. GPS altitude and ground speed also available (Baro alt and airspeed are desirable, but would require another piece of hardware tied into the pito static system.) Traffic alerts from ADSB-in.

Only need to use the orientation of the goggles, you don't really care where they are inside the plane. Compared to what we want to show the airplane is tiny, so shifting your head position doesn't need to change the image.

Something like this: (drawn over the top of a screen shot from recent flight)



Note that you can see a traffic alert (circle with line showing direction of travel) through the plane.

So, I did some research into what is available in AR goggles these days. Field of view is terrible, usually being around 22-25 degrees or so. Bigger than a smart phone at arms length, but not something that would be useful to alert you to traffic or to be a virtual horizon for staying rightside up in poor visibility. Even the big ones only go up to 50 degrees. That would be useable, if not what I would really want. Some people are working on wider ones, but they don't seem to be on the market yet.

Screen resolution is usually in the 1920x1080 range, so way more than enough.

More expensive than I thought, usually in the $1000+ range for the hardware. Many require an additional box somewhere on your body.

Cheap stuff like this: https://www.amazon.com/Headset-Glasses-Augmented-Reality-Android/dp/B07N2PXK8W is available, but that wouldn't have the image focused at infinity. Not good, the whole point is to make it so you are focused outside the plane on things that are far away.

Reading/listening to reviews they say things like "heavy, hot and uncomfortable". Maybe Ok for 5 minute demo, or a 15 minute battery limited drone flight, but not good for a 3-5 hour airplane flight. Not good for pulling Gs. They also mention frequent nausea in users, I believe from there being a lag between movement and the visual updates. These should get better as the tech improves.

I wasn't able to find any real specifications on the orientation sensors other than the number of gyroscopes and accelerometers. I couldn't tell if the orientation was gyro only or if they use the accelerometers to decide which way is up. Many also have a magnatometer, so would provide compass data. Could be a problem in flight.

TLDR: The software side would be easy, but I don't think the hardware is ready yet. Probably 2-5 years away. Curious if anyone has actually worked with AR goggles to see if my thoughts match reality.
 

ScaleBirdsScott

Well-Known Member
Joined
Feb 10, 2015
Messages
1,222
Location
Uncasville, CT
I've seen some early attempts at this by actually going through the "Innovation Station" tents or buildings or whatever at Oshkosh. They have some ways to go before its as clean as your screenshot, and most of them get too gaudy with graphics and "highway in the sky" sort of stuff, when what I think we all want is just a faithful F-14 HUD that goes with us wherever we look.

The one I saw most of was http://glass.aero/ but I don't remember seeing them at Osh 2019 so who knows where they are.

In a few more years we'll see some of this stuff get more common and the tech will improve rapidly. Already the dedicated VR headsets you can buy now are way better than the original Oculus Rift I picked up and used for all of 2 hours, and that was leagues better than the "Oculus for Cellphone" headsets I tried back on my old Samsung Note 4.

I'm guessing by 2022 we'll have the headset hardware to do the job being reasonably available. I'm thinking by around that time you'll see wearable flight goggles/glasses that are perhaps integrated with a headset that provides an active HUD with true focus at infinity, and will even be smart enough to detect the surroundings and, say, differentiate between images inside/outside the cockpit. I can think of so, so many cool features for AR that, probably, are already being done in other industries that if only applied to flying would be able to increase a lot of safety without getting in the way.

I'm also figuring by coordinating with something like a Wingbug and some bluetooth sensors, you might get away with an ultralight that has absolutely zero instrument panel.
 

Hephaestus

Well-Known Member
Joined
Jun 25, 2014
Messages
1,672
Location
YMM
Check out the Epson bt glasses. I played with some in the summer, pretty sure it's about what you're after.

The office ones are definitely more apropos than the drone ones.

Most of them run a gyro + accelerometers. Some of the off the shelf (inexpensive) uav controllers have a pitot input and would be another batch of sensors - wouldn't rely solely on the glass mounted one (think turbulent air).
 

BBerson

Well-Known Member
HBA Supporter
Joined
Dec 16, 2007
Messages
13,092
Location
Port Townsend WA
I think audio alerts into the headphones might be an option. " traffic 3 a-clock at 1800 feet" or even behind " traffic 6:00 at 2200 feet, turn right 20 degrees".
 

TFF

Well-Known Member
Joined
Apr 28, 2010
Messages
12,812
Location
Memphis, TN
This is what a Far Side would be. Looking through goggles two video lines right in the center view would say “ video data unavailable“. User would say, “ this sure is realistic “.
 

Hot Wings

Grumpy Cynic
HBA Supporter
Log Member
Joined
Nov 14, 2009
Messages
6,992
Location
Rocky Mountains
I think audio alerts into the headphones might be an option. " traffic 3 a-clock at 1800 feet" or even behind " traffic 6:00 at 2200 feet, turn right 20 degrees".
Could be even simpler than that with stereo head phones. Humans actually do a pretty passable job of echo location.

Still trying to figure out a Thevenin equivalent for those 2 dots. Probably going to take some calculus............
 

Hephaestus

Well-Known Member
Joined
Jun 25, 2014
Messages
1,672
Location
YMM
Could be even simpler than that with stereo head phones. Humans actually do a pretty passable job of echo location.

Still trying to figure out a Thevenin equivalent for those 2 dots. Probably going to take some calculus............
A couple of the drone projects already overlay adsb data ;) there's some copy and paste options out there.
 

FritzW

Well-Known Member
Log Member
Joined
Jan 30, 2011
Messages
3,886
Location
Las Cruces, NM

FritzW

Well-Known Member
Log Member
Joined
Jan 30, 2011
Messages
3,886
Location
Las Cruces, NM
Could be even simpler than that with stereo head phones. Humans actually do a pretty passable job of echo location.
Even with artificial, computer generated stereo sound it's amazing how well you can echo locate a noise.

In my video games, when I can't figure out where the bad guys are shooting at me from, it's usually because I have my headphones on backwards.
 

Pops

Well-Known Member
Log Member
Joined
Jan 1, 2013
Messages
8,016
Location
USA.
Even with artificial, computer generated stereo sound it's amazing how well you can echo locate a noise.

In my video games, when I can't figure out where the bad guys are shooting at me from, it's usually because I have my headphones on backwards.

What about people like me, only hearing on the right side . Lost hearing on left side from a building that I was in exploded from an collection of Acetylene gas in an elevator shaft. Don't remember it. I can't tell sound directions. Some frequencies I hear 80% and some just 20%. Nice and quiet where I live.
 

Hephaestus

Well-Known Member
Joined
Jun 25, 2014
Messages
1,672
Location
YMM
There's always finer points and user configuration things that can be managed later after a working demo exists.

Personally I'd be after a yellow or red highlight in the vicinity I'm looking for traffic. When the radios busy, I don't want more overlapping noise to try to filter through. But a colored highlight in the ballpark of where you'd expect to see that traffic wouldn't hurt.
 

Hephaestus

Well-Known Member
Joined
Jun 25, 2014
Messages
1,672
Location
YMM
The radio won't be busy when we get to the future separation scheme, whatever it is.
Haha and then when the FAA decides, 10yrs later transport Canada will roll out something similar but way more expensive to implement here. Like they did with adsb :D

I'll deal with upgrading later once that day comes :cool:
 
2
Top