Augmented Reality in Aviation

Discussion in 'Aircraft Design / Aerodynamics / New Technology' started by bifft, Oct 1, 2019.

Help Support HomeBuiltAirplanes Forum by donating:

  1. Oct 1, 2019 #1

    bifft

    bifft

    bifft

    Well-Known Member

    Joined:
    Apr 17, 2011
    Messages:
    234
    Likes Received:
    87
    Location:
    Utah
    So, this little aside in the Rutan Catbird thread totally nerd-sniped me (https://xkcd.com/356/).

    Spent quite a bit of time over the last couple of days researching, wanted to share what I found. Seems like more of a new thread thing.

    Even though such a thing wouldn't really be all that useful with the kind of flying I do (recreational in good weather, aerobatics), it seems like it wouldn't be hard to do. At least from the software side. Also, it's actually something I'm qualified to create. Could get a beta version running in a couple of months of my free time.

    Almost all of the data you would want would be available from an ADSB-in box. Combine that with orientation from the goggles (which they need to have to work at all) and the only thing missing is your flight plan for the "highway in the sky" part.

    From orientation, display current heading, virtual horizon. From GPS position and history display current track, both horizontal and vertical. GPS altitude and ground speed also available (Baro alt and airspeed are desirable, but would require another piece of hardware tied into the pito static system.) Traffic alerts from ADSB-in.

    Only need to use the orientation of the goggles, you don't really care where they are inside the plane. Compared to what we want to show the airplane is tiny, so shifting your head position doesn't need to change the image.

    Something like this: (drawn over the top of a screen shot from recent flight)

    [​IMG]

    Note that you can see a traffic alert (circle with line showing direction of travel) through the plane.

    So, I did some research into what is available in AR goggles these days. Field of view is terrible, usually being around 22-25 degrees or so. Bigger than a smart phone at arms length, but not something that would be useful to alert you to traffic or to be a virtual horizon for staying rightside up in poor visibility. Even the big ones only go up to 50 degrees. That would be useable, if not what I would really want. Some people are working on wider ones, but they don't seem to be on the market yet.

    Screen resolution is usually in the 1920x1080 range, so way more than enough.

    More expensive than I thought, usually in the $1000+ range for the hardware. Many require an additional box somewhere on your body.

    Cheap stuff like this: https://www.amazon.com/Headset-Glasses-Augmented-Reality-Android/dp/B07N2PXK8W is available, but that wouldn't have the image focused at infinity. Not good, the whole point is to make it so you are focused outside the plane on things that are far away.

    Reading/listening to reviews they say things like "heavy, hot and uncomfortable". Maybe Ok for 5 minute demo, or a 15 minute battery limited drone flight, but not good for a 3-5 hour airplane flight. Not good for pulling Gs. They also mention frequent nausea in users, I believe from there being a lag between movement and the visual updates. These should get better as the tech improves.

    I wasn't able to find any real specifications on the orientation sensors other than the number of gyroscopes and accelerometers. I couldn't tell if the orientation was gyro only or if they use the accelerometers to decide which way is up. Many also have a magnatometer, so would provide compass data. Could be a problem in flight.

    TLDR: The software side would be easy, but I don't think the hardware is ready yet. Probably 2-5 years away. Curious if anyone has actually worked with AR goggles to see if my thoughts match reality.
     
  2. Oct 1, 2019 #2

    ScaleBirdsScott

    ScaleBirdsScott

    ScaleBirdsScott

    Well-Known Member

    Joined:
    Feb 10, 2015
    Messages:
    1,034
    Likes Received:
    656
    Location:
    Uncasville, CT
    I've seen some early attempts at this by actually going through the "Innovation Station" tents or buildings or whatever at Oshkosh. They have some ways to go before its as clean as your screenshot, and most of them get too gaudy with graphics and "highway in the sky" sort of stuff, when what I think we all want is just a faithful F-14 HUD that goes with us wherever we look.

    The one I saw most of was http://glass.aero/ but I don't remember seeing them at Osh 2019 so who knows where they are.

    In a few more years we'll see some of this stuff get more common and the tech will improve rapidly. Already the dedicated VR headsets you can buy now are way better than the original Oculus Rift I picked up and used for all of 2 hours, and that was leagues better than the "Oculus for Cellphone" headsets I tried back on my old Samsung Note 4.

    I'm guessing by 2022 we'll have the headset hardware to do the job being reasonably available. I'm thinking by around that time you'll see wearable flight goggles/glasses that are perhaps integrated with a headset that provides an active HUD with true focus at infinity, and will even be smart enough to detect the surroundings and, say, differentiate between images inside/outside the cockpit. I can think of so, so many cool features for AR that, probably, are already being done in other industries that if only applied to flying would be able to increase a lot of safety without getting in the way.

    I'm also figuring by coordinating with something like a Wingbug and some bluetooth sensors, you might get away with an ultralight that has absolutely zero instrument panel.
     
  3. Oct 1, 2019 #3

    Hephaestus

    Hephaestus

    Hephaestus

    Well-Known Member

    Joined:
    Jun 25, 2014
    Messages:
    1,180
    Likes Received:
    258
    Location:
    YMM
    Check out the Epson bt glasses. I played with some in the summer, pretty sure it's about what you're after.

    The office ones are definitely more apropos than the drone ones.

    Most of them run a gyro + accelerometers. Some of the off the shelf (inexpensive) uav controllers have a pitot input and would be another batch of sensors - wouldn't rely solely on the glass mounted one (think turbulent air).
     
  4. Oct 1, 2019 #4

    BBerson

    BBerson

    BBerson

    Well-Known Member HBA Supporter

    Joined:
    Dec 16, 2007
    Messages:
    12,062
    Likes Received:
    2,355
    Location:
    Port Townsend WA
    I think audio alerts into the headphones might be an option. " traffic 3 a-clock at 1800 feet" or even behind " traffic 6:00 at 2200 feet, turn right 20 degrees".
     
  5. Oct 1, 2019 #5

    BJC

    BJC

    BJC

    Well-Known Member

    Joined:
    Oct 7, 2013
    Messages:
    9,561
    Likes Received:
    6,331
    Location:
    97FL, Florida, USA
    I really miss The Far Side.


    BJC
     
    Mcmark and MadRocketScientist like this.
  6. Oct 1, 2019 #6

    Dana

    Dana

    Dana

    Super Moderator Staff Member

    Joined:
    Apr 4, 2007
    Messages:
    8,629
    Likes Received:
    3,012
    Location:
    CT, USA
    It's coming back!

    But xkcd is cool too...
     
  7. Oct 1, 2019 #7

    BJC

    BJC

    BJC

    Well-Known Member

    Joined:
    Oct 7, 2013
    Messages:
    9,561
    Likes Received:
    6,331
    Location:
    97FL, Florida, USA
    Thanks, Dana.


    BJC
     
  8. Oct 1, 2019 #8

    TFF

    TFF

    TFF

    Well-Known Member

    Joined:
    Apr 28, 2010
    Messages:
    11,644
    Likes Received:
    3,273
    Location:
    Memphis, TN
    This is what a Far Side would be. Looking through goggles two video lines right in the center view would say “ video data unavailable“. User would say, “ this sure is realistic “.
     
  9. Oct 1, 2019 #9

    Hot Wings

    Hot Wings

    Hot Wings

    Well-Known Member HBA Supporter

    Joined:
    Nov 14, 2009
    Messages:
    6,391
    Likes Received:
    2,311
    Location:
    Rocky Mountains
    Could be even simpler than that with stereo head phones. Humans actually do a pretty passable job of echo location.

    Still trying to figure out a Thevenin equivalent for those 2 dots. Probably going to take some calculus............
     
  10. Oct 1, 2019 #10

    bmcj

    bmcj

    bmcj

    Well-Known Member HBA Supporter

    Joined:
    Apr 10, 2007
    Messages:
    12,918
    Likes Received:
    4,867
    Location:
    Fresno, California
    FOOOOM
     
  11. Oct 1, 2019 #11

    Hephaestus

    Hephaestus

    Hephaestus

    Well-Known Member

    Joined:
    Jun 25, 2014
    Messages:
    1,180
    Likes Received:
    258
    Location:
    YMM
    A couple of the drone projects already overlay adsb data ;) there's some copy and paste options out there.
     
  12. Oct 1, 2019 #12

    FritzW

    FritzW

    FritzW

    Well-Known Member

    Joined:
    Jan 31, 2011
    Messages:
    3,662
    Likes Received:
    3,278
    Location:
    Las Cruces, NM
    Doesn't the shape of the reflectors focus the *images at infinity?

    *I'm assuming the HUD would be a pair of stereo images

    index.jpeg
     
  13. Oct 1, 2019 #13

    Hephaestus

    Hephaestus

    Hephaestus

    Well-Known Member

    Joined:
    Jun 25, 2014
    Messages:
    1,180
    Likes Received:
    258
    Location:
    YMM
  14. Oct 1, 2019 #14

    FritzW

    FritzW

    FritzW

    Well-Known Member

    Joined:
    Jan 31, 2011
    Messages:
    3,662
    Likes Received:
    3,278
    Location:
    Las Cruces, NM
    Even with artificial, computer generated stereo sound it's amazing how well you can echo locate a noise.

    In my video games, when I can't figure out where the bad guys are shooting at me from, it's usually because I have my headphones on backwards.
     
    ScaleBirdsScott likes this.
  15. Oct 2, 2019 #15

    Pops

    Pops

    Pops

    Well-Known Member

    Joined:
    Jan 1, 2013
    Messages:
    7,142
    Likes Received:
    6,031
    Location:
    USA.

    What about people like me, only hearing on the right side . Lost hearing on left side from a building that I was in exploded from an collection of Acetylene gas in an elevator shaft. Don't remember it. I can't tell sound directions. Some frequencies I hear 80% and some just 20%. Nice and quiet where I live.
     
  16. Oct 2, 2019 #16

    FritzW

    FritzW

    FritzW

    Well-Known Member

    Joined:
    Jan 31, 2011
    Messages:
    3,662
    Likes Received:
    3,278
    Location:
    Las Cruces, NM
    ...don't play Arma 3, it won't go well ;)
     
    Pops likes this.
  17. Oct 2, 2019 #17

    Hephaestus

    Hephaestus

    Hephaestus

    Well-Known Member

    Joined:
    Jun 25, 2014
    Messages:
    1,180
    Likes Received:
    258
    Location:
    YMM
    There's always finer points and user configuration things that can be managed later after a working demo exists.

    Personally I'd be after a yellow or red highlight in the vicinity I'm looking for traffic. When the radios busy, I don't want more overlapping noise to try to filter through. But a colored highlight in the ballpark of where you'd expect to see that traffic wouldn't hurt.
     
  18. Oct 2, 2019 #18

    BBerson

    BBerson

    BBerson

    Well-Known Member HBA Supporter

    Joined:
    Dec 16, 2007
    Messages:
    12,062
    Likes Received:
    2,355
    Location:
    Port Townsend WA
    The radio won't be busy when we get to the future separation scheme, whatever it is.
     
  19. Oct 2, 2019 #19

    Hephaestus

    Hephaestus

    Hephaestus

    Well-Known Member

    Joined:
    Jun 25, 2014
    Messages:
    1,180
    Likes Received:
    258
    Location:
    YMM
    Haha and then when the FAA decides, 10yrs later transport Canada will roll out something similar but way more expensive to implement here. Like they did with adsb :D

    I'll deal with upgrading later once that day comes :cool:
     
  20. Oct 2, 2019 #20

    rdj

    rdj

    rdj

    Well-Known Member

    Joined:
    Sep 10, 2009
    Messages:
    293
    Likes Received:
    186
    Location:
    Northern California

Share This Page



arrow_white