Patent · US10838210B2 · B2 · US

Imaging modification, display and visualization using augmented and virtual reality eyewear

(11) Publication number

US10838210B2

(21) Application number

15/657,589

(22) Filing date

2017-07-24

(30) Priority date

2016-07-25

(43) Publication date

2020-11-17

(45) Date of grant

2020-11-17

(51) IPC

B60K 35/10; B60K 35/22; B60K 35/26; B60K 35/28; B60K 35/60; B60K 35/80; B60K 35/90; G02B 27/01; G06F 3/01; G06T 19/00; A61B 17/00; A61B 34/00; A61B 34/20; A61B 90/00; A61B 90/50; G06F 3/03

(52) CPC

G02B Optical elements, systems or apparatus: 27/0172, 27/0093, 27/01
A61B Diagnosis; surgery; identification: 2017/00203, 2017/00207, 2017/00216, 2034/2048, 2034/254, 2034/258, 2090/365, 2090/367, 2090/368, 2090/371, 2090/372, 2090/378, 2090/502, 34/00, 34/25, 90/36, 90/37
B60K Arrangement or mounting of propulsion units or of transmissions in vehicles; arrangement or mounting of plural diverse prime-movers in vehicles; auxiliary drives for vehicles; instrumentation or dashboards for vehicles; arrangements in connection with cooling, air intake, gas exhaust or fuel supply of propulsion units in vehicles: 35/10, 35/22, 35/26, 35/28, 35/60, 35/80, 35/90
G06F Electric digital data processing: 3/011, 3/012, 3/013, 3/017, 3/0304
G06T Image data processing or generation, in general: 19/006, 19/20
G16H Healthcare informatics, i.e. information and communication technology [ICT] specially adapted for the handling or processing of medical or healthcare data: 20/40

(73) Assignee

Magic Leap Inc

(72) Inventors

Nastasja U. Robaina; Nicole Elizabeth Samec; Christopher M. Harrises; Rony Abovitz; Mark Baerenrodt; Brian Lloyd Schmidt

(54) Title

Imaging modification, display and visualization using augmented and virtual reality eyewear

(57) Abstract

A display system can include a head-mounted display configured to project light to an eye of a user to display augmented reality image content to the user. The display system can include one or more user sensors configured to sense the user and can include one or more environmental sensors configured to sense surroundings of the user. The display system can also include processing electronics in communication with the display, the one or more user sensors, and the one or more environmental sensors. The processing electronics can be configured to sense a situation involving user focus, determine user intent for the situation, and alter user perception of a real or virtual object within the vision field of the user based at least in part on the user intent and/or sensed situation involving user focus. The processing electronics can be configured to at least one of enhance or de-emphasize the user perception of the real or virtual object within the vision field of the user.

Full text

View on Google Patents

Claims (31)

A head-mounted display system configured to project light to an eye of a user to display augmented reality image content in a vision field of said user, said head-mounted display system comprising: a frame configured to be supported on a head of the user; a head-mounted display disposed on the frame, said display configured to project light into said user's eye to display augmented reality image content to the user's vision field at different amounts of divergences as if projected from different distances from the user's eye, at least a portion of said display being transparent and disposed at a location in front of the user's eye when the user wears said head-mounted display device such that said transparent portion transmits light from a portion of the environment in front of the user and said head-mounted display to the user's eye to provide a view of said portion of the environment in front of the user and said head-mounted display; one or more user sensors configured to sense the user; one or more environmental sensors configured to sense surroundings of the user; and processing electronics in communication with the display, the one or more user sensors, and the one or more environmental sensors, the processing electronics configured to: sense a situation involving increased user focus; and alter user perception of a real or virtual peripheral object within the vision field of the user based at least in part on sensing the increased focus on the peripheral object, wherein sensing the increased user focus on the peripheral object is based at least in part on measurement of accomodation of the user's eye when directed at the peripheral object, and wherein altering user perception comprises moving image content comprising the peripheral object from a peripheral location to a central location.
The display system of claim 1, wherein altering user perception further comprises altering at least one of contrast, opacity, color, color saturation, color balance, size, background, brightness, edges, or sharpness of the moved image content comprising the peripheral object.
The display system of claim 1, wherein the one or more environmental sensors comprise a depth sensor, a pair of binocular world cameras, a geolocation sensor, a proximity sensor, or a GPS.
The display system of claim 1, wherein the one or more user sensors comprise one or more cameras.
The display system of claim 1, wherein said situation involving increased user focus comprises driving a motor vehicle.
The display system of claim 1, wherein said one or more environmental sensors comprise a sensor configured to detect a radio signal.
The display system of claim 1, wherein said one or more environmental sensors comprise a sensor configured to detect a blue tooth signal from an automobile.
The display system of claim 5, wherein said processing electronics are configured to alter user perception of said real or virtual object within the vision field of the user based at least in part on one or more data records regarding the user, said one or more data records comprising a driving record of said user.
A head-mounted display system configured to project light to an eye of a user to display augmented reality image content, said user's eye having a vision field having a central region and a peripheral region disposed about said central region, said head-mounted display system comprising: a frame configured to be supported on a head of the user; a head-mounted display disposed on the frame, said display configured to project light into said user's eye so as to present image content at said central region of said user's vision field, at least a portion of said display being transparent and disposed at a location in front of the user's eye when the user wears said head-mounted display device such that said transparent portion transmits light from a portion of the environment in front of the user and said head-mounted display to the user's eye to provide a view of said portion of the environment in front of the user and said head-mounted display; and processing electronics in communication with said display to control presentation of image content on said display, the processing electronics configured to: sense a situation involving increased user focus; and alter user perception of a real or virtual peripheral object within the vision field of the user based at least in part on sensing the increased focus on the peripheral object, wherein altering user perception comprises presenting image content to said peripheral region of the user's vision field that is enhanced in comparison to image content presented to the central region of the user's vision field.
The system of claim 9, further comprising one or more sensors configured to monitor the environment.
The system of claim 10, wherein said one or more sensors comprise one or more outward-facing image capture devices configured to image said environment.
The system of claim 10, wherein said one or more sensors comprise a distance measuring device.
The system of claim 9, further comprising an eye tracking device configured to track position and/or movement of said user's eye.
The system of claim 9, wherein the head-mounted display device is configured to process image content presented to at least a portion of said peripheral region of the user's vision field differently in comparison to image content presented to the central region of the user's vision field.
The system of claim 14, wherein the head-mounted display device is configured to process image content differently by magnifying image content presented to at least a portion of said peripheral region of the user's vision field in comparison to image content presented to the central region of the user's vision field.
The system of claim 14, wherein the head-mounted display device is configured to process image content differently by increasing brightness in image content presented to at least a portion of said peripheral region of the user's vision field in comparison to image content presented to the central region of the user's vision field.
The display system of claim 9, wherein the processing electronics is configured to sense a situation involving increased user focus on the peripheral object based at least in part on the measurement of accommodation of the user's eye when directed at the peripheral object.
The display system of claim 3, wherein the one or more environmental sensors comprise a depth sensor or a GPS.
The display system of claim 3, wherein the one or more environmental sensors comprise a pair of binocular world cameras.
The display system of claim 3, wherein the one or more environmental sensors comprise a geolocation sensor or a proximity sensor.
A head-mounted display system configured to project light to an eye of a user to display augmented reality image content in a vision field of said user, said head-mounted display system comprising: a frame configured to be supported on a head of the user; a head-mounted display disposed on the frame, said display configured to project light into said user's eye to display augmented reality image content to the user's vision field at different amounts of divergences as if projected from different distances from the user's eye, at least a portion of said display being transparent and disposed at a location in front of the user's eye when the user wears said head-mounted display device such that said transparent portion transmits light from a portion of the environment in front of the user and said head-mounted display to the user's eye to provide a view of said portion of the environment in front of the user and said head-mounted display; one or more user sensors configured to sense the user; one or more environmental sensors configured to sense surroundings of the user; and processing electronics in communication with the display, the one or more user sensors, and the one or more environmental sensors, the processing electronics configured to: sense a situation involving increased user focus; and alter user perception of a real or virtual peripheral object within the vision field of the user based at least in part on sensing the increased focus on the peripheral object without reliance on head movement towards the peripheral object, wherein altering user perception comprises moving image content comprising the peripheral object from a peripheral location to a central location.
The display system of claim 21, wherein altering user perception further comprises altering at least one of contrast, opacity, color, color saturation, color balance, size, background, brightness, edges, or sharpness of the moved image content comprising the peripheral object.
The display system of claim 21, wherein the one or more environmental sensors comprise a depth sensor or a GPS.
The display system of claim 21, wherein the one or more environmental sensors comprise a pair of binocular world cameras.
The display system of claim 21, wherein the one or more environmental sensors comprise a geolocation sensor or a proximity sensor.
The display system of claim 21, wherein the one or more user sensors comprise one or more cameras.
The display system of claim 21, wherein said situation involving increased user focus comprises driving a motor vehicle.
The display system of claim 21, wherein said one or more environmental sensors comprise a sensor configured to detect a radio signal.
The display system of claim 21, wherein said one or more environmental sensors comprise a sensor configured to detect a blue tooth signal from an automobile.
The display system of claim 27, wherein said processing electronics are configured to alter user perception of said real or virtual object within the vision field of the user based at least in part on one or more data records regarding the user, said one or more data records comprising a driving record of said user.
The display system of claim 9, further comprising one or more sensors, wherein the processing electronics is in communication with the one or more sensors.

Description

The present disclosure relates to display systems and, more particularly, to augmented reality display systems.

Modern computing and display technologies have facilitated the development of systems for so called “virtual reality” or “augmented reality” experiences, wherein digitally reproduced images or portions thereof are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A virtual reality, or “VR”, scenario typically involves presentation of digital or virtual image information without transparency to other actual real-world visual input; an augmented reality, or “AR”, scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual world around the user. A mixed reality, or “MR”, scenario is a type of AR scenario and typically involves virtual objects that are integrated into, and responsive to, the natural world. For example, in an MR scenario, AR image content may be blocked by or otherwise be perceived as interacting with objects in the real world.

Referring to FIG. 1A, an augmented reality scene 1 is depicted wherein a user of an AR technology sees a real-world park- like setting 1100 featuring people, trees, buildings in the background, and a concrete platform 1120.

Citations (70)

US9081426B2
US6850221B1
US20130250135A1
US20070132663A1
US8681256B2
US20110112756A1
US20120212399A1
US20160161740A1
US20120127062A1
US20120154557A1
US9213405B2
US20160011419A1
US9348143B2
US20130125027A1
US8223088B1
US8988474B2
US9323325B2
US20130082922A1
US20130117377A1
US9215293B2
US8950867B2
US20140177023A1
US20140218468A1
US9547174B2
US9851563B2
US9671566B2
US20150016777A1
US9310559B2
US20150309263A2
US9740006B2
US20140071539A1
US20140125558A1
US20140192084A1
US20140198017A1
US20150268415A1
US20140306866A1
US20140267410A1
US20140267420A1
US9417452B2
US9470906B2
US20150103306A1
WO2015077766A1
US20150205126A1
US9791700B2
US9874749B2
US20150241707A1
US20150178939A1
US20160131912A1
US20150222884A1
US20150222883A1
US20160026253A1
US20170206691A1
US20150302652A1
WO2015161307A1
US20160085302A1
US20150326570A1
US20150346495A1
US9857591B2
US20150346490A1
US20160098861A1
US20170308262A1
US20160131905A1
US20160203359A1
US20160270656A1
USD758367S1
US20170109936A1
US20170323485A1
US20170365101A1
WO2018022523A1
US20180136486A1

Record as JSON

{
  "publication_number": "US10838210B2",
  "country": "US",
  "kind": "B2",
  "title": "Imaging modification, display and visualization using augmented and virtual reality eyewear",
  "abstract": "A display system can include a head-mounted display configured to project light to an eye of a user to display augmented reality image content to the user. The display system can include one or more user sensors configured to sense the user and can include one or more environmental sensors configured to sense surroundings of the user. The display system can also include processing electronics in communication with the display, the one or more user sensors, and the one or more environmental sensors. The processing electronics can be configured to sense a situation involving user focus, determine user intent for the situation, and alter user perception of a real or virtual object within the vision field of the user based at least in part on the user intent and/or sensed situation involving user focus. The processing electronics can be configured to at least one of enhance or de-emphasize the user perception of the real or virtual object within the vision field of the user.",
  "claims": [
    "1. A head-mounted display system configured to project light to an eye of a user to display augmented reality image content in a vision field of said user, said head-mounted display system comprising: a frame configured to be supported on a head of the user; a head-mounted display disposed on the frame, said display configured to project light into said user's eye to display augmented reality image content to the user's vision field at different amounts of divergences as if projected from different distances from the user's eye, at least a portion of said display being transparent and disposed at a location in front of the user's eye when the user wears said head-mounted display device such that said transparent portion transmits light from a portion of the environment in front of the user and said head-mounted display to the user's eye to provide a view of said portion of the environment in front of the user and said head-mounted display; one or more user sensors configured to sense the user; one or more environmental sensors configured to sense surroundings of the user; and processing electronics in communication with the display, the one or more user sensors, and the one or more environmental sensors, the processing electronics configured to: sense a situation involving increased user focus; and alter user perception of a real or virtual peripheral object within the vision field of the user based at least in part on sensing the increased focus on the peripheral object, wherein sensing the increased user focus on the peripheral object is based at least in part on measurement of accomodation of the user's eye when directed at the peripheral object, and wherein altering user perception comprises moving image content comprising the peripheral object from a peripheral location to a central location.",
    "2. The display system of claim 1, wherein altering user perception further comprises altering at least one of contrast, opacity, color, color saturation, color balance, size, background, brightness, edges, or sharpness of the moved image content comprising the peripheral object.",
    "3. The display system of claim 1, wherein the one or more environmental sensors comprise a depth sensor, a pair of binocular world cameras, a geolocation sensor, a proximity sensor, or a GPS.",
    "4. The display system of claim 1, wherein the one or more user sensors comprise one or more cameras.",
    "5. The display system of claim 1, wherein said situation involving increased user focus comprises driving a motor vehicle.",
    "6. The display system of claim 1, wherein said one or more environmental sensors comprise a sensor configured to detect a radio signal.",
    "7. The display system of claim 1, wherein said one or more environmental sensors comprise a sensor configured to detect a blue tooth signal from an automobile.",
    "8. The display system of claim 5, wherein said processing electronics are configured to alter user perception of said real or virtual object within the vision field of the user based at least in part on one or more data records regarding the user, said one or more data records comprising a driving record of said user.",
    "9. A head-mounted display system configured to project light to an eye of a user to display augmented reality image content, said user's eye having a vision field having a central region and a peripheral region disposed about said central region, said head-mounted display system comprising: a frame configured to be supported on a head of the user; a head-mounted display disposed on the frame, said display configured to project light into said user's eye so as to present image content at said central region of said user's vision field, at least a portion of said display being transparent and disposed at a location in front of the user's eye when the user wears said head-mounted display device such that said transparent portion transmits light from a portion of the environment in front of the user and said head-mounted display to the user's eye to provide a view of said portion of the environment in front of the user and said head-mounted display; and processing electronics in communication with said display to control presentation of image content on said display, the processing electronics configured to: sense a situation involving increased user focus; and alter user perception of a real or virtual peripheral object within the vision field of the user based at least in part on sensing the increased focus on the peripheral object, wherein altering user perception comprises presenting image content to said peripheral region of the user's vision field that is enhanced in comparison to image content presented to the central region of the user's vision field.",
    "10. The system of claim 9, further comprising one or more sensors configured to monitor the environment.",
    "11. The system of claim 10, wherein said one or more sensors comprise one or more outward-facing image capture devices configured to image said environment.",
    "12. The system of claim 10, wherein said one or more sensors comprise a distance measuring device.",
    "13. The system of claim 9, further comprising an eye tracking device configured to track position and/or movement of said user's eye.",
    "14. The system of claim 9, wherein the head-mounted display device is configured to process image content presented to at least a portion of said peripheral region of the user's vision field differently in comparison to image content presented to the central region of the user's vision field.",
    "15. The system of claim 14, wherein the head-mounted display device is configured to process image content differently by magnifying image content presented to at least a portion of said peripheral region of the user's vision field in comparison to image content presented to the central region of the user's vision field.",
    "16. The system of claim 14, wherein the head-mounted display device is configured to process image content differently by increasing brightness in image content presented to at least a portion of said peripheral region of the user's vision field in comparison to image content presented to the central region of the user's vision field.",
    "17. The display system of claim 9, wherein the processing electronics is configured to sense a situation involving increased user focus on the peripheral object based at least in part on the measurement of accommodation of the user's eye when directed at the peripheral object.",
    "18. The display system of claim 3, wherein the one or more environmental sensors comprise a depth sensor or a GPS.",
    "19. The display system of claim 3, wherein the one or more environmental sensors comprise a pair of binocular world cameras.",
    "20. The display system of claim 3, wherein the one or more environmental sensors comprise a geolocation sensor or a proximity sensor.",
    "21. A head-mounted display system configured to project light to an eye of a user to display augmented reality image content in a vision field of said user, said head-mounted display system comprising: a frame configured to be supported on a head of the user; a head-mounted display disposed on the frame, said display configured to project light into said user's eye to display augmented reality image content to the user's vision field at different amounts of divergences as if projected from different distances from the user's eye, at least a portion of said display being transparent and disposed at a location in front of the user's eye when the user wears said head-mounted display device such that said transparent portion transmits light from a portion of the environment in front of the user and said head-mounted display to the user's eye to provide a view of said portion of the environment in front of the user and said head-mounted display; one or more user sensors configured to sense the user; one or more environmental sensors configured to sense surroundings of the user; and processing electronics in communication with the display, the one or more user sensors, and the one or more environmental sensors, the processing electronics configured to: sense a situation involving increased user focus; and alter user perception of a real or virtual peripheral object within the vision field of the user based at least in part on sensing the increased focus on the peripheral object without reliance on head movement towards the peripheral object, wherein altering user perception comprises moving image content comprising the peripheral object from a peripheral location to a central location.",
    "22. The display system of claim 21, wherein altering user perception further comprises altering at least one of contrast, opacity, color, color saturation, color balance, size, background, brightness, edges, or sharpness of the moved image content comprising the peripheral object.",
    "23. The display system of claim 21, wherein the one or more environmental sensors comprise a depth sensor or a GPS.",
    "24. The display system of claim 21, wherein the one or more environmental sensors comprise a pair of binocular world cameras.",
    "25. The display system of claim 21, wherein the one or more environmental sensors comprise a geolocation sensor or a proximity sensor.",
    "26. The display system of claim 21, wherein the one or more user sensors comprise one or more cameras.",
    "27. The display system of claim 21, wherein said situation involving increased user focus comprises driving a motor vehicle.",
    "28. The display system of claim 21, wherein said one or more environmental sensors comprise a sensor configured to detect a radio signal.",
    "29. The display system of claim 21, wherein said one or more environmental sensors comprise a sensor configured to detect a blue tooth signal from an automobile.",
    "30. The display system of claim 27, wherein said processing electronics are configured to alter user perception of said real or virtual object within the vision field of the user based at least in part on one or more data records regarding the user, said one or more data records comprising a driving record of said user.",
    "31. The display system of claim 9, further comprising one or more sensors, wherein the processing electronics is in communication with the one or more sensors."
  ],
  "description_excerpt": "The present disclosure relates to display systems and, more particularly, to augmented reality display systems.\n\nModern computing and display technologies have facilitated the development of systems for so called “virtual reality” or “augmented reality” experiences, wherein digitally reproduced images or portions thereof are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A virtual reality, or “VR”, scenario typically involves presentation of digital or virtual image information without transparency to other actual real-world visual input; an augmented reality, or “AR”, scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual world around the user. A mixed reality, or “MR”, scenario is a type of AR scenario and typically involves virtual objects that are integrated into, and responsive to, the natural world. For example, in an MR scenario, AR image content may be blocked by or otherwise be perceived as interacting with objects in the real world.\n\nReferring to FIG. 1A, an augmented reality scene 1 is depicted wherein a user of an AR technology sees a real-world park- like setting 1100 featuring people, trees, buildings in the background, and a concrete platform 1120.",
  "cpc": [
    "G02B 27/0172",
    "A61B 2017/00203",
    "A61B 2017/00207",
    "A61B 2017/00216",
    "A61B 2034/2048",
    "A61B 2034/254",
    "A61B 2034/258",
    "A61B 2090/365",
    "A61B 2090/367",
    "A61B 2090/368",
    "A61B 2090/371",
    "A61B 2090/372",
    "A61B 2090/378",
    "A61B 2090/502",
    "A61B 34/00",
    "A61B 34/25",
    "A61B 90/36",
    "A61B 90/37",
    "B60K 35/10",
    "B60K 35/22",
    "B60K 35/26",
    "B60K 35/28",
    "B60K 35/60",
    "B60K 35/80",
    "B60K 35/90",
    "G02B 27/0093",
    "G02B 27/01",
    "G06F 3/011",
    "G06F 3/012",
    "G06F 3/013",
    "G06F 3/017",
    "G06F 3/0304",
    "G06T 19/006",
    "G06T 19/20",
    "G16H 20/40"
  ],
  "ipc": [
    "B60K 35/10",
    "B60K 35/22",
    "B60K 35/26",
    "B60K 35/28",
    "B60K 35/60",
    "B60K 35/80",
    "B60K 35/90",
    "G02B 27/01",
    "G06F 3/01",
    "G06T 19/00",
    "A61B 17/00",
    "A61B 34/00",
    "A61B 34/20",
    "A61B 90/00",
    "A61B 90/50",
    "G06F 3/03"
  ],
  "assignees": [
    "Magic Leap Inc"
  ],
  "inventors": [
    "Nastasja U. Robaina",
    "Nicole Elizabeth Samec",
    "Christopher M. Harrises",
    "Rony Abovitz",
    "Mark Baerenrodt",
    "Brian Lloyd Schmidt"
  ],
  "filing_date": "2017-07-24",
  "publication_date": "2020-11-17",
  "grant_date": "2020-11-17",
  "priority_date": "2016-07-25",
  "application_number": "US-201715657589-A",
  "family_id": "61016802",
  "cited_by_count": 42,
  "citations": [
    "US9081426B2",
    "US6850221B1",
    "US20130250135A1",
    "US20070132663A1",
    "US8681256B2",
    "US20110112756A1",
    "US20120212399A1",
    "US20160161740A1",
    "US20120127062A1",
    "US20120154557A1",
    "US9213405B2",
    "US20160011419A1",
    "US9348143B2",
    "US20130125027A1",
    "US8223088B1",
    "US8988474B2",
    "US9323325B2",
    "US20130082922A1",
    "US20130117377A1",
    "US9215293B2",
    "US8950867B2",
    "US20140177023A1",
    "US20140218468A1",
    "US9547174B2",
    "US9851563B2",
    "US9671566B2",
    "US20150016777A1",
    "US9310559B2",
    "US20150309263A2",
    "US9740006B2",
    "US20140071539A1",
    "US20140125558A1",
    "US20140192084A1",
    "US20140198017A1",
    "US20150268415A1",
    "US20140306866A1",
    "US20140267410A1",
    "US20140267420A1",
    "US9417452B2",
    "US9470906B2",
    "US20150103306A1",
    "WO2015077766A1",
    "US20150205126A1",
    "US9791700B2",
    "US9874749B2",
    "US20150241707A1",
    "US20150178939A1",
    "US20160131912A1",
    "US20150222884A1",
    "US20150222883A1",
    "US20160026253A1",
    "US20170206691A1",
    "US20150302652A1",
    "WO2015161307A1",
    "US20160085302A1",
    "US20150326570A1",
    "US20150346495A1",
    "US9857591B2",
    "US20150346490A1",
    "US20160098861A1",
    "US20170308262A1",
    "US20160131905A1",
    "US20160203359A1",
    "US20160270656A1",
    "USD758367S1",
    "US20170109936A1",
    "US20170323485A1",
    "US20170365101A1",
    "WO2018022523A1",
    "US20180136486A1"
  ]
}

Record 1,894 of 8,000 in Patents full text (MLC-0201). Request the full dataset.