Patent · US10466953B2 · B2 · US

Sharing neighboring map data across devices

(11) Publication number

US10466953B2

(21) Application number

15/593,147

(22) Filing date

2017-05-11

(30) Priority date

2017-03-30

(43) Publication date

2019-11-05

(45) Date of grant

2019-11-05

(51) IPC

B65G 1/04; G02B 27/01; G03H 1/00; G03H 1/22; G06F 3/01; G06F 3/0481; G06F 3/14; G06F 3/147; G06T 19/00; G09G 3/00

(52) CPC

G06F Electric digital data processing: 3/1454, 3/011, 3/012, 3/04815, 3/1423, 3/147
B65G Transport or storage devices, e.g. conveyors for loading or tipping, shop conveyor systems or pneumatic tube conveyors: 1/0492
G02B Optical elements, systems or apparatus: 2027/0138, 2027/0141, 27/017
G03H Holographic processes or apparatus: 1/0005, 1/0808, 1/2249, 2001/0088, 2001/2252, 2226/02, 2226/04, 2226/05
G05B Control or regulating systems in general; functional elements of such systems; monitoring or testing arrangements for such systems or elements: 2219/40126
G05D Systems for controlling or regulating non-electric variables: 2201/0216
G06T Image data processing or generation, in general: 19/006
G09G Arrangements or circuits for control of indicating devices using static means to present variable information: 3/002, 3/003
Y10S Technical subjects covered by former uspc cross-reference art collections [xracs] and digests: 901/01

(73) Assignee

Microsoft Technology Licensing LLC

(72) Inventors

Ethan Eade; Jeroen Vanturennout; Jonathan LYONS; David Fields; Gavin Dean Lazarow; Tushar Cyril BHATNAGAR

(54) Title

Sharing neighboring map data across devices

(57) Abstract

A computing device and method are provided for transmitting a relevant subset of map data, called a neighborhood, to enable mutual spatial understanding by multiple display devices around a target virtual location to display a shared hologram in the same exact location in the physical environment at the same moment in time. The computing device may comprise a processor, a memory operatively coupled to the processor, and an anchor transfer program stored in the memory and executed by the processor.

Full text

View on Google Patents

Claims (18)

A server computing device, comprising: a processor; a non-volatile storage device operatively coupled to the processor; and an anchor transfer program stored in the non-volatile storage device and executed by the processor of the computing device, wherein the anchor transfer program is configured to: receive a transfer request from a first display device to transfer anchor data of a target virtual place-located anchor of one or more holograms at a target virtual location to a second display device, the transfer request initiated by a user via the first display device and specifying the target virtual place-located anchor, the anchor data comprising neighboring map data corresponding to the target virtual location for the specified target virtual place-located anchor, the neighboring map data being map data of a neighborhood in a vicinity around the target virtual location including pose graphs created by sensor measurements performed by the first display device; retrieve and transmit the anchor data to the second display device responsive to the transfer request; cause the second display device to incorporate, via a stitching process, the neighboring map data into existing map data of the second display device, the stitching process including stitching together the pose graphs of the neighboring map data created by the first display device and the existing map data of the second display device by applying a common global coordinate system corresponding to an aligned three-dimensional coordinate space of the neighborhood; and cause the second display device to display the one or more holograms at the target virtual place-located anchor at the target virtual location from a vantage point of the second display device based on the incorporated map data and the existing map data of the second display device.
The server computing device of claim 1, further comprising: an anchor program stored in the non-volatile storage device and executed by the processor of the computing device, wherein the anchor program is configured to: receive an instruction from the first display device to generate the target virtual place-located anchor at the target virtual location; generate the target virtual place-located anchor at the target virtual location; and send the target virtual place-located anchor to the first display device.
The server computing device of claim 1, wherein the anchor transfer program is configured to transmit to the first display device neighboring map data in a serialized format; and the second display device receives the neighboring map data in a deserialized format.
The server computing device of claim 1, wherein the target virtual location is world-locked to a position that is fixed in a three-dimensional coordinate space overlaid upon a real world three-dimensional environment.
The server computing device of claim 1, wherein the target virtual location is world-locked to a position relative to an object in a real world three-dimensional environment.
The server computing device of claim 1, wherein the neighboring map data comprises keyframes and at least a portion of pose-graphs describing rotational and translational motion of the display devices through a real world three-dimensional environment.
The server computing device of claim 6, further comprising: visual sensors and/or inertial measurement sensors, wherein the visual sensors and/or inertial measurement sensors track the rotational and translational motion of the display devices for the keyframes and the at least a portion of pose-graphs.
The server computing device of claim 6, wherein the keyframes comprise at least one of a fingerprint of a Wi-Fi beacon, gravity data, temperature data, global positioning data, and calibration data.
A method, comprising: receiving a transfer request from a first display device to transfer anchor data of a target virtual place-located anchor of one or more holograms at a target virtual location to a second display device, the transfer request initiated by a user via the first display device and specifying the target virtual place-located anchor, the anchor data comprising neighboring map data corresponding to the target virtual location for the specified target virtual place-located anchor, the neighboring map data being map data of a neighborhood in a vicinity around the target virtual location including pose graphs created by sensor measurements performed by the first display device; retrieving and transmitting the anchor data to the second display device responsive to the transfer request; and subsequent to the first display device transferring anchor data to the second display device, causing the second display device to incorporate, via a stitching process, the neighboring map data into existing map data of the second display device, and causing the second display device to display the one or more holograms at the target virtual place-located anchor at the target virtual location from a vantage point of the second display device based on the incorporated map data and the existing map data of the second display device, the stitching process including stitching together the pose graphs of the neighboring map data created by the first display device and the existing map data of the second display device by applying a common global coordinate system corresponding to an aligned three-dimensional coordinate space of the neighborhood.
The method of claim 9, further comprising: receiving an instruction from the first display device to generate the target virtual place-located anchor at the target virtual location; generating the target virtual place-located anchor at the target virtual location; and sending the target virtual place-located anchor to the first display device.
The method of claim 9, wherein the neighboring map data is transmitted to the first display device in a serialized format, and the second display device receives the neighboring map data in a deserialized format.
The method of claim 9, wherein the target virtual location is world-locked to a position that is fixed in a three-dimensional coordinate space overlaid upon a real world three-dimensional environment.
The method of claim 9, wherein the target virtual location is world-locked to a position relative to an object in a real world three-dimensional environment.
The method of claim 9, wherein the neighboring map data comprises keyframes and pose-graphs recording rotational and translational motion of the display devices through a real world three-dimensional environment.
The method of claim 14, wherein visual sensors and/or inertial measurement sensors track the rotational and translational motion of the display devices for the keyframes and pose-graphs.
The method of claim 14, wherein the keyframes comprise at least one of a fingerprint of a Wi-Fi beacon, gravity data, temperature data, global positioning data, and calibration data.
A server computing device, comprising: a processor; a non-volatile storage device operatively coupled to the processor; and an anchor transfer program stored in the non-volatile storage device and executed by the processor of the computing device, wherein the anchor transfer program is configured to: receive a transfer request from a first display device to transfer anchor data of a target virtual place-located anchor of one or more holograms at a target virtual location to a second display device, the transfer request initiated by a user via the first display device and specifying the target virtual place-located anchor, the anchor data comprising neighboring map data corresponding to the target virtual location for the specified target virtual place-located anchor, the neighboring map data being map data of a neighborhood in a vicinity around the target virtual location including pose graphs created by sensor measurements performed by the first display device; retrieve and transmit the anchor data to the second display device responsive to the transfer request; cause the second display device to incorporate, via a stitching process, the neighboring map data into existing map data of the second display device, the stitching process including stitching together the pose graphs of the neighboring map data created by the first display device and the existing map data of the second display device by applying a common global coordinate system corresponding to an aligned three-dimensional coordinate space of the neighborhood; and cause the second display device to align movements in aligned coordinate space with the first display device based on the incorporated map data and the existing map data of the second display device, wherein the first and second display devices are configured to be autonomous robots.
The server computing device of claim 17, wherein the anchor transfer program is further configured to: cause the second display device to render and interact with one or more virtual objects and/or virtual spaces at the target virtual place-located anchor at the target virtual location from the vantage point of the second display device based on the incorporated map data and the existing map data of the second display device.

Description

6-DoF tracking, also known as six degrees of freedom tracking, is a method by which a device (e.g. mixed-reality head-mounted device (HMD), robot, smartphone, etc.) uses sensors (e.g. cameras, inertial measurement units, etc.) to determine its position relative to its surrounding physical environment. For example, a mixed-reality HMD or smartphone can use this positional understanding to place holograms or digital content so as to appear to be world-locked to a position in the physical world, and a robot can use this positional understanding to navigate itself relative to its surroundings. Recently, scenarios have arisen in which it is useful to have two or more such devices operating with a common understanding of their positions relative to a physical environment, and thus relative to each other. As discussed in detail below, there are several general approaches to developing this common understanding of positions between such devices, each with significant challenges recognized by the inventors.

To address these issues, a computing device and method are provided for transmitting a relevant subset of map data, called a neighborhood, to enable mutual spatial understanding by multiple display devices around a target virtual location to display a shared hologram in the same exact location in the physical environment at the same moment in time. The computing device may comprise a processor, a memory operatively coupled to the processor, and an anchor transfer program stored in the memory and executed by the processor.

Citations (22)

US20050052714A1
US7557971B1
US8115768B2
US9020187B2
US9286711B2
WO2013083650A1
US20130326364A1
US9460198B1
US20140049559A1
US20140267234A1
US20140354685A1
US20150070274A1
US20160026253A1
US20160025981A1
US20160133230A1
US20160180797A1
US20160210784A1
US20160371884A1
US20160379409A1
US20170248963A1
US10203209B2
US20180188042A1

Record as JSON

{
  "publication_number": "US10466953B2",
  "country": "US",
  "kind": "B2",
  "title": "Sharing neighboring map data across devices",
  "abstract": "A computing device and method are provided for transmitting a relevant subset of map data, called a neighborhood, to enable mutual spatial understanding by multiple display devices around a target virtual location to display a shared hologram in the same exact location in the physical environment at the same moment in time. The computing device may comprise a processor, a memory operatively coupled to the processor, and an anchor transfer program stored in the memory and executed by the processor.",
  "claims": [
    "1. A server computing device, comprising: a processor; a non-volatile storage device operatively coupled to the processor; and an anchor transfer program stored in the non-volatile storage device and executed by the processor of the computing device, wherein the anchor transfer program is configured to: receive a transfer request from a first display device to transfer anchor data of a target virtual place-located anchor of one or more holograms at a target virtual location to a second display device, the transfer request initiated by a user via the first display device and specifying the target virtual place-located anchor, the anchor data comprising neighboring map data corresponding to the target virtual location for the specified target virtual place-located anchor, the neighboring map data being map data of a neighborhood in a vicinity around the target virtual location including pose graphs created by sensor measurements performed by the first display device; retrieve and transmit the anchor data to the second display device responsive to the transfer request; cause the second display device to incorporate, via a stitching process, the neighboring map data into existing map data of the second display device, the stitching process including stitching together the pose graphs of the neighboring map data created by the first display device and the existing map data of the second display device by applying a common global coordinate system corresponding to an aligned three-dimensional coordinate space of the neighborhood; and cause the second display device to display the one or more holograms at the target virtual place-located anchor at the target virtual location from a vantage point of the second display device based on the incorporated map data and the existing map data of the second display device.",
    "2. The server computing device of claim 1, further comprising: an anchor program stored in the non-volatile storage device and executed by the processor of the computing device, wherein the anchor program is configured to: receive an instruction from the first display device to generate the target virtual place-located anchor at the target virtual location; generate the target virtual place-located anchor at the target virtual location; and send the target virtual place-located anchor to the first display device.",
    "3. The server computing device of claim 1, wherein the anchor transfer program is configured to transmit to the first display device neighboring map data in a serialized format; and the second display device receives the neighboring map data in a deserialized format.",
    "4. The server computing device of claim 1, wherein the target virtual location is world-locked to a position that is fixed in a three-dimensional coordinate space overlaid upon a real world three-dimensional environment.",
    "5. The server computing device of claim 1, wherein the target virtual location is world-locked to a position relative to an object in a real world three-dimensional environment.",
    "6. The server computing device of claim 1, wherein the neighboring map data comprises keyframes and at least a portion of pose-graphs describing rotational and translational motion of the display devices through a real world three-dimensional environment.",
    "7. The server computing device of claim 6, further comprising: visual sensors and/or inertial measurement sensors, wherein the visual sensors and/or inertial measurement sensors track the rotational and translational motion of the display devices for the keyframes and the at least a portion of pose-graphs.",
    "8. The server computing device of claim 6, wherein the keyframes comprise at least one of a fingerprint of a Wi-Fi beacon, gravity data, temperature data, global positioning data, and calibration data.",
    "9. A method, comprising: receiving a transfer request from a first display device to transfer anchor data of a target virtual place-located anchor of one or more holograms at a target virtual location to a second display device, the transfer request initiated by a user via the first display device and specifying the target virtual place-located anchor, the anchor data comprising neighboring map data corresponding to the target virtual location for the specified target virtual place-located anchor, the neighboring map data being map data of a neighborhood in a vicinity around the target virtual location including pose graphs created by sensor measurements performed by the first display device; retrieving and transmitting the anchor data to the second display device responsive to the transfer request; and subsequent to the first display device transferring anchor data to the second display device, causing the second display device to incorporate, via a stitching process, the neighboring map data into existing map data of the second display device, and causing the second display device to display the one or more holograms at the target virtual place-located anchor at the target virtual location from a vantage point of the second display device based on the incorporated map data and the existing map data of the second display device, the stitching process including stitching together the pose graphs of the neighboring map data created by the first display device and the existing map data of the second display device by applying a common global coordinate system corresponding to an aligned three-dimensional coordinate space of the neighborhood.",
    "10. The method of claim 9, further comprising: receiving an instruction from the first display device to generate the target virtual place-located anchor at the target virtual location; generating the target virtual place-located anchor at the target virtual location; and sending the target virtual place-located anchor to the first display device.",
    "11. The method of claim 9, wherein the neighboring map data is transmitted to the first display device in a serialized format, and the second display device receives the neighboring map data in a deserialized format.",
    "12. The method of claim 9, wherein the target virtual location is world-locked to a position that is fixed in a three-dimensional coordinate space overlaid upon a real world three-dimensional environment.",
    "13. The method of claim 9, wherein the target virtual location is world-locked to a position relative to an object in a real world three-dimensional environment.",
    "14. The method of claim 9, wherein the neighboring map data comprises keyframes and pose-graphs recording rotational and translational motion of the display devices through a real world three-dimensional environment.",
    "15. The method of claim 14, wherein visual sensors and/or inertial measurement sensors track the rotational and translational motion of the display devices for the keyframes and pose-graphs.",
    "16. The method of claim 14, wherein the keyframes comprise at least one of a fingerprint of a Wi-Fi beacon, gravity data, temperature data, global positioning data, and calibration data.",
    "17. A server computing device, comprising: a processor; a non-volatile storage device operatively coupled to the processor; and an anchor transfer program stored in the non-volatile storage device and executed by the processor of the computing device, wherein the anchor transfer program is configured to: receive a transfer request from a first display device to transfer anchor data of a target virtual place-located anchor of one or more holograms at a target virtual location to a second display device, the transfer request initiated by a user via the first display device and specifying the target virtual place-located anchor, the anchor data comprising neighboring map data corresponding to the target virtual location for the specified target virtual place-located anchor, the neighboring map data being map data of a neighborhood in a vicinity around the target virtual location including pose graphs created by sensor measurements performed by the first display device; retrieve and transmit the anchor data to the second display device responsive to the transfer request; cause the second display device to incorporate, via a stitching process, the neighboring map data into existing map data of the second display device, the stitching process including stitching together the pose graphs of the neighboring map data created by the first display device and the existing map data of the second display device by applying a common global coordinate system corresponding to an aligned three-dimensional coordinate space of the neighborhood; and cause the second display device to align movements in aligned coordinate space with the first display device based on the incorporated map data and the existing map data of the second display device, wherein the first and second display devices are configured to be autonomous robots.",
    "18. The server computing device of claim 17, wherein the anchor transfer program is further configured to: cause the second display device to render and interact with one or more virtual objects and/or virtual spaces at the target virtual place-located anchor at the target virtual location from the vantage point of the second display device based on the incorporated map data and the existing map data of the second display device."
  ],
  "description_excerpt": "6-DoF tracking, also known as six degrees of freedom tracking, is a method by which a device (e.g. mixed-reality head-mounted device (HMD), robot, smartphone, etc.) uses sensors (e.g. cameras, inertial measurement units, etc.) to determine its position relative to its surrounding physical environment. For example, a mixed-reality HMD or smartphone can use this positional understanding to place holograms or digital content so as to appear to be world-locked to a position in the physical world, and a robot can use this positional understanding to navigate itself relative to its surroundings. Recently, scenarios have arisen in which it is useful to have two or more such devices operating with a common understanding of their positions relative to a physical environment, and thus relative to each other. As discussed in detail below, there are several general approaches to developing this common understanding of positions between such devices, each with significant challenges recognized by the inventors.\n\nTo address these issues, a computing device and method are provided for transmitting a relevant subset of map data, called a neighborhood, to enable mutual spatial understanding by multiple display devices around a target virtual location to display a shared hologram in the same exact location in the physical environment at the same moment in time. The computing device may comprise a processor, a memory operatively coupled to the processor, and an anchor transfer program stored in the memory and executed by the processor.",
  "cpc": [
    "G06F 3/1454",
    "B65G 1/0492",
    "G02B 2027/0138",
    "G02B 2027/0141",
    "G02B 27/017",
    "G03H 1/0005",
    "G03H 1/0808",
    "G03H 1/2249",
    "G03H 2001/0088",
    "G03H 2001/2252",
    "G03H 2226/02",
    "G03H 2226/04",
    "G03H 2226/05",
    "G05B 2219/40126",
    "G05D 2201/0216",
    "G06F 3/011",
    "G06F 3/012",
    "G06F 3/04815",
    "G06F 3/1423",
    "G06F 3/147",
    "G06T 19/006",
    "G09G 3/002",
    "G09G 3/003",
    "Y10S 901/01"
  ],
  "ipc": [
    "B65G 1/04",
    "G02B 27/01",
    "G03H 1/00",
    "G03H 1/22",
    "G06F 3/01",
    "G06F 3/0481",
    "G06F 3/14",
    "G06F 3/147",
    "G06T 19/00",
    "G09G 3/00"
  ],
  "assignees": [
    "Microsoft Technology Licensing LLC"
  ],
  "inventors": [
    "Ethan Eade",
    "Jeroen Vanturennout",
    "Jonathan LYONS",
    "David Fields",
    "Gavin Dean Lazarow",
    "Tushar Cyril BHATNAGAR"
  ],
  "filing_date": "2017-05-11",
  "publication_date": "2019-11-05",
  "grant_date": "2019-11-05",
  "priority_date": "2017-03-30",
  "application_number": "US-201715593147-A",
  "family_id": "63670655",
  "cited_by_count": 14,
  "citations": [
    "US20050052714A1",
    "US7557971B1",
    "US8115768B2",
    "US9020187B2",
    "US9286711B2",
    "WO2013083650A1",
    "US20130326364A1",
    "US9460198B1",
    "US20140049559A1",
    "US20140267234A1",
    "US20140354685A1",
    "US20150070274A1",
    "US20160026253A1",
    "US20160025981A1",
    "US20160133230A1",
    "US20160180797A1",
    "US20160210784A1",
    "US20160371884A1",
    "US20160379409A1",
    "US20170248963A1",
    "US10203209B2",
    "US20180188042A1"
  ]
}

Record 2,463 of 8,000 in Patents full text (MLC-0201). Request the full dataset.