Patent · US11145043B2 · B2 · US

Using unmanned aerial vehicles to inspect autonomous vehicles

(11) Publication number

US11145043B2

(21) Application number

16/342,761

(22) Filing date

2016-10-24

(30) Priority date

2016-10-24

(43) Publication date

2021-10-12

(45) Date of grant

2021-10-12

(51) IPC

B64F 1/22; B64U 50/38; B64U 80/25; G05B 13/02; G05D 1/00; G06N 3/04; G06Q 10/00; G06T 7/00; G07B 15/00; G07C 5/08; H04N 23/90; H04N 7/18

(52) CPC

G06T Image data processing or generation, in general: 7/0002, 2207/10032, 2207/20084, 2207/30252, 2207/30268
B64C Aeroplanes; helicopters: 2201/123, 39/024, 39/028
B64F Ground or aircraft-carrier-deck installations specially adapted for use in connection with aircraft; designing, manufacturing, assembling, cleaning, maintaining or repairing aircraft, not otherwise provided for; handling, transporting, testing or inspecting aircraft components, not otherwise provided for: 1/222
B64U Unmanned aerial vehicles [uav]; equipment therefor: 2101/26, 2101/30, 2101/70, 2201/10, 50/38, 80/25
G05B Control or regulating systems in general; functional elements of such systems; monitoring or testing arrangements for such systems or elements: 13/027
G05D Systems for controlling or regulating non-electric variables: 1/0094
G06N Computing arrangements based on specific computational models: 3/04, 3/0464
G06Q Information and communication technology [ICT] specially adapted for administrative, commercial, financial, managerial or supervisory purposes; systems or methods specially adapted for administrative, commercial, financial, managerial or supervisory purposes, not otherwise provided for: 10/083, 10/20
G07B Ticket-issuing apparatus; fare-registering apparatus; franking apparatus: 15/00
G07C Time or attendance registers; registering or indicating the working of machines; generating random numbers; voting or lottery apparatus; arrangements, systems or apparatus for checking not provided for elsewhere: 2205/02, 5/08, 5/0808
H04N Pictorial communication, e.g. television: 23/661, 23/90, 5/23206, 5/247, 7/18, 7/185

(73) Assignee

Ford Motor Co

(72) Inventors

Scott Vincent Myers; Mark Crawford; Harpreetsingh Banvait; Alexandru Mihai Gurghian; Nikhil Nagraj Rao; Alexandro Walsh; Lisa Scaria

(54) Title

Using unmanned aerial vehicles to inspect autonomous vehicles

(57) Abstract

The present invention extends to methods, systems, and computer program products for using Unmanned Aerial Vehicles (UAVs) to inspect autonomous vehicles. An autonomous vehicle carries a UAV (or “drone”) in a protected area, for example, in a glove compartment, trunk, etc. Between rides, the UAV can be deployed to inspect the autonomous vehicle. Images from the UAV can be sent to other components for image analysis. When an inspection is completed, the UAV can return to the protected area. The UAV can inspect both the interior and exterior of an autonomous vehicle. When an inspection is passed, the autonomous vehicle can begin a new ride. When an inspection is failed, the autonomous vehicle can report for repairs or summon a tow vehicle.

Full text

View on Google Patents

Claims (19)

A method, the method comprising: detecting termination of a fare-based ride given to one or more passengers; determining that an autonomous vehicle is to be inspected in accordance with inspection rules; releasing an Unmanned Aerial Vehicle (UAV) from a protected area within the autonomous vehicle to inspect the autonomous vehicle; receiving images of one or more of: interior vehicle components and exterior vehicle components of the autonomous vehicle from the Unmanned Aerial Vehicle (UAV) inspecting the autonomous vehicle; forwarding the images for image analysis; and receiving instructions instructing the autonomous vehicle how to proceed based on the image analysis of the images.
The method of claim 1, further comprising capturing the Unmanned Aerial Vehicle (UAV) in the protected area after the inspection is completed.
The method of claim 1, wherein receiving images from an Unmanned Aerial Vehicle (UAV) comprises receiving images from the Unmanned Aerial Vehicle (UAV) as the Unmanned Aerial Vehicle (UAV) flies around the autonomous vehicle.
The method of claim 1, wherein receiving images from an Unmanned Aerial Vehicle (UAV) comprises receiving images captured by the Unmanned Aerial Vehicle (UAV) as the autonomous vehicle drives by the Unmanned Aerial Vehicle (UAV).
The method of claim 1, wherein receiving images from an Unmanned Aerial Vehicle (UAV) comprises receiving images captured by the Unmanned Aerial Vehicle (UAV) as the autonomous vehicle drives over the Unmanned Aerial Vehicle (UAV).
An autonomous vehicle, the autonomous vehicle comprising: one or more processors; system memory coupled to one or more processors, the system memory storing instructions that are executable by the one or more processors; an Unmanned Aerial Vehicle (UAV) stored in a protected area within the autonomous vehicle; and the one or more processors configured to execute the instructions stored in the system memory to use the Unmanned Aerial Vehicle (UAV) to inspect the autonomous vehicle, including the following: detect termination of a fare-based ride given to one or more passengers; determine that the autonomous vehicle is to be inspected in accordance with inspection rules; release the Unmanned Aerial Vehicle (UAV) from the protected area to inspect the autonomous vehicle; receive images of one or more of: interior vehicle components and exterior vehicle components of the autonomous vehicle from the Unmanned Aerial Vehicle (UAV) as the Unmanned Aerial Vehicle (UAV) inspects the autonomous vehicle; forward the images to other computing resources for image analysis; receive instructions from the other computing resources instructing the autonomous vehicle how to proceed based on image analysis of the images; and capture the Unmanned Aerial Vehicle (UAV) in the protected area.
The autonomous vehicle of claim 6, further comprising a charging station in the protected area for wirelessly charging the Unmanned Aerial Vehicle (UAV), and wherein the one or more processors configured to execute the instructions stored in the system memory to capture the Unmanned Aerial Vehicle (UAV) in the protected area comprises the one or more processors configured to execute the instructions stored in the system memory to anchor the Unmanned Aerial Vehicle (UAV) to the charging station.
The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to determine that the autonomous vehicle is to be inspected in accordance with inspection rules comprises the one or more processors configured to execute the instructions stored in the system memory to determine that the autonomous vehicle is to be inspected based on a service level associated with the autonomous vehicle.
The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to receive images from the Unmanned Aerial Vehicle (UAV) comprises the one or more processors configured to execute the instructions stored in the system memory to receive images indicative of the cleanliness of the interior of the autonomous vehicle.
The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to receive images from the Unmanned Aerial Vehicle (UAV) comprises the one or more processors configured to execute the instructions stored in the system memory to receive images of one or more exterior components of the autonomous vehicle, the one or more exterior components selected from among: tires, brake pads, turning signals, and head lights.
The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to forward the images to other computing resources for image analysis comprises the one or more processors configured to execute the instructions stored in the system memory to forward the images to a neural network.
The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to receive instructions from the other computing resources comprises the one or more processors configured to execute the instructions stored in the system memory to receive instructions from a neural network.
The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to receive instructions from the other computing resources comprises the one or more processors configured to execute the instructions stored in the system memory to receive instructions instructing the autonomous vehicle to return to a repair facility.
The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to receive instructions from the other computing resources comprises the one or more processors configured to execute the instructions stored in the system memory to receive instructions instructing the autonomous vehicle to electronically request a tow vehicle.
The autonomous vehicle of claim 6, further comprising the one or more processors configured to execute the instructions stored in the system memory to instruct the Unmanned Aerial Vehicle (UAV) to perform a remedial maintenance operation on the autonomous vehicle.
A method for use at an autonomous vehicle, the method for using an Unmanned Aerial Vehicle (UAV) to inspect an autonomous vehicle, the method comprising: detecting termination of a fare-based ride given to one or more passengers; determining that the autonomous vehicle is to be inspected in accordance with inspection rules; releasing the Unmanned Aerial Vehicle (UAV) from a protected area within the autonomous vehicle to inspect the autonomous vehicle; receiving images of one or more of: interior vehicle components and exterior vehicle components of the autonomous vehicle from the Unmanned Aerial Vehicle (UAV) as the Unmanned Aerial Vehicle (UAV) inspects the autonomous vehicle; forwarding the images to other computing resources for image analysis; receiving instructions from the other computing resources instructing the autonomous vehicle how to proceed based on image analysis of the images; and capturing the Unmanned Aerial Vehicle (UAV) in the protected area.
The method of claim 16, wherein receiving images from an Unmanned Aerial Vehicle (UAV) comprises receiving images from the Unmanned Aerial Vehicle (UAV) as the Unmanned Aerial Vehicle (UAV) flies around the autonomous vehicle.
The method of claim 16, wherein receiving images from an Unmanned Aerial Vehicle (UAV) comprises receiving images captured by the Unmanned Aerial Vehicle (UAV) as the autonomous vehicle drives by the Unmanned Aerial Vehicle (UAV).
The method of claim 16 wherein receiving images from an Unmanned Aerial Vehicle (UAV) comprises receiving images captured by the Unmanned Aerial Vehicle (UAV) as the autonomous vehicle drives over the Unmanned Aerial Vehicle (UAV).

Description

Not applicable.

This invention relates generally to the field of vehicle inspection, and, more particularly, to using Unmanned Aerial Vehicles (UAVs) to inspect autonomous vehicles.

Autonomous vehicles have no driver. Thus, when an autonomous vehicle is used to provide fare-based rides for 3 rd parties, it can be difficult to inspect the autonomous vehicle for cleanliness, maintenance issues, etc., between rides. Cameras can be mounted on an autonomous vehicle to inspect some areas of the autonomous vehicle between rides. However, over time, camera lenses can be damaged or covered with dirt limiting their usefulness. Dirt and grime is a significant concern for externally mounted cameras, especially in locations that are prone inclement weather. For example, it can be difficult to keep undercarriage cameras clean from grime and snow in locations that experience winter weather.

The specific features, aspects and advantages of the present invention will become better understood with regard to the following description and accompanying drawings where:

FIG. 1 illustrates an example block diagram of a computing device.

FIG. 2 illustrates an environment that facilitates using an Unmanned Aerial Vehicle (UAV) to inspect an autonomous vehicle.

FIG. 3 illustrates a flow chart of an example method for using an Unmanned Aerial Vehicle (UAV) to inspect an autonomous vehicle.

FIGS. 4A and 4B illustrate examples of an Unmanned Aerial Vehicle (UAV) being released from a protected area within an autonomous vehicle.

Citations (12)

US20150210388A1
US20150302669A1
US20160016663A1
US9129355B1
US20180170540A1
US20170080900A1
US20170140603A1
US20170355459A1
US20180040171A1
US20180039838A1
US20180082379A1
US20180107210A1

Record as JSON

{
  "publication_number": "US11145043B2",
  "country": "US",
  "kind": "B2",
  "title": "Using unmanned aerial vehicles to inspect autonomous vehicles",
  "abstract": "The present invention extends to methods, systems, and computer program products for using Unmanned Aerial Vehicles (UAVs) to inspect autonomous vehicles. An autonomous vehicle carries a UAV (or “drone”) in a protected area, for example, in a glove compartment, trunk, etc. Between rides, the UAV can be deployed to inspect the autonomous vehicle. Images from the UAV can be sent to other components for image analysis. When an inspection is completed, the UAV can return to the protected area. The UAV can inspect both the interior and exterior of an autonomous vehicle. When an inspection is passed, the autonomous vehicle can begin a new ride. When an inspection is failed, the autonomous vehicle can report for repairs or summon a tow vehicle.",
  "claims": [
    "1. A method, the method comprising: detecting termination of a fare-based ride given to one or more passengers; determining that an autonomous vehicle is to be inspected in accordance with inspection rules; releasing an Unmanned Aerial Vehicle (UAV) from a protected area within the autonomous vehicle to inspect the autonomous vehicle; receiving images of one or more of: interior vehicle components and exterior vehicle components of the autonomous vehicle from the Unmanned Aerial Vehicle (UAV) inspecting the autonomous vehicle; forwarding the images for image analysis; and receiving instructions instructing the autonomous vehicle how to proceed based on the image analysis of the images.",
    "2. The method of claim 1, further comprising capturing the Unmanned Aerial Vehicle (UAV) in the protected area after the inspection is completed.",
    "3. The method of claim 1, wherein receiving images from an Unmanned Aerial Vehicle (UAV) comprises receiving images from the Unmanned Aerial Vehicle (UAV) as the Unmanned Aerial Vehicle (UAV) flies around the autonomous vehicle.",
    "4. The method of claim 1, wherein receiving images from an Unmanned Aerial Vehicle (UAV) comprises receiving images captured by the Unmanned Aerial Vehicle (UAV) as the autonomous vehicle drives by the Unmanned Aerial Vehicle (UAV).",
    "5. The method of claim 1, wherein receiving images from an Unmanned Aerial Vehicle (UAV) comprises receiving images captured by the Unmanned Aerial Vehicle (UAV) as the autonomous vehicle drives over the Unmanned Aerial Vehicle (UAV).",
    "6. An autonomous vehicle, the autonomous vehicle comprising: one or more processors; system memory coupled to one or more processors, the system memory storing instructions that are executable by the one or more processors; an Unmanned Aerial Vehicle (UAV) stored in a protected area within the autonomous vehicle; and the one or more processors configured to execute the instructions stored in the system memory to use the Unmanned Aerial Vehicle (UAV) to inspect the autonomous vehicle, including the following: detect termination of a fare-based ride given to one or more passengers; determine that the autonomous vehicle is to be inspected in accordance with inspection rules; release the Unmanned Aerial Vehicle (UAV) from the protected area to inspect the autonomous vehicle; receive images of one or more of: interior vehicle components and exterior vehicle components of the autonomous vehicle from the Unmanned Aerial Vehicle (UAV) as the Unmanned Aerial Vehicle (UAV) inspects the autonomous vehicle; forward the images to other computing resources for image analysis; receive instructions from the other computing resources instructing the autonomous vehicle how to proceed based on image analysis of the images; and capture the Unmanned Aerial Vehicle (UAV) in the protected area.",
    "7. The autonomous vehicle of claim 6, further comprising a charging station in the protected area for wirelessly charging the Unmanned Aerial Vehicle (UAV), and wherein the one or more processors configured to execute the instructions stored in the system memory to capture the Unmanned Aerial Vehicle (UAV) in the protected area comprises the one or more processors configured to execute the instructions stored in the system memory to anchor the Unmanned Aerial Vehicle (UAV) to the charging station.",
    "8. The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to determine that the autonomous vehicle is to be inspected in accordance with inspection rules comprises the one or more processors configured to execute the instructions stored in the system memory to determine that the autonomous vehicle is to be inspected based on a service level associated with the autonomous vehicle.",
    "9. The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to receive images from the Unmanned Aerial Vehicle (UAV) comprises the one or more processors configured to execute the instructions stored in the system memory to receive images indicative of the cleanliness of the interior of the autonomous vehicle.",
    "10. The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to receive images from the Unmanned Aerial Vehicle (UAV) comprises the one or more processors configured to execute the instructions stored in the system memory to receive images of one or more exterior components of the autonomous vehicle, the one or more exterior components selected from among: tires, brake pads, turning signals, and head lights.",
    "11. The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to forward the images to other computing resources for image analysis comprises the one or more processors configured to execute the instructions stored in the system memory to forward the images to a neural network.",
    "12. The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to receive instructions from the other computing resources comprises the one or more processors configured to execute the instructions stored in the system memory to receive instructions from a neural network.",
    "13. The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to receive instructions from the other computing resources comprises the one or more processors configured to execute the instructions stored in the system memory to receive instructions instructing the autonomous vehicle to return to a repair facility.",
    "14. The autonomous vehicle of claim 6, wherein the one or more processors configured to execute the instructions stored in the system memory to receive instructions from the other computing resources comprises the one or more processors configured to execute the instructions stored in the system memory to receive instructions instructing the autonomous vehicle to electronically request a tow vehicle.",
    "15. The autonomous vehicle of claim 6, further comprising the one or more processors configured to execute the instructions stored in the system memory to instruct the Unmanned Aerial Vehicle (UAV) to perform a remedial maintenance operation on the autonomous vehicle.",
    "16. A method for use at an autonomous vehicle, the method for using an Unmanned Aerial Vehicle (UAV) to inspect an autonomous vehicle, the method comprising: detecting termination of a fare-based ride given to one or more passengers; determining that the autonomous vehicle is to be inspected in accordance with inspection rules; releasing the Unmanned Aerial Vehicle (UAV) from a protected area within the autonomous vehicle to inspect the autonomous vehicle; receiving images of one or more of: interior vehicle components and exterior vehicle components of the autonomous vehicle from the Unmanned Aerial Vehicle (UAV) as the Unmanned Aerial Vehicle (UAV) inspects the autonomous vehicle; forwarding the images to other computing resources for image analysis; receiving instructions from the other computing resources instructing the autonomous vehicle how to proceed based on image analysis of the images; and capturing the Unmanned Aerial Vehicle (UAV) in the protected area.",
    "17. The method of claim 16, wherein receiving images from an Unmanned Aerial Vehicle (UAV) comprises receiving images from the Unmanned Aerial Vehicle (UAV) as the Unmanned Aerial Vehicle (UAV) flies around the autonomous vehicle.",
    "18. The method of claim 16, wherein receiving images from an Unmanned Aerial Vehicle (UAV) comprises receiving images captured by the Unmanned Aerial Vehicle (UAV) as the autonomous vehicle drives by the Unmanned Aerial Vehicle (UAV).",
    "19. The method of claim 16 wherein receiving images from an Unmanned Aerial Vehicle (UAV) comprises receiving images captured by the Unmanned Aerial Vehicle (UAV) as the autonomous vehicle drives over the Unmanned Aerial Vehicle (UAV)."
  ],
  "description_excerpt": "Not applicable.\n\nThis invention relates generally to the field of vehicle inspection, and, more particularly, to using Unmanned Aerial Vehicles (UAVs) to inspect autonomous vehicles.\n\nAutonomous vehicles have no driver. Thus, when an autonomous vehicle is used to provide fare-based rides for 3 rd parties, it can be difficult to inspect the autonomous vehicle for cleanliness, maintenance issues, etc., between rides. Cameras can be mounted on an autonomous vehicle to inspect some areas of the autonomous vehicle between rides. However, over time, camera lenses can be damaged or covered with dirt limiting their usefulness. Dirt and grime is a significant concern for externally mounted cameras, especially in locations that are prone inclement weather. For example, it can be difficult to keep undercarriage cameras clean from grime and snow in locations that experience winter weather.\n\nThe specific features, aspects and advantages of the present invention will become better understood with regard to the following description and accompanying drawings where:\n\nFIG. 1 illustrates an example block diagram of a computing device.\n\nFIG. 2 illustrates an environment that facilitates using an Unmanned Aerial Vehicle (UAV) to inspect an autonomous vehicle.\n\nFIG. 3 illustrates a flow chart of an example method for using an Unmanned Aerial Vehicle (UAV) to inspect an autonomous vehicle.\n\nFIGS. 4A and 4B illustrate examples of an Unmanned Aerial Vehicle (UAV) being released from a protected area within an autonomous vehicle.",
  "cpc": [
    "G06T 7/0002",
    "B64C 2201/123",
    "B64C 39/024",
    "B64C 39/028",
    "B64F 1/222",
    "B64U 2101/26",
    "B64U 2101/30",
    "B64U 2101/70",
    "B64U 2201/10",
    "B64U 50/38",
    "B64U 80/25",
    "G05B 13/027",
    "G05D 1/0094",
    "G06N 3/04",
    "G06N 3/0464",
    "G06Q 10/083",
    "G06Q 10/20",
    "G06T 2207/10032",
    "G06T 2207/20084",
    "G06T 2207/30252",
    "G06T 2207/30268",
    "G07B 15/00",
    "G07C 2205/02",
    "G07C 5/08",
    "G07C 5/0808",
    "H04N 23/661",
    "H04N 23/90",
    "H04N 5/23206",
    "H04N 5/247",
    "H04N 7/18",
    "H04N 7/185"
  ],
  "ipc": [
    "B64F 1/22",
    "B64U 50/38",
    "B64U 80/25",
    "G05B 13/02",
    "G05D 1/00",
    "G06N 3/04",
    "G06Q 10/00",
    "G06T 7/00",
    "G07B 15/00",
    "G07C 5/08",
    "H04N 23/90",
    "H04N 7/18"
  ],
  "assignees": [
    "Ford Motor Co"
  ],
  "inventors": [
    "Scott Vincent Myers",
    "Mark Crawford",
    "Harpreetsingh Banvait",
    "Alexandru Mihai Gurghian",
    "Nikhil Nagraj Rao",
    "Alexandro Walsh",
    "Lisa Scaria"
  ],
  "filing_date": "2016-10-24",
  "publication_date": "2021-10-12",
  "grant_date": "2021-10-12",
  "priority_date": "2016-10-24",
  "application_number": "US-201616342761-A",
  "family_id": "62025326",
  "cited_by_count": 4,
  "citations": [
    "US20150210388A1",
    "US20150302669A1",
    "US20160016663A1",
    "US9129355B1",
    "US20180170540A1",
    "US20170080900A1",
    "US20170140603A1",
    "US20170355459A1",
    "US20180040171A1",
    "US20180039838A1",
    "US20180082379A1",
    "US20180107210A1"
  ]
}

Record 1,450 of 8,000 in Patents full text (MLC-0201). Request the full dataset.