Imaging-based tactile sensor with multi-lens array

{
  "publication_number": "US10038854B1",
  "country": "US",
  "kind": "B1",
  "title": "Imaging-based tactile sensor with multi-lens array",
  "abstract": "An example device for reducing the size and color cross-talk of an imaging-based tactile sensor includes an elastic material, one or more light sources, and an image capture device. The elastic material includes a reflective membrane. The reflective membrane conforms to a shape of an object pressed against the elastic material. Each light source of the one or more light sources is configured to illuminate at least a portion of the reflective membrane. The image capture device is configured to capture at least one image of the reflective membrane. The image capture device includes (i) an image sensor configured to generate the at least one image based on light incident on the image sensor and (ii) a plurality of lenses configured to direct light onto the image sensor. Each lens of the plurality of lenses is configured to direct light onto a corresponding portion of the image sensor.",
  "claims": [
    "1. A device comprising: an elastic material comprising a reflective membrane, wherein the reflective membrane conforms to a shape of an object pressed against the elastic material; two or more light sources situated at known positions proximate to the reflective membrane, wherein each light source of the one or more light sources is configured to illuminate at least a portion of the reflective membrane by emitting a corresponding color of light; an image capture device configured to capture at least one image of the reflective membrane, wherein the image capture device comprises: an image sensor configured to generate the at least one image based on light incident on the image sensor, wherein the image sensor comprises a plurality of photodetector groups separated by non-photoactive material, and wherein each photodetector group comprises a plurality of adjacent photodetectors arranged in a grid; a plurality of lenses configured to direct light onto the image sensor, wherein each lens of the plurality of lenses is configured to direct light onto a corresponding photodetector group of the plurality of photodetector groups, and wherein the plurality of lenses is arranged in a planar array situated directly between the elastic material and the image sensor such that the plurality of lenses is separated from the elastic material by free space; and a color filter array situated between the plurality of lenses and the image sensor, wherein the color filter array comprises a plurality of color filters corresponding to the colors of light emitted by the two or more light sources, wherein each respective color filter of the color filter array is configured to pass light of a given color onto a corresponding photodetector group of the plurality of photodetector groups such that each photodetector group has incident thereon light of a single color, and wherein the at least one image of the reflective membrane comprises two or more simultaneously-captured sub-images corresponding to the colors of light emitted by the two or more light sources, wherein a given sub-image of the two or more sub-images represents the reflective membrane illuminated by a single color of light; and processing hardware configured to generate a depth map of the reflective membrane based on the two or more sub-images corresponding to the colors of light emitted by the two or more light sources by way of one or more photometric stereo techniques.",
    "2. The device of claim 1, further comprising processing hardware configured to generate an estimate of forces acting on the reflective membrane based on an extent of deformation of the reflective membrane.",
    "3. The device of claim 1, wherein the plurality of photodetector groups are arranged in a planar array.",
    "4. The device of claim 1, wherein each lens of the plurality of lenses is configured to direct light onto one or more corresponding photodetectors of the plurality of adjacent photodetectors arranged in the grid.",
    "5. The device of claim 1, wherein each lens of the plurality of lenses is configured to direct light onto a corresponding color filter of the color filter array.",
    "6. The device of claim 1, wherein each light source of the two or more light sources is configured to emit light toward the reflective membrane from a different angle.",
    "7. The device of claim 1, wherein each light source of the two or more light sources is configured to emit a different color of light.",
    "8. The device of claim 1, wherein each lens of the plurality of lenses is configured to direct light reflected from a section of the reflective membrane onto the corresponding photodetector group, and wherein the two or more sub-images correspond to the sections of the reflective membrane.",
    "9. The device of claim 8, wherein a field-of-view of a given sub-image of the two or more sub-images overlaps with a field-of-view of a different sub-image of the two or more sub-images.",
    "10. The device of claim 1, wherein the elastic material further comprises a transparent volume situated between the reflective membrane and the image capture device.",
    "11. A tactile sensor comprising: an elastic material comprising a reflective membrane, wherein the reflective membrane conforms to a shape of an object pressed against the elastic material; a plurality of light sources situated at known positions proximate to the reflective membrane, wherein each light source of the plurality of light sources is configured to illuminate at least a portion of the reflective membrane by emitting a corresponding color of light; an image capture device configured to capture at least one image of the reflective membrane, wherein the image capture device comprises: an image sensor configured to generate a plurality of sub-images based on light incident on the image sensor, wherein the image sensor comprises a plurality of photodetector groups separated by non-photoactive material, and wherein each photodetector group comprises a plurality of adjacent photodetectors arranged in a grid; a plurality of lenses configured to direct light onto the image sensor, wherein each lens of the plurality of lenses is configured to direct light onto a corresponding photodetector group of the plurality of photodetector groups to generate a corresponding sub-image of the plurality of sub-images, and wherein the plurality of lenses is arranged in a planar array situated directly between the elastic material and the image sensor such that the plurality of lenses is separated from the elastic material by free space; and a color filter array situated between the plurality of lenses and the image sensor, wherein the color filter array comprises a plurality of color filters corresponding to the colors of light emitted by the plurality of light sources, wherein each respective color filter of the color filter array is configured to pass light of a given color onto a corresponding photodetector group of the plurality of photodetector groups such that each photodetector group has incident thereon light of a single color, and wherein the plurality of sub-images correspond to the colors of light emitted by the plurality of light sources, and wherein each sub-image of the plurality of sub-images represents the reflective membrane illuminated by a single color of light; and processing hardware configured to generate a depth map of the reflective membrane based on the plurality of sub-images by way of one or more photometric stereo techniques.",
    "12. A robotic system comprising: a robotic manipulator comprising an end effector; and a tactile sensor coupled to the end effector, wherein the tactile sensor comprises: an elastic material comprising a reflective membrane, wherein the reflective membrane conforms to a shape of an object pressed against the elastic material; two or more light sources situated at known positions proximate to the reflective membrane, wherein each light source of the one or more light sources is configured to illuminate at least a portion of the reflective membrane by emitting a corresponding color of light; and an image capture device configured to capture at least one image of the reflective membrane, wherein the image capture device comprises: an image sensor configured to generate the at least one image based on light incident on the image sensor, wherein the image sensor comprises a plurality of photodetector groups separated by non-photoactive material, and wherein each photodetector group comprises a plurality of adjacent photodetectors arranged in a grid; a plurality of lenses configured to direct light onto the image sensor, wherein each lens of the plurality of lenses is configured to direct light onto a corresponding photodetector group of the plurality of photodetector groups, and wherein the plurality of lenses is arranged in a planar array situated directly between the elastic material and the image sensor such that the plurality of lenses is separated from the elastic material by free space; and a color filter array situated between the plurality of lenses and the image sensor, wherein the color filter array comprises a plurality of color filters corresponding to the colors of light emitted by the two or more light sources, wherein each respective color filter of the color filter array is configured to pass light of a given color onto a corresponding photodetector group of the plurality of photodetector groups such that each photodetector group has incident thereon light of a single color, and wherein the at least one image of the reflective membrane comprises two or more simultaneously-captured sub-images corresponding to the colors of light emitted by the two or more light sources, wherein a given sub-image of the two or more sub-images represents the reflective membrane illuminated by a single color of light; and processing hardware configured to generate a depth map of the reflective membrane based on the two or more sub-images corresponding to the colors of light emitted by the two or more light sources by way of one or more photometric stereo techniques.",
    "13. The device of claim 1, wherein each lens of the plurality of lenses has a planar side and a convex side, and wherein the planar side faces the image sensor and the convex side faces the elastic material.",
    "14. The tactile sensor of claim 11, further comprising processing hardware configured to generate an estimate of forces acting on the reflective membrane based on an extent of deformation of the reflective membrane.",
    "15. The tactile sensor of claim 11, wherein the plurality of photodetector groups are arranged in a planar array.",
    "16. The tactile sensor of claim 11, wherein each lens of the plurality of lenses is configured to direct light onto one or more corresponding photodetectors of the plurality of adjacent photodetectors arranged in the grid.",
    "17. The tactile sensor of claim 11, wherein each light source of the plurality of light sources is configured to emit light toward the reflective membrane from a different angle.",
    "18. The tactile sensor of claim 11, wherein each light source of the plurality of light sources is configured to emit a different color of light.",
    "19. The tactile sensor of claim 11, wherein each lens of the plurality of lenses is configured to direct light reflected from a section of the reflective membrane onto the corresponding photodetector group, and wherein the plurality of sub-images correspond to the sections of the reflective membrane.",
    "20. The tactile sensor of claim 11, wherein a field-of-view of a given sub-image of the plurality of sub-images overlaps with a field-of-view of a different sub-image of the plurality of sub-images."
  ],
  "description_excerpt": "As technology advances, various types of robotic devices are being created for performing a variety of functions that may assist users. Robotic devices may be used for applications involving material handling, transportation, welding, assembly, and dispensing, among others. Over time, the manner in which these robotic systems operate is becoming more intelligent, efficient, and intuitive. As robotic systems become increasingly prevalent in numerous aspects of modern life, it is desirable for robotic systems to be efficient. Therefore, a demand for efficient robotic systems has helped open up a field of innovation in actuators, movement, sensing techniques, as well as component design and assembly.\n\nThe present application discloses implementations that relate to imaging-based tactile sensors with multi-lens arrays. In one example, the present application describes a device. The device includes an elastic material, one or more light sources, and an image capture device. The elastic material includes a reflective membrane. The reflective membrane conforms to a shape of an object pressed against the elastic material. The one or more light sources is situated proximate to the reflective membrane. Each light source of the one or more light sources is configured to illuminate at least a portion of the reflective membrane. The image capture device is configured to capture at least one image of the reflective membrane.",
  "cpc": [
    "H04N 23/74",
    "B25J 13/084",
    "G08B 6/00",
    "H04N 23/10",
    "H04N 23/55",
    "H04N 23/56",
    "H04N 5/2254",
    "H04N 5/2354",
    "H10F 39/12",
    "H10F 39/198",
    "H10F 39/805",
    "H10F 39/8053",
    "H10F 39/8063"
  ],
  "ipc": [
    "B25J 13/08",
    "G08B 6/00",
    "H04N 5/225",
    "H04N 5/235"
  ],
  "assignees": [
    "X Development LLC"
  ],
  "inventors": [
    "Emily Cooper",
    "Xiaoyu Miao"
  ],
  "filing_date": "2015-10-27",
  "publication_date": "2018-07-31",
  "grant_date": "2018-07-31",
  "priority_date": "2015-08-14",
  "application_number": "US-201514924687-A",
  "family_id": "62948639",
  "cited_by_count": 15,
  "citations": [
    "US7773143B2",
    "US20100149183A1",
    "US20090315989A1",
    "US8411140B2",
    "US8905610B2",
    "US8514491B2",
    "US20130041235A1",
    "US9127938B2",
    "US20130038689A1",
    "US20130041216A1",
    "US20150011894A1",
    "US20130181114A1",
    "US8956303B2",
    "US20140104395A1",
    "US20140267762A1",
    "US20150268450A1"
  ]
}