Vision-based tactile measurement method and apparatus, chip, and storage medium

{
  "publication_number": "US12214487B2",
  "country": "US",
  "kind": "B2",
  "title": "Vision-based tactile measurement method and apparatus, chip, and storage medium",
  "abstract": "A vision-based tactile measurement method is provided, performed by a computer device (e.g., a chip) connected to a tactile sensor, the tactile sensor including a sensing face and an image sensing component, and the sensing face being provided with a marking pattern. The method includes: obtaining an image sequence collected by the image sensing component of the sensing face, each image of the image sequence comprising one instance of the marking pattern; calculating a difference feature of the marking patterns in adjacent images of the image sequence; and processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result, a quantity of hidden layers in the feedforward neural network being less than a threshold.",
  "claims": [
    "1. A vision-based tactile measurement method performed by a computer device connected to a tactile sensor, the tactile sensor comprising a sensing face and an image sensing component, and the sensing face being provided with a marking pattern; and the method comprising: obtaining an image sequence collected by the image sensing component of the sensing face that is in physical contact with a surface of an object, each image of the image sequence comprising one instance of the marking pattern; calculating a difference feature of the marking patterns in adjacent images of the image sequence; and processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result, further comprising: calling a hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object, the feature presentation further including a curvature prediction on the surface of the object; and calling an output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result, the tactile measurement result comprising a local curvature radius of the surface of the object.",
    "2. The method according to claim 1, wherein n hidden neurons are provided in the hidden layer, n being an integer; the hidden layer is constructed based on hidden neurons of a logistic sigmoid function; and the output layer is constructed based on output neurons of a normalized exponential softmax function or linear output neurons.",
    "3. The method according to claim 1, wherein the feedforward neural network comprises: a location estimation model, and the location estimation model comprises a first hidden layer and a first output layer; the calling the hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object comprises: calling the first hidden layer in the location estimation model to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of a contact location on the surface of the object; and the calling the output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result comprises: calling the first output layer in the location estimation model to process the feature representation of the contact location, to obtain the contact location on the surface of the object.",
    "4. The method according to claim 1, wherein the feedforward neural network comprises: a contact force estimation model, and the contact force estimation model comprises a second hidden layer and a second output layer; the calling the hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object comprises: calling the second hidden layer in the contact force estimation model to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the contact force on the surface of the object; and the calling the output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result comprises: calling the second output layer in the contact force estimation model to process the feature representation of the contact force, to obtain three-dimensional information of the contact force on the surface of the object, the three-dimensional information comprising at least one of a magnitude and a direction.",
    "5. The method according to claim 1, wherein the feedforward neural network comprises: a surface classification model and at least two curvature estimation models; and the processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result comprises: calling the surface classification model to perform surface recognition on the difference feature of the marking patterns to obtain a surface type of the surface of the object; and calling a target curvature estimation model in the at least two curvature estimation models based on the surface type to perform the curvature prediction on the surface of the object, to obtain the local curvature radius of the surface of the object.",
    "6. The method according to claim 5, wherein the curvature estimation model comprises: a spherical surface estimation model and a cylindrical surface estimation model; and the calling a target curvature estimation model in the at least two curvature estimation models based on the surface type to perform the curvature prediction on the surface of the object, to obtain the local curvature radius of the surface of the object comprises: in a case that the surface type is a spherical surface, calling the spherical surface estimation model to perform first curvature prediction on the spherical surface, to obtain the local curvature radius of the spherical surface; and in a case that the surface type is a cylindrical surface, calling the cylindrical surface estimation model to perform second curvature prediction on the cylindrical surface, to obtain the local curvature radius of the cylindrical surface.",
    "7. The method according to claim 5, wherein the surface classification model comprises a third hidden layer and a third output layer; and the calling the surface classification model to perform surface recognition on the difference feature of the marking patterns to obtain a surface type of the surface of the object comprises: calling the third hidden layer in the surface classification model to perform surface recognition on the difference feature of the marking patterns to obtain a feature representation of the surface type; and calling the third output layer in the surface classification model to process the feature representation of the surface type, to obtain the surface type of the surface of the object.",
    "8. The method according to claim 6, wherein the spherical surface estimation model comprises a fourth hidden layer and a fourth output layer; and the calling the spherical surface estimation model to perform first curvature prediction on the spherical surface, to obtain the local curvature radius of the spherical surface comprises: calling the fourth hidden layer in the spherical surface estimation model to perform the first curvature prediction on the spherical surface, to obtain a feature representation of the curvature prediction of the spherical surface; and calling the fourth output layer in the spherical surface estimation model to process the feature representation of the curvature prediction of the spherical surface, to obtain the local curvature radius of the spherical surface.",
    "9. The method according to claim 6, wherein the cylindrical surface estimation model comprises a fifth hidden layer and a fifth output layer; and the calling the cylindrical surface estimation model to perform second curvature prediction on the cylindrical surface, to obtain the local curvature radius of the cylindrical surface comprises: calling the fifth hidden layer in the cylindrical surface estimation model to perform the second curvature prediction on the cylindrical surface, to obtain a feature representation of the curvature prediction of the cylindrical surface; and calling the fifth output layer in the cylindrical surface estimation model to process the feature representation of the curvature prediction of the cylindrical surface, to obtain the local curvature radius of the cylindrical surface.",
    "10. The method according to claim 1, wherein the marking pattern comprises at least two marking points, and the difference feature of the marking patterns comprises at least one of displacement and deformation of the marking points.",
    "11. The method according to claim 1, wherein the marking pattern comprises a grid, and the difference feature of the marking patterns comprises at least one of displacement of grid points and deformation of grid lines in the grid.",
    "12. The method according to claim 1, wherein the calculating a difference feature of the marking patterns based on the marking patterns in adjacent images of the image sequence comprises: determining two closest marking patterns from adjacent images at an i th frame and an (i+1) th frame in the image sequence as the same marking pattern, i being an integer; and calculating the difference feature of the marking patterns based on at least one of locations and deformation of the marking patterns in the images at the i th frame and the (i+1) th frame.",
    "13. A computer device in connection with a tactile sensor, the tactile sensor comprising a sensing face and an image sensing component, and the sensing face being provided with a marking pattern, the computer device further comprising a memory and a processor, wherein the memory stores a plurality of computer-readable instructions; and when the processor executes the plurality of computer-readable instructions, causes the computer device to perform a plurality of operations including: obtaining an image sequence collected by the image sensing component of the sensing face that is in physical contact with a surface of an object, each image of the image sequence comprising one instance of the marking pattern; calculating a difference feature of the marking patterns in adjacent images of the image sequence; and processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result, further comprising: calling a hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object, the feature presentation further including a curvature prediction on the surface of the object; and calling an output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result, the tactile measurement result comprising a local curvature radius of the surface of the object.",
    "14. The computer device according to claim 13, wherein the marking pattern comprises a grid, and the difference feature of the marking patterns comprises at least one of displacement of grid points and deformation of grid lines in the grid.",
    "15. The computer device according to claim 13, wherein the calculating a difference feature of the marking patterns based on the marking patterns in adjacent images of the image sequence comprises: determining two closest marking patterns from adjacent images at an i th frame and an (i+1) th frame in the image sequence as the same marking pattern, i being an integer; and calculating the difference feature of the marking patterns based on at least one of locations and deformation of the marking patterns in the images at the i th frame and the (i+1) th frame.",
    "16. The computer device according to claim 13, wherein the feedforward neural network comprises: a location estimation model, and the location estimation model comprises a first hidden layer and a first output layer; the calling the hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object comprises: calling the first hidden layer in the location estimation model to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of a contact location on the surface of the object; and the calling the output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result comprises: calling the first output layer in the location estimation model to process the feature representation of the contact location, to obtain the contact location on the surface of the object.",
    "17. The computer device according to claim 13, wherein the feedforward neural network comprises: a contact force estimation model, and the contact force estimation model comprises a second hidden layer and a second output layer; the calling the hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object comprises: calling the second hidden layer in the contact force estimation model to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the contact force on the surface of the object; and the calling the output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result comprises: calling the second output layer in the contact force estimation model to process the feature representation of the contact force, to obtain three-dimensional information of the contact force on the surface of the object, the three-dimensional information comprising at least one of a magnitude and a direction.",
    "18. A non-transitory computer-readable storage medium, storing a plurality of computer-readable instructions, wherein the plurality of computer-readable instruction when executed by a processor of a computer device in connection with a tactile sensor, the tactile sensor comprising a sensing face and an image sensing component, and the sensing face being provided with a marking pattern, cause the computer device to perform a plurality of operations including: obtaining an image sequence collected by the image sensing component of the sensing face that is in physical contact with a surface of an object, each image of the image sequence comprising one instance of the marking pattern; calculating a difference feature of the marking patterns in adjacent images of the image sequence; and processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result, further comprising: calling a hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object, the feature presentation further including a curvature prediction on the surface of the object; and calling an output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result, the tactile measurement result comprising a local curvature radius of the surface of the object.",
    "19. The non-transitory computer-readable storage medium according to claim 18, wherein the marking pattern comprises a grid, and the difference feature of the marking patterns comprises at least one of displacement of grid points and deformation of grid lines in the grid.",
    "20. The non-transitory computer-readable storage medium according to claim 18, wherein the calculating a difference feature of the marking patterns based on the marking patterns in adjacent images of the image sequence comprises: determining two closest marking patterns from adjacent images at an i th frame and an (i+1) th frame in the image sequence as the same marking pattern, i being an integer; and calculating the difference feature of the marking patterns based on at least one of locations and deformation of the marking patterns in the images at the i th frame and the (i+1) th frame."
  ],
  "description_excerpt": "This application relates to the field of human-machine interaction, and in particular, to a vision-based tactile measurement method and apparatus, a chip, and a storage medium.\n\nA tactile sensor is a sensor configured to imitate a tactile function, and can perform tactile measurement, such as a contact location and a contact force, on a contact object. Currently, tactile sensors are mostly used in the field of robots. A tactile sensor is provided in the related art. The tactile sensor is provided with a semicircular flexible sensing face. An inner surface of the flexible sensing face is provided with a plurality of marking points arranged in an array, and an image sensing component provided toward the inner surface. After an outer surface of the flexible sensing face comes into contact with an object, the flexible sensing face is deformed, which causes the plurality of marking points on the inner surface to change locations due to the deformation. The image sensing component collects an image of the inner surface of the flexible sensing face, and transmits the image of the inner surface to a chip. The chip is provided with a convolutional neural network (CNN), and the image of the inner surface is processed through the CNN to obtain an analysis result of a contact force.\n\nA training process of the foregoing CNN is relatively complex and up to 20,000 training samples are required to achieve a good training effect.",
  "cpc": [
    "B25J 13/084",
    "G01B 5/20",
    "G06F 18/21",
    "G06F 18/22",
    "G06F 18/24",
    "G06F 3/016",
    "G06N 3/044",
    "G06N 3/045",
    "G06N 3/048",
    "G06N 3/0499",
    "G06N 3/08",
    "G06N 3/084",
    "G06N 3/09",
    "G06T 2207/20084",
    "G06T 7/40",
    "G06T 7/64",
    "G06V 10/225",
    "G06V 10/758",
    "G06V 10/764",
    "G06V 10/774",
    "G06V 10/776",
    "G06V 10/82",
    "G06V 20/10"
  ],
  "ipc": [
    "B25J 13/08",
    "G01B 5/20",
    "G06F 18/21",
    "G06F 18/22",
    "G06F 18/24",
    "G06N 3/08",
    "G06T 7/40",
    "G06T 7/64",
    "G06V 10/22",
    "G06V 10/75",
    "G06V 10/764",
    "G06V 10/774",
    "G06V 10/776",
    "G06V 10/82",
    "G06V 20/10"
  ],
  "assignees": [
    "TENCENT TECH SHENZHEN CO LTD"
  ],
  "inventors": [
    "ZHENG YU",
    "XU ZHONGJIN",
    "ZHANG ZHENGYOU"
  ],
  "filing_date": "2021-07-07",
  "publication_date": "2025-02-04",
  "grant_date": "2025-02-04",
  "priority_date": "2019-05-16",
  "application_number": "US-202117369837-A",
  "family_id": "67631149",
  "citations": [
    "CN106092382A",
    "CN106204620A",
    "CN106650918A",
    "CN107589830A",
    "CN108229548A",
    "CN109716361A",
    "CN110162175A",
    "CN1203142A",
    "CN1539604A",
    "US2011029470A1",
    "US2022207864A1",
    "WO2017209660A1"
  ]
}