MLchartDataset catalogue

Patent · US12403211B2 · B2 · US

Robot and control method for identifying a sterilization area

(11) Publication number
US12403211B2
(21) Application number
18/118,591
(22) Filing date
2023-03-07
(30) Priority date
2020-10-07
(43) Publication date
2025-09-02
(45) Date of grant
2025-09-02
(51) IPC
A61L 2/10; A61L 2/18; A61L 2/24; G06T 7/70; G06V 10/82; G06V 20/50; G06V 40/10; G10L 25/30; G10L 25/66; H04N 23/695; H04N 7/18; H04R 1/40; H04R 3/00
(52) CPC
  • A61L Methods or apparatus for sterilising materials or objects in general; disinfection, sterilisation or deodorisation of air; chemical aspects of bandages, dressings, absorbent pads or surgical articles; materials for bandages, dressings, absorbent pads or surgical articles: 2/24, 2/10, 2/18, 2103/75, 2202/14, 2202/15, 2202/16, 9/14, 9/20
  • B25J Manipulators; chambers provided with manipulation devices: 11/00, 11/0085, 13/00, 13/003, 19/02, 19/023, 9/00, 9/0009, 9/16, 9/161, 9/1679, 9/1697
  • G05D Systems for controlling or regulating non-electric variables: 1/0088, 1/0246, 1/0255
  • G06T Image data processing or generation, in general: 2207/20084, 2207/30196, 7/70
  • G06V Image or video recognition or understanding: 10/82, 20/50, 20/52, 40/10, 40/103, 40/16
  • G10L Speech analysis techniques or speech synthesis; speech recognition; speech or voice processing techniques; speech or audio coding or decoding: 25/30, 25/66
  • H04N Pictorial communication, e.g. television: 23/60, 23/695, 7/183, 7/188
  • H04R Loudspeakers, microphones, gramophone pick-ups or like acoustic electromechanical transducers; electric hearing AIDS; public address systems: 1/028, 1/406, 2430/20, 3/005
(73) Assignee
SAMSUNG ELECTRONICS CO LTD
(72) Inventors
KIM DOHOON; PARK YOSUB
(54) Title
Robot and control method for identifying a sterilization area
(57) Abstract

Provided is a robot comprising a driving part, a camera, a plurality of microphones arranged in different directions. The robot further comprises a memory storing instructions, and a processor configured to execute the instructions to identify an originating direction of an audio signal input through the plurality of microphones based on the audio signal being identified as corresponding to a coughing sound, control the camera to capture an image in the originating direction, identify a sterilization area based on a position of a user who does not wear a mask identified in the image, and control the driving part to move the robot to the sterilization area.

Full text
View on Google Patents

Claims (21)

  1. A method of controlling a robot comprising: identifying that an audio signal input through a plurality of microphones on the robot corresponds to a coughing sound, based on the audio signal input through the plurality of microphones on the robot being identified as corresponding to the coughing sound, identifying an originating direction of the audio signal based on an arrangement direction of each microphone and a difference in intensity of audio signals that are respectively input to the plurality of microphones, wherein the plurality of microphones are arranged in different directions; based on the audio signal input through the plurality of microphones being identified as corresponding to the coughing sound, capturing an image in the originating direction of the audio signal corresponding to the coughing sound using a camera on the robot; identifying a sterilization area based on a position of a non-mask wearing user identified from the image; and moving the robot to the sterilization area.
  2. The method of claim 1, wherein identifying the sterilization area comprises: identifying the sterilization area based on the position of the non-mask wearing user and a threshold range corresponding to an intensity of the audio signal.
  3. The method of claim 1, wherein identifying the sterilization area comprises: determining a scale of the sterilization area based on a number of non-mask wearing users identified from the image.
  4. The method of claim 1, wherein identifying the sterilization area comprises: identifying the sterilization area based on a plurality of positions corresponding to a plurality of non-mask wearing users identified from the image.
  5. The method of claim 1, wherein identifying the sterilization area comprises: determining at least one of a scale of the sterilization area and a sterilization intensity, based on whether the position of the non-mask wearing user is within a pre-set area.
  6. A non-transitory computer readable medium for storing computer readable program code or instructions which are executable by a processor to perform a method of controlling a robot, the method comprising: based on an audio signal input through a plurality of microphones on the robot being identified as corresponding to a coughing sound, identifying an originating direction of the audio signal based on an arrangement direction of each microphone and a difference in intensity of audio signals that are respectively input to the plurality of microphones, wherein the plurality of microphones are arranged in different directions; based on the audio signal input through the plurality of microphones being identified as corresponding to the coughing sound, capturing an image in the originating direction of the audio signal corresponding to the coughing sound using a camera on the robot; identifying a sterilization area based on a position of a non-mask wearing user identified from the image; and moving the robot to the sterilization area.
  7. The non-transitory computer readable medium of claim 6, wherein identifying the sterilization area comprises: identifying the sterilization area based on the position of the non-mask wearing user and a threshold range corresponding to an intensity of the audio signal.
  8. The non-transitory computer readable medium of claim 6, wherein identifying the sterilization area comprises: determining a scale of the sterilization area based on a number of non-mask wearing users identified from the image.
  9. The non-transitory computer readable medium of claim 6, wherein identifying the sterilization area comprises: identifying the sterilization area based on a plurality of positions corresponding to a plurality of non-mask wearing users identified from the image.
  10. The non-transitory computer readable medium of claim 6, wherein identifying the sterilization area comprises: determining at least one of a scale of the sterilization area and a sterilization intensity, based on whether the position of the non-mask wearing user is within a pre-set area.
  11. A robot, comprising: a driving part; a camera; a plurality of microphones arranged in different directions; a memory storing instructions; and a processor configured to execute the instructions to: based on an audio signal input through the plurality of microphones being identified as corresponding to a coughing sound, identify an originating direction of the audio signal based on a difference in intensity of audio signals that are respectively input to the plurality of microphones arranged in different directions, based on the audio signal input through the plurality of microphones being identified as corresponding to the coughing sound, control the camera to capture an image in the originating direction of the audio signal corresponding to the coughing sound, identify a sterilization area based on a position of a non-mask wearing user identified from the image, and control the driving part to move to the sterilization area.
  12. The robot of claim 11, wherein the processor is further configured to execute the instructions to: identify the sterilization area based on the position of the non-mask wearing user and a threshold range corresponding to an intensity of the audio signal.
  13. The robot of claim 11, wherein the processor is further configured to execute the instructions to: determine a scale of the sterilization area based on a number of non-mask wearing users identified from the image.
  14. The robot of claim 11, wherein the processor is further configured to execute the instructions to: identify the sterilization area based on a plurality of positions corresponding to a plurality of non-mask wearing users identified from the image.
  15. The robot of claim 11, wherein the processor is further configured to execute the instructions to: determine at least one of a scale of the sterilization area, area and a sterilization intensity, based on whether the position of the non-mask wearing user is within a pre-set area.
  16. The robot of claim 15, wherein the pre-set area comprises at least one of an area including an object with high contact frequency by users, an area with high visiting frequency by users, and an area with low mask wearing frequency by users.
  17. The robot of claim 11, wherein the processor is further configured to execute the instructions to: identify whether the audio signal corresponds to the coughing sound by using a first neural network model trained to identify whether the audio signal comprises a coughing sound.
  18. The robot of claim 11, wherein the processor is further configured to execute the instructions to: identify whether the image comprises the non-mask wearing user by using a neural network model trained to identify whether the image comprises a non-mask wearing user.
  19. The robot of claim 11, further comprising: a distance sensor, wherein the processor is further configured to execute the instructions to: based on a plurality of sterilization areas being identified, identify a direction and a distance with respect to each sterilization area using the distance sensor, and set a traveling route for a sterilization operation based on the direction and distance to each sterilization area.
  20. The robot of claim 11, further comprising: a sterilization device configured to perform a sterilization function, wherein the processor is further configured to execute the instructions to: control the sterilization device to perform the sterilization function, based on the robot having moved to the sterilization area.
  21. The robot of claim 11, wherein the processor is further configured to execute the instructions to: control the camera to capture an image of a space in which the robot is positioned, identify a ratio of users who are wearing a mask among users in the image, identify the sterilization area as an area in which a ratio of non-mask wearing people exceeds a threshold ratio.

Description

1. Field The disclosure relates to a robot that performs a sterilization function and a control method thereof. 2. Description of Related Art Preventing infections of contagious diseases (e.g., the highly contagious Coronavirus disease) is an important social issue. One way that an individual is infected by a virus is through droplet spraying via coughing or contact, but because sterilization robots of related art sterilize a droplet spraying point or a contact point through a user operation, there is difficulty in performing an automated sterilization function.

Provided is a robot which can identify a sterilization area on its own without user operation, and a control method thereof. According to the various embodiments of the disclosure, areas that require sterilization may be sterilized with only the robot and without using other sensors installed in an indoor space. According to an aspect of the disclosure, a robot includes: a driving part; a camera; a plurality of microphones arranged in different directions; a memory storing instructions; and a processor configured to execute the instructions to: based on an audio signal input through the plurality of microphones being identified as corresponding to a coughing sound, identify an originating direction of the audio signal, control the camera to capture an image in the originating direction, identify a sterilization area based on a position of a non-mask wearing user identified from the image, and control the driving part to move to the sterilization area.

Citations (32)

  • CN109701060A
  • CN111331571A
  • CN111358990A
  • CN111376268A
  • JP2017169613A
  • KR100695172B1
  • KR101742489B1
  • KR102156511B1
  • KR20170139354A
  • KR20180079824A
  • KR20190065569A
  • US11421905B2
  • US11763366B1
  • US12087284B1
  • US2007209143A1
  • US2019213438A1
  • US2020046302A1
  • US2020168339A1
  • US2020393159A1
  • US2021027893A1
  • US2021287792A1
  • US2021353785A1
  • US2022047209A1
  • US2022355481A1
  • US2022359069A1
  • US2022364758A1
  • US2022387641A1
  • US2022409089A1
  • US2023120290A1
  • US2023149581A1
  • US9352469B2
  • WO2019239812A1
Record as JSON 
{
  "publication_number": "US12403211B2",
  "country": "US",
  "kind": "B2",
  "title": "Robot and control method for identifying a sterilization area",
  "abstract": "Provided is a robot comprising a driving part, a camera, a plurality of microphones arranged in different directions. The robot further comprises a memory storing instructions, and a processor configured to execute the instructions to identify an originating direction of an audio signal input through the plurality of microphones based on the audio signal being identified as corresponding to a coughing sound, control the camera to capture an image in the originating direction, identify a sterilization area based on a position of a user who does not wear a mask identified in the image, and control the driving part to move the robot to the sterilization area.",
  "claims": [
    "1. A method of controlling a robot comprising: identifying that an audio signal input through a plurality of microphones on the robot corresponds to a coughing sound, based on the audio signal input through the plurality of microphones on the robot being identified as corresponding to the coughing sound, identifying an originating direction of the audio signal based on an arrangement direction of each microphone and a difference in intensity of audio signals that are respectively input to the plurality of microphones, wherein the plurality of microphones are arranged in different directions; based on the audio signal input through the plurality of microphones being identified as corresponding to the coughing sound, capturing an image in the originating direction of the audio signal corresponding to the coughing sound using a camera on the robot; identifying a sterilization area based on a position of a non-mask wearing user identified from the image; and moving the robot to the sterilization area.",
    "2. The method of claim 1, wherein identifying the sterilization area comprises: identifying the sterilization area based on the position of the non-mask wearing user and a threshold range corresponding to an intensity of the audio signal.",
    "3. The method of claim 1, wherein identifying the sterilization area comprises: determining a scale of the sterilization area based on a number of non-mask wearing users identified from the image.",
    "4. The method of claim 1, wherein identifying the sterilization area comprises: identifying the sterilization area based on a plurality of positions corresponding to a plurality of non-mask wearing users identified from the image.",
    "5. The method of claim 1, wherein identifying the sterilization area comprises: determining at least one of a scale of the sterilization area and a sterilization intensity, based on whether the position of the non-mask wearing user is within a pre-set area.",
    "6. A non-transitory computer readable medium for storing computer readable program code or instructions which are executable by a processor to perform a method of controlling a robot, the method comprising: based on an audio signal input through a plurality of microphones on the robot being identified as corresponding to a coughing sound, identifying an originating direction of the audio signal based on an arrangement direction of each microphone and a difference in intensity of audio signals that are respectively input to the plurality of microphones, wherein the plurality of microphones are arranged in different directions; based on the audio signal input through the plurality of microphones being identified as corresponding to the coughing sound, capturing an image in the originating direction of the audio signal corresponding to the coughing sound using a camera on the robot; identifying a sterilization area based on a position of a non-mask wearing user identified from the image; and moving the robot to the sterilization area.",
    "7. The non-transitory computer readable medium of claim 6, wherein identifying the sterilization area comprises: identifying the sterilization area based on the position of the non-mask wearing user and a threshold range corresponding to an intensity of the audio signal.",
    "8. The non-transitory computer readable medium of claim 6, wherein identifying the sterilization area comprises: determining a scale of the sterilization area based on a number of non-mask wearing users identified from the image.",
    "9. The non-transitory computer readable medium of claim 6, wherein identifying the sterilization area comprises: identifying the sterilization area based on a plurality of positions corresponding to a plurality of non-mask wearing users identified from the image.",
    "10. The non-transitory computer readable medium of claim 6, wherein identifying the sterilization area comprises: determining at least one of a scale of the sterilization area and a sterilization intensity, based on whether the position of the non-mask wearing user is within a pre-set area.",
    "11. A robot, comprising: a driving part; a camera; a plurality of microphones arranged in different directions; a memory storing instructions; and a processor configured to execute the instructions to: based on an audio signal input through the plurality of microphones being identified as corresponding to a coughing sound, identify an originating direction of the audio signal based on a difference in intensity of audio signals that are respectively input to the plurality of microphones arranged in different directions, based on the audio signal input through the plurality of microphones being identified as corresponding to the coughing sound, control the camera to capture an image in the originating direction of the audio signal corresponding to the coughing sound, identify a sterilization area based on a position of a non-mask wearing user identified from the image, and control the driving part to move to the sterilization area.",
    "12. The robot of claim 11, wherein the processor is further configured to execute the instructions to: identify the sterilization area based on the position of the non-mask wearing user and a threshold range corresponding to an intensity of the audio signal.",
    "13. The robot of claim 11, wherein the processor is further configured to execute the instructions to: determine a scale of the sterilization area based on a number of non-mask wearing users identified from the image.",
    "14. The robot of claim 11, wherein the processor is further configured to execute the instructions to: identify the sterilization area based on a plurality of positions corresponding to a plurality of non-mask wearing users identified from the image.",
    "15. The robot of claim 11, wherein the processor is further configured to execute the instructions to: determine at least one of a scale of the sterilization area, area and a sterilization intensity, based on whether the position of the non-mask wearing user is within a pre-set area.",
    "16. The robot of claim 15, wherein the pre-set area comprises at least one of an area including an object with high contact frequency by users, an area with high visiting frequency by users, and an area with low mask wearing frequency by users.",
    "17. The robot of claim 11, wherein the processor is further configured to execute the instructions to: identify whether the audio signal corresponds to the coughing sound by using a first neural network model trained to identify whether the audio signal comprises a coughing sound.",
    "18. The robot of claim 11, wherein the processor is further configured to execute the instructions to: identify whether the image comprises the non-mask wearing user by using a neural network model trained to identify whether the image comprises a non-mask wearing user.",
    "19. The robot of claim 11, further comprising: a distance sensor, wherein the processor is further configured to execute the instructions to: based on a plurality of sterilization areas being identified, identify a direction and a distance with respect to each sterilization area using the distance sensor, and set a traveling route for a sterilization operation based on the direction and distance to each sterilization area.",
    "20. The robot of claim 11, further comprising: a sterilization device configured to perform a sterilization function, wherein the processor is further configured to execute the instructions to: control the sterilization device to perform the sterilization function, based on the robot having moved to the sterilization area.",
    "21. The robot of claim 11, wherein the processor is further configured to execute the instructions to: control the camera to capture an image of a space in which the robot is positioned, identify a ratio of users who are wearing a mask among users in the image, identify the sterilization area as an area in which a ratio of non-mask wearing people exceeds a threshold ratio."
  ],
  "description_excerpt": "1. Field The disclosure relates to a robot that performs a sterilization function and a control method thereof. 2. Description of Related Art Preventing infections of contagious diseases (e.g., the highly contagious Coronavirus disease) is an important social issue. One way that an individual is infected by a virus is through droplet spraying via coughing or contact, but because sterilization robots of related art sterilize a droplet spraying point or a contact point through a user operation, there is difficulty in performing an automated sterilization function.\n\nProvided is a robot which can identify a sterilization area on its own without user operation, and a control method thereof. According to the various embodiments of the disclosure, areas that require sterilization may be sterilized with only the robot and without using other sensors installed in an indoor space. According to an aspect of the disclosure, a robot includes: a driving part; a camera; a plurality of microphones arranged in different directions; a memory storing instructions; and a processor configured to execute the instructions to: based on an audio signal input through the plurality of microphones being identified as corresponding to a coughing sound, identify an originating direction of the audio signal, control the camera to capture an image in the originating direction, identify a sterilization area based on a position of a non-mask wearing user identified from the image, and control the driving part to move to the sterilization area.",
  "cpc": [
    "A61L 2/24",
    "A61L 2/10",
    "A61L 2/18",
    "A61L 2103/75",
    "A61L 2202/14",
    "A61L 2202/15",
    "A61L 2202/16",
    "A61L 9/14",
    "A61L 9/20",
    "B25J 11/00",
    "B25J 11/0085",
    "B25J 13/00",
    "B25J 13/003",
    "B25J 19/02",
    "B25J 19/023",
    "B25J 9/00",
    "B25J 9/0009",
    "B25J 9/16",
    "B25J 9/161",
    "B25J 9/1679",
    "B25J 9/1697",
    "G05D 1/0088",
    "G05D 1/0246",
    "G05D 1/0255",
    "G06T 2207/20084",
    "G06T 2207/30196",
    "G06T 7/70",
    "G06V 10/82",
    "G06V 20/50",
    "G06V 20/52",
    "G06V 40/10",
    "G06V 40/103",
    "G06V 40/16",
    "G10L 25/30",
    "G10L 25/66",
    "H04N 23/60",
    "H04N 23/695",
    "H04N 7/183",
    "H04N 7/188",
    "H04R 1/028",
    "H04R 1/406",
    "H04R 2430/20",
    "H04R 3/005"
  ],
  "ipc": [
    "A61L 2/10",
    "A61L 2/18",
    "A61L 2/24",
    "G06T 7/70",
    "G06V 10/82",
    "G06V 20/50",
    "G06V 40/10",
    "G10L 25/30",
    "G10L 25/66",
    "H04N 23/695",
    "H04N 7/18",
    "H04R 1/40",
    "H04R 3/00"
  ],
  "assignees": [
    "SAMSUNG ELECTRONICS CO LTD"
  ],
  "inventors": [
    "KIM DOHOON",
    "PARK YOSUB"
  ],
  "filing_date": "2023-03-07",
  "publication_date": "2025-09-02",
  "grant_date": "2025-09-02",
  "priority_date": "2020-10-07",
  "application_number": "US-202318118591-A",
  "family_id": "81126628",
  "citations": [
    "CN109701060A",
    "CN111331571A",
    "CN111358990A",
    "CN111376268A",
    "JP2017169613A",
    "KR100695172B1",
    "KR101742489B1",
    "KR102156511B1",
    "KR20170139354A",
    "KR20180079824A",
    "KR20190065569A",
    "US11421905B2",
    "US11763366B1",
    "US12087284B1",
    "US2007209143A1",
    "US2019213438A1",
    "US2020046302A1",
    "US2020168339A1",
    "US2020393159A1",
    "US2021027893A1",
    "US2021287792A1",
    "US2021353785A1",
    "US2022047209A1",
    "US2022355481A1",
    "US2022359069A1",
    "US2022364758A1",
    "US2022387641A1",
    "US2022409089A1",
    "US2023120290A1",
    "US2023149581A1",
    "US9352469B2",
    "WO2019239812A1"
  ]
}

Record 87 of 5,000 in Patents full text (MLC-0201). Request the full dataset.