Dynamic adaptation of feature identification and annotation

{
  "publication_number": "US9715508B1",
  "country": "US",
  "kind": "B1",
  "title": "Dynamic adaptation of feature identification and annotation",
  "abstract": "A system may use a configurable detected to identify a feature in a received image and an associated candidate tag based on user-defined items of interest, and to determine an associated accuracy metric. Moreover, based on the accuracy metric, costs of requesting the feedback from one or more individuals and a feedback threshold, the system may use a scheduler to selectively obtain feedback, having a feedback accuracy, about the candidate tag from the one or more individuals. Then, the system may generate a revised tag based on the feedback when the feedback indicates the candidate tag is incorrect. Next, the system presents a result with the feature and the candidate tag or the revised tag to another electronic device. Furthermore, based on a quality metric, the system may update labeled data that are to be used to retrain the configurable detector.",
  "claims": [
    "1. A system, comprising: an interface circuit that, during operation, communicates, via a network, with another electronic device; a processor, and memory, coupled to the processor, which stores a program module that, during operation, is executed by the processor, the program module including: instructions for receiving input data including an image; instructions for identifying, using a configurable detector associated with labeled data, a feature in the image and an associated candidate tag based on user-defined items of interest, and determining an associated accuracy metric based on the feature and the candidate tag, wherein the user-defined items of interest include one or more features and one or more associated tags; instructions for selectively obtaining, using a scheduler, feedback, having a feedback accuracy, about the candidate tag from a set of individuals based on the accuracy metric, a cost of requesting the feedback and a feedback threshold, wherein the scheduler dynamically selects a given individual in the set of individuals based on a given cost of requesting the feedback from the given individual; instructions for generating a revised tag based on the feedback when the feedback indicates the candidate tag is incorrect; instructions for presenting a result to the other electronic device, wherein the result includes one of: the feature and the candidate tag when the accuracy metric exceeds a detection threshold; and the feature and the revised tag when the feedback accuracy exceeds the detection threshold; and instructions for updating, based on a quality metric, the labeled data to include the input data, the feature and one of the candidate tag and the revised tag, wherein the updated labeled data are to be used to retrain the configurable detector.",
    "2. The system of claim 1, wherein the accuracy metric is one of: a probability of accuracy, a distance from a target and the probability of accuracy and the distance from the target.",
    "3. The system of claim 1, wherein the cost is determined based on user-specified information and behavior of the given individual in response to prior requests for feedback.",
    "4. The system of claim 3, wherein the user-specified information includes one or more of: a frequency of feedback requests, a number of feedback requests, and a value of the feedback.",
    "5. The system of claim 1, wherein the scheduler further dynamically selects the given individual based on at least one of: an area of expertise of the given individual, behavior of the given individual in response to prior requests for feedback, and a feedback accuracy of prior feedback from the given individual.",
    "6. The system of claim 1, wherein the program module further comprises instructions for adjusting types of feedback obtained from the set of individuals based on the accuracy metric.",
    "7. The system of claim 1, wherein the feedback from the given individual includes one of: a first indication that the candidate tag is correct, a second indication that the candidate tag is incorrect, and a third indication that it is unknown whether the candidate tag is correct.",
    "8. The system of claim 7, wherein, when the feedback includes the second indication, the feedback further includes a correction for the candidate tag.",
    "9. The system of claim 1, wherein the program module further comprises instructions for adapting the feedback threshold based on the accuracy metric.",
    "10. The system of claim 1, wherein the program module further comprises instructions for adapting the detection threshold based on: the feedback from the set of individuals; and an economic value associated with the feature.",
    "11. The system of claim 1, wherein the set of individuals includes two or more individuals and the feedback accuracy for the given individual is based on the feedback obtained from the set of individuals.",
    "12. The system of claim 1, wherein the input data further includes other types of data than the image.",
    "13. The system of claim 1, wherein the program module further comprises instructions for receiving metadata associated with the input data; and wherein the metadata includes one or more of: information specifying a location of a source of the input data, a type of the source, other features in the image than the feature, attributes of the image other than the candidate tag, and a time of day the image was acquired.",
    "14. The system of claim 13, wherein the program module further comprises instructions for: selecting a subset of the labeled data based on the metadata; and training the configurable detector based on the selected subset.",
    "15. The system of claim 13, wherein the program module further comprises instructions for selecting the configurable detector in a set of configurable detectors based on the metadata.",
    "16. The system of claim 1, wherein the feedback is obtained when the accuracy metric is less than the feedback threshold.",
    "17. The system of claim 1, wherein, when the accuracy metric is greater than the feedback threshold, the feedback is obtained based on one of: a pseudorandom value, and a random value.",
    "18. The system of claim 1, wherein the quality metric is based on one or more of: the accuracy metric, the feedback accuracy, and a number of individuals in the set of individuals.",
    "19. The system of claim 1, wherein the program module further comprises instructions for retraining the configurable detector based on the updated labeled data.",
    "20. The system of claim 1, wherein presenting the result involves automatically triggering an application on the other electronic device that presents the result.",
    "21. A system, comprising: a configurable detector, associated with labeled data, comprising a non-transitory computer readable medium storing instructions that, when executed, cause the system to: receive input data including an image; and identify a feature in the image and an associated candidate tag based on user-defined items of interest, and determine an associated accuracy metric based on the feature and the candidate tag, wherein the user-defined items of interest include one or more features and one or more associated tags; a scheduler comprising a non-transitory computer readable medium storing instructions that, when executed, cause the system to selectively obtain feedback, having a feedback accuracy, about the candidate tag from a set of individuals based on the accuracy metric, a cost of requesting the feedback and a feedback threshold, wherein the scheduler dynamically selects a given individual in the set of individuals based on a given cost of requesting the feedback from the given individual; a revision module comprising a non-transitory computer readable medium storing instructions that, when executed, cause the system to generate a revised tag based on the feedback when the feedback indicates the candidate tag is incorrect; a presentation module comprising a non-transitory computer readable medium storing instructions that, when executed, cause the system to present a result to another electronic device, wherein the result includes one of: the feature and the candidate tag when the accuracy metric exceeds a detection threshold; and the feature and the revised tag when the feedback accuracy exceeds the detection threshold; and a training module comprising a non-transitory computer readable medium storing instructions that, when executed, cause the system to update, based on a quality metric, the labeled data to include the input data, the feature and one of the candidate tag and the revised tag, wherein the updated labeled data are to be used to retrain the configurable detector.",
    "22. A computer-program product for use in conjunction with a system, the computer-program product comprising a non-transitory computer-readable storage medium and a computer-program mechanism embedded therein to provide results, the computer-program mechanism including: instructions for receiving input data including an image; instructions for identifying, using a configurable detector associated with labeled data, a feature in the image and an associated candidate tag based on user-defined items of interest, and determining an associated accuracy metric based on the feature and the candidate tag, wherein the user-defined items of interest include one or more features and one or more associated tags; instructions for selectively obtaining, using a scheduler, feedback, having a feedback accuracy, about the candidate tag from a set of individuals based on the accuracy metric, a cost of requesting the feedback and a feedback threshold, wherein the scheduler dynamically selects a given individual in the set of individuals based on a given cost of requesting the feedback from the given individual; instructions for generating a revised tag based on the feedback when the feedback indicates the candidate tag is incorrect; instructions for presenting a result to the other electronic device, wherein the result includes one of: the feature and the candidate tag when the accuracy metric exceeds a detection threshold; and the feature and the revised tag when the feedback accuracy exceeds the detection threshold; and instructions for updating, based on a quality metric, the labeled data to include the input data, the feature and one of the candidate tag and the revised tag, wherein the updated labeled data are to be used to retrain the configurable detector.",
    "23. A method for selectively presenting results, the method comprising: receiving input data including an image; identifying, using a configurable detector associated with labeled data, a feature in the image and an associated candidate tag based on user-defined items of interest, and determining an associated accuracy metric based on the feature and the candidate tag, wherein the user-defined items of interest include one or more features and one or more associated tags; selectively obtaining, using a scheduler, feedback, having a feedback accuracy, about the candidate tag from a set of individuals based on the accuracy metric, a cost of requesting the feedback and a feedback threshold, wherein the scheduler dynamically selects a given individual in the set of individuals based on a given cost of requesting the feedback from the given individual; generating a revised tag based on the feedback when the feedback indicates the candidate tag is incorrect; presenting the result to the other electronic device, wherein the result includes one of: the feature and the candidate tag when the accuracy metric exceeds a detection threshold; and the feature and the revised tag when the feedback accuracy exceeds the detection threshold; and updating, based on a quality metric, the labeled data to include the input data, the feature and one of the candidate tag and the revised tag, wherein the updated labeled data are to be used to retrain the configurable detector."
  ],
  "description_excerpt": "The described embodiments relate to a technique for identifying and annotating a feature in an image using a configurable detector, including selectively obtaining feedback about a candidate tag associated with the identified feature and updating labeled data used to train the configurable detector based, at least in part, on the feedback.\n\nOngoing advances in the capabilities of electronic devices are making them increasingly popular. In addition, the widespread availability of electronic devices and their increasing functionality has resulted in a large number of applications.\n\nFor example, many electronic devices include image sensors, such as CMOS image sensors, that users and applications use to acquire images and videos. The content in these images and videos often includes useful information for the users. In principle, if the associated subset of the content that includes the information can be accurately identified, additional value-added services can be provided to the users.\n\nIn practice, it can be difficult to accurately identify the subset of the content (which are sometimes referred to as ‘features’) in the images. In particular, in order for a detector to accurately identify features in images, the detector typically needs to be first trained using a dataset. However, the dataset usually needs to include multiple instances of the features. It can be challenging to obtain multiple instances of an arbitrary feature a priori, which often degrades the quality of the trained detector and, thus, the accuracy of the subsequent feature identification.",
  "cpc": [
    "G06F 16/5866",
    "G06F 17/30268",
    "G06F 18/214",
    "G06F 18/2178",
    "G06K 9/03",
    "G06K 9/46",
    "G06K 9/6256",
    "G06V 10/7784",
    "G06V 10/96",
    "G06V 20/10",
    "G06V 20/70"
  ],
  "ipc": [
    "G06F 17/30",
    "G06K 9/00",
    "G06K 9/03",
    "G06K 9/46",
    "G06K 9/62"
  ],
  "assignees": [
    "Cogniac Corp"
  ],
  "inventors": [
    "William S. Kish",
    "Victor Shtrom",
    "Huayan Wang"
  ],
  "filing_date": "2016-03-28",
  "publication_date": "2017-07-25",
  "grant_date": "2017-07-25",
  "priority_date": "2016-03-28",
  "application_number": "US-201615081949-A",
  "family_id": "59350434",
  "cited_by_count": 35,
  "citations": [
    "US5963670A",
    "US20090179895A1",
    "US20130311868A1",
    "US20140016820A1",
    "US20140093174A1",
    "US20140104309A1",
    "US20150268058A1"
  ]
}