Anomalous object interaction detection and reporting

{
  "publication_number": "US9911043B2",
  "country": "US",
  "kind": "B2",
  "title": "Anomalous object interaction detection and reporting",
  "abstract": "Techniques are disclosed for analyzing a scene depicted in an input stream of video frames captured by a video camera. The techniques include evaluating sequence pairs representing segments of object trajectories. Assuming the objects interact, each of the sequences of the sequence pair may be mapped to a sequence cluster of an adaptive resonance theory (ART) network. A rareness value for the pair of sequence clusters may be determined based on learned joint probabilities of sequence cluster pairs. A statistical anomaly model, which may be specific to an interaction type or general to a plurality of interaction types, is used to determine an anomaly temperature, and alerts are issued based at least on the anomaly temperature. In addition, the ART network and the statistical anomaly model are updated based on the current interaction.",
  "claims": [
    "1. A computer-implemented method for analyzing a scene depicted in an input stream of video frames captured by a video camera, the method comprising: receiving at least two sequences, wherein each sequence of the at least two sequences corresponds to a segment of a trajectory taken by a respective object through the scene; determining, via one or more processors, whether the objects interact based on a spatio-temporal proximity of the objects; and if the objects interact: mapping each sequence of the at least two sequences to a respective sequence cluster, retrieving, from an ngram trie, a learned joint probability indicating a likelihood of a given sequence cluster pair of a plurality of sequence cluster pairs occurring in the scene, the ngram trie including a plurality of nodes, the given sequence cluster pair being represented jointly by a first node in a first layer of the ngram trie and a second node in a second layer of the ngram trie, the first node and the second node representing sequence clusters in the given sequence cluster pair, and determining a rareness value for each sequence cluster pair based on the learned joint probability and a frequency of a most frequently observed sequence cluster pair from the scene, the rareness value given by R ij = 1 - f ij f max, where f ij is a frequency with which sequence cluster pair {C i, C j } has been observed and f max is the frequency of the most frequently observed sequence cluster pair; determining, using a statistical anomaly model, an anomaly temperature for the rareness value, wherein the anomaly temperature indicates a frequency of occurrence of the rareness value; and upon determining one or more reporting criteria are met based at least on the anomaly temperature, reporting the interaction of the objects.",
    "2. The method of claim 1, wherein the received sequences each include one or more data points, each data point representing a position of the respective object at a given point in time, further comprising: normalizing the received sequences to n-dimensional vectors; and normalizing the position and time components of the received sequences.",
    "3. The method of claim 1, wherein the sequence clusters are clusters of an adaptive resonance theory (ART) network, and wherein the ART network is updated based on observed sequences.",
    "4. The method of claim 1, wherein the ngram trie is updated based on observed sequence cluster pairs.",
    "5. The method of claim 1, wherein the objects are deemed to interact if the objects pass within a given spatial neighborhood of each other in the scene within a given time period.",
    "6. The method of claim 1, further comprising, updating the statistical anomaly model based on observed interactions.",
    "7. The method of claim 1, further comprising: determining, based on the sequence clusters to which the sequences map and/or spatio-temporal properties of the sequences, a type of interaction between the objects, wherein the statistical anomaly model used is specific to the type of interaction.",
    "8. The method of claim 7, wherein the type of interaction is one of jail-breaking, collision/scattering, an extended interaction, an anomalous temporal interaction, an anomalous special interaction, and an anomalous spatio-temporal interaction.",
    "9. The method of claim 1, wherein the interaction is reported to a user interface and includes interaction properties (xInt, yInt, tInt), where xInt and yInt indicate x and y positions of the interaction and tInt indicates a time difference of the objects' trajectories at a spatial intersection of the trajectories, and wherein the user is permitted to create an alert directive to publish another alert if another interacting sequence pair occurs within a given spatial neighborhood (dx, dy) of (xInt, yInt) and temporal neighborhood dt of tInt.",
    "10. A non-transitory computer-readable storage medium storing instructions, which when executed by a computer system, perform operations for analyzing a scene depicted in an input stream of video frames captured by a video camera, the instructions comprising instructions to: receive at least two sequences, each sequence of the at least two sequences corresponding to a segment of a trajectory taken by a respective object through the scene; determine, via one or more processors, whether the objects interact based on a spatio-temporal proximity of the objects; and if the objects interact: map each sequence of the at least two sequences to a sequence cluster, retrieve, from an ngram trie, a learned joint probability indicating a likelihood of a given sequence cluster pair from a plurality of sequence cluster pairs occurring in the scene, the ngram trie including a plurality of nodes, the (liven sequence cluster pair being represented jointly by a first node in a first layer of the ngram trie and a second node in a second layer of the ngram trie, the first node and the second node representing sequence clusters in the (liven sequence cluster pair, and determine a rareness value for each sequence cluster pair of the plurality of sequence cluster pairs based on learned joint probabilities of sequence cluster pairs and a frequency of a most frequently observed sequence cluster pair, the rareness value based on R ij = 1 - f ij f max, where f ij is a frequency with which sequence cluster pair {C i, C j } has been observed and f max is the frequency of the most frequently observed sequence cluster pair; determine, using a statistical anomaly model, an anomaly temperature for the rareness value, wherein the anomaly temperature indicates a frequency of occurrence of the rareness value; and upon determination that one or more reporting criteria are met based at least on the anomaly temperature, report the interaction.",
    "11. The computer-readable storage medium of claim 10, wherein the received sequences each include one or more data points, each data point representing a position of the respective object at a given point in time, the instructions further comprising instructions to: normalize the received sequences to n-dimensional vectors; and normalize the position and time components of the received sequences.",
    "12. The computer-readable storage medium of claim 10, wherein the sequence clusters are clusters of an adaptive resonance theory (ART) network, and wherein the ART network is updated based on observed sequences.",
    "13. The computer-readable storage medium of claim 10, wherein the ngram trie is updated based on observed sequence cluster pairs.",
    "14. The computer-readable storage medium of claim 10, wherein the objects are deemed to interact if the objects pass within a given spatial neighborhood of each other in the scene within a given time period.",
    "15. The computer-readable storage medium of claim 10, the instructions further comprising instructions to: update the statistical anomaly model based on observed interactions.",
    "16. The computer-readable storage medium of claim 10, the instructions further comprising instructions to: determine, based on the sequence clusters to which the sequences map and/or spatio-temporal properties of the sequences, a type of interaction between the objects, wherein the statistical anomaly model used is specific to the type of interaction.",
    "17. The computer-readable storage medium of claim 16, wherein the type of interaction is one of jail-breaking, collision/scattering, an extended interaction, an anomalous temporal interaction, an anomalous special interaction, and an anomalous spatio-temporal interaction.",
    "18. The computer-readable storage medium of claim 10, wherein the interaction is reported to a user interface and includes interaction properties (xInt, yInt, tInt), where xInt and yInt indicate x and y positions of the interaction and tInt indicates a time difference of the objects' trajectories at a spatial intersection of the trajectories, and wherein the user is permitted to create an alert directive to publish another alert if another interacting sequence pair occurs within a given spatial neighborhood (dx, dy) of (xInt, yInt) and temporal neighborhood dt of tInt.",
    "19. A system, comprising: a processor; and a memory, wherein the memory includes an application program configured to perform operations for analyzing a scene depicted in an input stream of video frames captured by a video camera, the operations comprising: receiving at least two sequences, wherein each sequence of the at least two sequences corresponds to a segment of a trajectory taken by a respective object through the scene, determining, via one or more processors, whether the objects interact based on a spatio-temporal proximity of the objects, and if the objects interact: mapping each sequence of the at least two sequences to a sequence cluster; retrieving, from an ngram trie, learned joint probabilities of sequence cluster pairs, a learned joint probability indicating a likelihood of a sequence cluster pair occurring in the scene, the ngram trie including a plurality of nodes, the sequence cluster pair being represented jointly by a first node in a first layer of the ngram trie and a second node in a second layer of the ngram trie, the first node and the second node representing sequence clusters in the sequence cluster pair, and determining a rareness value based on the learned joint probabilities and a frequency of a most frequently observed sequence cluster pair, the rareness value based on R ij = 1 - f ij f max, where f ij is a frequency with which sequence cluster pair {C i, C j } has been observed and f max is the frequency of the most frequently observed sequence cluster pair; determining, using a statistical anomaly model, an anomaly temperature for the rareness value, wherein the anomaly temperature indicates a frequency of occurrence of the rareness value, and upon determining one or more reporting criteria are met based at least on the anomaly temperature, reporting the interaction.",
    "20. The system of claim 19, wherein the received sequences each include one or more data points, each data point representing a position of the respective object at a given point in time, the operations further comprising: normalizing the received sequences to n-dimensional vectors; and normalizing the position and time components of the received sequences.",
    "21. The system of claim 19, wherein the ngram trie is updated based on observed sequence cluster pairs.",
    "22. The system of claim 19, the operations further comprising: determining, based on the sequence clusters to which the sequences map and/or spatio-temporal properties of the sequences, a type of interaction between the objects, wherein the statistical anomaly model used is specific to the type of interaction.",
    "23. The system of claim 22, wherein the type of interaction is one of jail-breaking, collision/scattering, an extended interaction, an anomalous temporal interaction, an anomalous special interaction, and an anomalous spatio-temporal interaction."
  ],
  "description_excerpt": "Field of the Invention\n\nEmbodiments of the invention provide techniques for analyzing a sequence of video frames. More particularly, to analyzing and learning behavior based on streaming video data, including detection and reporting of anomalous object interactions.\n\nDescription of the Related Art\n\nSome currently available video surveillance systems provide simple object recognition capabilities. For example, a video surveillance system may be configured to classify a group of pixels (referred to as a “blob”) in a given frame as being a particular object (e.g., a person or vehicle). Once identified, a “blob” may be tracked from frame-to-frame in order to follow the “blob” moving through the scene over time, e.g., a person walking across the field of vision of a video surveillance camera. Further, such systems may be configured to determine when an object has engaged in certain predefined behaviors. For example, the system may include definitions used to recognize the occurrence of a number of pre-defined events, e.g., the system may evaluate the appearance of an object classified as depicting a car (a vehicle-appear event) coming to a stop over a number of frames (a vehicle-stop event). Thereafter, a new foreground object may appear and be classified as a person (a person-appear event) and the person then walks out of frame (a person-disappear event).",
  "cpc": [
    "G06V 20/40",
    "G06F 18/2321",
    "G06F 18/2433",
    "G06K 9/00711",
    "G06K 9/6221",
    "G06K 9/6284",
    "G06N 3/0409",
    "G06N 3/088",
    "G06V 10/763",
    "G06V 10/764",
    "G06V 20/52"
  ],
  "ipc": [
    "G06T 7/00",
    "G06V 10/764",
    "G06T 13/00"
  ],
  "assignees": [
    "Omni AI Inc"
  ],
  "inventors": [
    "Kishor Adinath Saitwal",
    "Dennis G. Urech",
    "Wesley Kenneth Cobb"
  ],
  "filing_date": "2013-06-28",
  "publication_date": "2018-03-06",
  "grant_date": "2018-03-06",
  "priority_date": "2012-06-29",
  "application_number": "US-201313931058-A",
  "family_id": "52115633",
  "cited_by_count": 4,
  "citations": [
    "US4679077A",
    "US5113507A",
    "US5748775A",
    "US6252974B1",
    "US5969755A",
    "US5751378A",
    "US6263088B1",
    "US20050240629A1",
    "US6570608B1",
    "US6661918B1",
    "US20060193516A1",
    "US7136525B1",
    "US6674877B1",
    "US6940998B2",
    "US7868912B2",
    "US6678413B1",
    "US20090297023A1",
    "US7076102B2",
    "US20030107650A1",
    "US20060165386A1",
    "US7436887B2",
    "US6856249B2",
    "US20030228058A1",
    "US7227893B1",
    "US7200266B2",
    "US20040151342A1",
    "US20050001759A1",
    "US20050105765A1",
    "US20060018516A1",
    "US7158680B2",
    "US20060138338A1",
    "US20090067716A1",
    "US20060190419A1",
    "US20080181453A1",
    "US20080208828A1",
    "US20060222206A1",
    "US7825954B2",
    "US20070250898A1",
    "US20090210367A1",
    "US20080002856A1",
    "US20080181499A1",
    "US20080193010A1",
    "US20080252723A1",
    "US20080240496A1",
    "US20080247599A1",
    "US20090022364A1",
    "WO2009049314A2",
    "US20090153661A1",
    "US20100322516A1",
    "US20090324107A1",
    "US20100063949A1",
    "US20100150471A1",
    "US20100208063A1",
    "US20110044498A1",
    "US20110052068A1",
    "US20110051992A1",
    "US20120224746A1",
    "US20120238234A1"
  ]
}