Automated detection of problem indicators in video of display output

{
  "publication_number": "US10341648B1",
  "country": "US",
  "kind": "B1",
  "title": "Automated detection of problem indicators in video of display output",
  "abstract": "Systems, methods, and computer-readable media are described for performing automated analysis of frame-over-frame pixel changes in recorded video of the display output of a computing device to determine whether the computing device presented an animated buffering indicator or other animated problem indicator. A system may be configured to detect motion by determining frame-over-frame pixel intensity changes at various pixel locations across a number of frames, then to determine whether the pixel locations that suggest motion (such as those pixel locations that had sufficient intensity change when accounting for potential noise in the video data) are concentrated in an area of the screen in which problem indicators are expected to be displayed. The system may then determine whether the shape of the pixel locations that indicated sufficient motion match an expected shape or path of motion for a given class of problem indictors.",
  "claims": [
    "1. A system comprising: an electronic data store that stores a digital video, wherein the digital video comprises a plurality of frames, wherein the digital video comprises a recording of display output from a computing device; and a hardware processor in communication with the electronic data store, the hardware processor configured to execute computer-executable instructions to at least: select a set of frames from the digital video; for each of a plurality of frame pairings within the set of frames, determine a pixel change value for each of a plurality of pixel locations, wherein the pixel change value for each pixel location represents a mathematical difference between an intensity value at the pixel location in a first frame of the frame pairing and an intensity value at the pixel location in a second frame of the frame pairing; determine adjusted pixel change values for each of the plurality of pixel locations, wherein the adjusted pixel change value for each individual pixel location comprises the lowest pixel change value determined within a window that includes the individual pixel location and one or more adjacent pixel locations; determine, for each pixel location, a final pixel change value for the set of frames, wherein the final pixel change value for each individual pixel location comprises the highest adjusted pixel change value determined for the individual pixel location in any frame pairing within the set of frames; generate a target motion score representing motion in a target area over the set of frames, wherein the target area is smaller than a size of each frame within the set of frames and has a fixed position across each of the set of frames, wherein the target motion score is based on the highest final pixel change value determined for any pixel location within the target area; generate a peripheral motion score representing motion outside of the target area over the set of frames, wherein the peripheral motion score is based on the highest final pixel change value determined for any pixel location outside of the target area; determine that isolated motion occurred within the target area over the set of frames based at least in part on a comparison of the target motion score to the peripheral motion score; based at least in part on the final pixel change values of pixel locations within the target area, identify that a shape of the motion within the target area matches a target motion shape associated with a problem indicator; and store an indication that a problem indicator appeared in the digital video during at least a portion of the set of frames.",
    "2. The system of claim 1, wherein the problem indicator is a buffering indicator that animates to have a spinning appearance, wherein the target motion shape is a circle.",
    "3. The system of claim 2, wherein the hardware processor is configured to apply a circle Hough Transform with respect to final pixel change values determined for pixel locations within the target area, wherein the shape of the motion within the target area is determined to match the target motion shape based at least in part on results of the circle Hough Transform.",
    "4. The system of claim 1, wherein the system further comprises a recording device comprising at least one of a camera or a capture card, wherein the recording device is configured to capture and store the display output from the computing device.",
    "5. The system of claim 1, wherein the system further comprises: a robotic arm having a tip configured to contact a touch-sensitive display screen of the computing device; and wherein the hardware processor is further configured to cause the robotic arm to perform one or more movements that cause the tip of the robotic arm to touch at least one specific location on the touch-sensitive display screen in response to the hardware processor determining that the problem indicator appeared in the digital video.",
    "6. A computer-implemented method comprising: as implemented by one or more computing devices configured with specific executable instructions, selecting a set of frames from a video, wherein the video comprises a plurality of frames depicting display output from a computing system; for each of a plurality of frame pairings within the set of frames, determining a pixel change value for each of a plurality of pixel locations, wherein the pixel change value for each pixel location represents a mathematical difference between an intensity value at the pixel location in a first frame of the frame pairing and an intensity value at the pixel location in a second frame of the frame pairing; determining adjusted pixel change values for each of the plurality of pixel locations, wherein the adjusted pixel change value for each individual pixel location is based at least in part on pixel change values of one or more other pixel locations that are adjacent to the individual pixel location; determining, for each pixel location, a final pixel change value for the set of frames, wherein the final pixel change value for each individual pixel location is based at least in part on the highest adjusted pixel change value determined for the individual pixel location in any frame pairing within the set of frames; based at least in part on the final pixel change values for the set of frames: generating a target motion score representing motion in a target area over the set of frames, wherein the target area is a subset of pixel locations of each individual frame and is consistently located across the set of frames; and generating a peripheral motion score representing motion outside of the target area over the set of frames; determining that isolated motion occurred within the target area over the set of frames based at least in part on a comparison of the target motion score to the peripheral motion score; identifying that a shape of the motion within the target area matches a target motion shape associated with a problem indicator; and storing an indication that a problem indicator appeared in the video during at least a portion of the set of frames.",
    "7. The computer-implemented method of claim 6, wherein at least a portion of the problem indicator follows a circular path of motion, and wherein the target motion shape is a circle.",
    "8. The computer-implemented method of claim 6, wherein the problem indicator represents that the computing system is in a state associated with at least one of (a) buffering content from a network source or (b) loading content locally.",
    "9. The computer-implemented method of claim 6, wherein the computing system having its display output depicted in the video is one of a mobile computing device, a desktop computer, or a set-top box.",
    "10. The computer-implemented method of claim 6, wherein a bounding region defining the target area is selected from among a plurality of stored bounding regions that are each associated with a different application or service.",
    "11. The computer-implemented method of claim 6, further comprising determining the target motion shape using a neural network that has been trained based on a plurality of videos known to include depiction of at least one problem indicator.",
    "12. The computer-implemented method of claim 6, further comprising storing the final pixel change values in a matrix, wherein the matrix has dimensions that match or approximate dimensions by pixel count of a frame in the video.",
    "13. The computer-implemented method of claim 6, further comprising, prior to determining the pixel change values, converting a color value of each pixel in the set of frames to be a gray scale value.",
    "14. The computer-implemented method of claim 6, further comprising, prior to storing an indication that a problem indicator appeared, determining that there is less than a threshold amount of noise in the final pixel change values for pixel locations outside of the identified target motion shape.",
    "15. A non-transitory physical computer storage medium storing computer-executable instructions that, when executed by one or more processors, configure the one or more processors to perform operations comprising: selecting a set of frames from a video, wherein the video comprises a plurality of frames depicting display output from a computing system; determining a pixel change value for each of a plurality of pixel locations, wherein the pixel change value for each pixel location represents an extent of pixel intensity change at the individual pixel location over at least two frames in the set of frames; determining, based at least in part on pixel change values for a plurality of pixel locations, that motion occurred within a target area over the set of frames; identifying that a shape of the motion within the target area matches a target motion shape associated with a problem indicator; and generating a signal indicating that a problem indicator appeared in the video during at least a portion of the set of frames.",
    "16. The non-transitory physical computer storage medium of claim 15, wherein the target area is a center portion of each frame in the set of frames.",
    "17. The non-transitory physical computer storage medium of claim 15, wherein the signal causes a responsive action to be performed with respect to the computing system.",
    "18. The non-transitory physical computer storage medium of claim 17, wherein the responsive action comprises one of: stopping an experiment being performed in association with the computing system, or presenting a notification that a problem has occurred with respect to the computing system.",
    "19. The non-transitory physical computer storage medium of claim 15, wherein the problem indicator is a buffering indicator indicating that content playback by the computing system has been interrupted or paused as a result of network conditions.",
    "20. The non-transitory physical computer storage medium of claim 15, wherein the operations further comprise, prior to determining that motion occurred within the target area: applying noise reduction to the pixel change values, wherein applying the noise reduction includes modifying a pixel change value of at least one pixel location based on pixel change values of one or more neighboring pixel locations."
  ],
  "description_excerpt": "Benchmarking and performance testing of computer software and hardware is commonly performed for a variety of reasons. For example, a developer of software, a hardware manufacturer, a network service provider, or other party may desire to compare the performance of different combinations of software and hardware under various conditions, such as with different hardware configurations, under different operating loads, with different network conditions, etc. When the party testing software is the software's developer, the software's code may be configured to output or store error codes or other specific data when specific problems are encountered during runtime. However, identifying specific problems that occur or that affect usability of a given system or specific software may be more difficult when such a problem must be assessed by a third party, particularly when the software or service being tested or otherwise subject to an experiment conveys problems in a visual manner that is designed to convey the problem to a human viewing a computer monitor or other display rather than in the form of an automated data output log or similar output.",
  "cpc": [
    "G06T 7/254",
    "G06T 13/80",
    "G06T 2207/10016",
    "G06T 7/0002",
    "G06T 7/20",
    "G06V 10/17",
    "G06V 10/40",
    "G06V 10/48",
    "G06V 20/46",
    "G06V 2201/06",
    "H04N 17/004"
  ],
  "ipc": [
    "G06T 13/80",
    "G06T 7/00",
    "G06T 7/20",
    "G06V 10/40",
    "H04N 17/00"
  ],
  "assignees": [
    "Amazon Technologies Inc"
  ],
  "inventors": [
    "Spencer Patrick Fonte",
    "Sean James Amos",
    "Michael Aaron Fox"
  ],
  "filing_date": "2017-09-20",
  "publication_date": "2019-07-02",
  "grant_date": "2019-07-02",
  "priority_date": "2017-09-20",
  "application_number": "US-201715710269-A",
  "family_id": "67069676",
  "cited_by_count": 4,
  "citations": [
    "US20090006999A1",
    "US20130249796A1",
    "US20150193001A1",
    "US20140354826A1",
    "US20150265920A1",
    "US20160019406A1",
    "US20180217680A1"
  ]
}