Measuring latency in a test system using captured images

{
  "publication_number": "US9514664B2",
  "country": "US",
  "kind": "B2",
  "title": "Measuring latency in a test system using captured images",
  "abstract": "A latency measurement system includes an event generation device that generates an initial event used to measure system latency. A component test system receives the event and in response outputs a test component output signal and a zero-latency indicator. An electronics system including a multifunction display unit receives the test component output signal and displays a visible element on the multifunction display unit. A camera generates a series of recorded images, where each recorded image contains an image of the zero-latency indicator and an image of the visible element. A processor then determines the system latency by determining a time difference in the series of recorded images between a representation of an occurrence of the event in the image of the zero-latency indicator and a representation of the occurrence of the event in the image of the visible element.",
  "claims": [
    "1. A latency measurement system, the system comprising: an event generation device that generates an event used to measure latency; a hardware test component for controlling a vehicle, wherein the test component is configured to receive the event and in response output a test component output signal; a zero-latency indicator configured to output a visual indication of the event, wherein the zero-latency indicator and the test component receive the event from the event generation device at substantially the same time; an electronics system including a multifunction display device, where the electronics system receives the test component output signal and, in response, displays an image on the multifunction display device; a camera that generates a series of recorded images, where each recorded image contains the zero-latency indicator and the image displayed by the multifunction display device; and a processor that determines a latency of the test component by determining a time difference in the series of recorded images between a representation of an occurrence of the event as indicated by the zero-latency indicator and a representation of the occurrence of the event as indicated by the image of the multifunction display device.",
    "2. The latency measurement system of claim 1, wherein the event further comprises one of: an event trigger signal; rotational motion; or electromagnetic radiation.",
    "3. The latency measurement system of claim 1, wherein the test component is one of an imaging device, an actuator, or an inertial navigation system and the zero-latency indicator comprises an electromagnetic radiation emitter.",
    "4. The latency measurement system of claim 1, zero-latency indicator comprises an electromagnetic radiation emitter.",
    "5. The latency measurement system of claim 4, wherein the electromagnetic radiation emitter further comprises: an infrared illumination source; a low-level visible light illumination source; a visible light source; or a laser output of a laser level.",
    "6. The latency measurement system of claim 1, wherein the electronics system further comprises at least one of: an avionics control system; a vehicle control system; or a ship control system.",
    "7. The latency measurement system of claim 1, wherein the multifunction display device further comprises at least one of: a graphical image display device that graphically displays the image based input received at the electronics system; a warning/caution/advisory (WCA) indicator display that is activated based input received at the electronics system; or an artificial horizon display that corresponds to input received at the electronics system.",
    "8. The latency measurement system of claim 1, where determining the time difference further comprises: detecting a threshold within the representation of the occurrence of the event indicated by the zero-latency indicator and a threshold within the representation of the occurrence of the event indicated by the image; creating a plurality of binary waveforms based on detecting the threshold within the representation of the occurrence of the event as indicated by the zero-latency indicator and the threshold within the representation of the occurrence of the event as indicated by the image; detecting edges for each of the plurality of binary waveforms that measure one of rising or falling; and measuring a delta time between the zero-latency indicator and the image in the multifunction display device based on the detected edges for each of the plurality of binary waveforms.",
    "9. The latency measurement system of claim 1, wherein determining the time difference further comprises: determining a difference in a rate of angular change between the representation of the occurrence of the event as indicated by the zero-latency indicator and a rate of angular change of the representation of the occurrence of the event as indicated by the image.",
    "10. A method of measuring latency, the method comprising: triggering an event that generates an indication of the event at both a hardware test component for controlling a vehicle and a zero-latency indicator at substantially the same time; sending the indication of the event from the test component to a multi-function display device which displays an image in response to the indication of the event; recording a series of images with a camera, where each image contains both the zero-latency indicator and the image displayed by the multi-function display device; and processing, by a computing device, the series of images to compute a latency value of the test component based on determining a time difference between the indication of the event by the zero-latency indicator and the multi-function display device.",
    "11. The method of measuring latency according to claim 10, wherein the triggering an event further comprises one of: generating an event trigger signal; generating rotational motion; or transmitting electromagnetic radiation.",
    "12. The method of measuring latency according to claim 10, further comprising: emitting an electromagnetic radiation emission from the zero-latency indicator based on receiving the indication of the event at the zero-latency indicator.",
    "13. The method of measuring latency according to claim 12, wherein the emitting the electromagnetic radiation emission further comprises one of: emitting an infrared illumination source; emitting a low-level visible light illumination source; emitting a visible light source; or emitting a laser output of a laser level.",
    "14. The method of measuring latency according to claim 10, wherein the sending the indication of the event further comprises sending the indication through an electronics system to the multi-function display device.",
    "15. The method of measuring latency according to claim 10, wherein the displaying image further comprises one of: displaying the image on a graphical image display device that graphically outputs display images based on input received at a the electronics system; displaying the image on a warning/caution/advisory (WCA) indicator that is activated based on input received at the electronics system; or displaying the image on an artificial horizon display that corresponds to input received at the electronics system.",
    "16. The method of measuring latency according to claim 10, wherein the processing the series of images further comprises: detecting a threshold corresponding to the zero-latency indicator and a threshold corresponding to the image displayed on the multi-function display device; creating a plurality of binary waveforms based on detecting the threshold corresponding to the zero-latency indicator and a threshold-corresponding to the image displayed on the multi-function display device; detecting edges for each of the plurality of binary waveforms that measure one of rising and falling; and measuring a delta time between the zero-latency indicator and the image on the multi-function display device based on the detected edges for each of the plurality of binary waveforms.",
    "17. The method of measuring latency according to claim 10, wherein determining the time difference further comprises: determining a difference in a rate of angular change between the indication of the event by the zero-latency indicator and a rate of angular change indicated by the image on the multi-function display device.",
    "18. A method for measuring latency in electronic equipment, comprising: generating pulsed source signals at a sensor aperture and at a multi-functional display; capturing a source illumination and multi-functional display illumination directly and through a sensor path using a high frame rate camera; isolating a plurality of regions of interest for the source illumination and the multi-functional display illumination in video captured by the high frame rate camera; detecting peak thresholds in the regions of interest; creating a plurality of binary waveforms based on the detected peak thresholds; detecting a plurality of edges of one of the plurality of binary waveforms by measuring one of rising or falling edges; and measuring a delta time between the generated pulsed source signals and the plurality of edges providing a latency measurement.",
    "19. The method of claim 18, further comprising: generating the source illumination by emitting an electromagnetic radiation emission based on receiving the generated pulsed source signals; and generating the multi-functional display illumination by receiving an event signal based on the generated pulsed source signals."
  ],
  "description_excerpt": "The field of the embodiments presented herein is directed toward a latency measurement system using a camera to capture end-to-end latency on operator displays that processes a series of recorded images to extract timing measurements.\n\nMany graphical operator display systems have critical timeline/latency requirements in the field of avionics and vehicle system control. However, system latency measurements in response to meeting respective latency requirements are often difficult to verify and frequently involve internal trigger signals that only generally approximate end-to-end latency. Manual timing measurements of latency on an oscilloscope are prone to error and are awkward in obtaining statistical distributions of system latency.\n\nTypically there is no method to measure end-to-end system latency on a variety of systems without perturbing the actual system operation. In addition, when these measurements were possible, they often required significant manual operations that are prone to error. Difficulties in measuring system latencies are caused by the limited ability to take large numbers of sequential timing measurements to form statistical distributions, the automated processing of sequential timing measurement to eliminate manual measurement error, the challenge to provide a non-evasive end-to-end measurement, and the relative inflexibility to measure a variety of avionic and vehicular components within electronics systems. It is with respect to these and other considerations that the disclosure herein is presented.",
  "cpc": [
    "G09G 3/006",
    "G09G 2360/145",
    "G09G 2380/12",
    "H04N 17/04"
  ],
  "ipc": [
    "H04N 7/18",
    "G09G 3/00",
    "H04N 17/04"
  ],
  "assignees": [
    "Boeing Co"
  ],
  "inventors": [
    "William Brendan Blanton",
    "Robert Crenshaw Allen",
    "Thomas Alfred DuBois"
  ],
  "filing_date": "2012-09-25",
  "publication_date": "2016-12-06",
  "grant_date": "2016-12-06",
  "priority_date": "2012-09-25",
  "application_number": "US-201213626144-A",
  "family_id": "48985946",
  "cited_by_count": 4,
  "citations": [
    "US5521907A",
    "US6556540B1",
    "US6322216B1",
    "US20030021241A1",
    "US6697097B1",
    "US20030133031A1",
    "US20100166065A1",
    "US20110107220A1",
    "US7823001B2",
    "JP2005184749A",
    "US20050275831A1",
    "US20070183493A1",
    "US7680545B2",
    "US7693082B2",
    "US20070081094A1",
    "US7908507B2",
    "US20100141762A1",
    "US20100214238A1",
    "US20080307307A1",
    "US20080310676A1",
    "JP2009171334A",
    "US20090279611A1",
    "US20090310672A1",
    "US20100002893A1",
    "US20110109644A1",
    "US8838863B2",
    "US8334716B1",
    "US20100167713A1",
    "US20110219112A1",
    "US20130188544A1",
    "US20120105473A1",
    "US20120144409A1",
    "US8290526B2",
    "US20120281767A1",
    "US20120287289A1",
    "US20120287288A1",
    "US8911087B2",
    "US20130182104A1",
    "US8898687B2",
    "US8743020B1",
    "US20140036095A1",
    "US8838322B1",
    "US20140075030A1"
  ]
}