Speech endpointing

{
  "publication_number": "US10269341B2",
  "country": "US",
  "kind": "B2",
  "title": "Speech endpointing",
  "abstract": "Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for speech endpointing are described. In one aspect, a method includes the action of accessing voice query log data that includes voice queries spoken by a particular user. The actions further include based on the voice query log data that includes voice queries spoken by a particular user, determining a pause threshold from the voice query log data that includes voice queries spoken by the particular user. The actions further include receiving, from the particular user, an utterance. The actions further include determining that the particular user has stopped speaking for at least a period of time equal to the pause threshold. The actions further include based on determining that the particular user has stopped speaking for at least a period of time equal to the pause threshold, processing the utterance as a voice query.",
  "claims": [
    "1. A computer-implemented method comprising: accessing a collection of voice queries that were submitted by a particular user, wherein the collection of voice queries is accessed by a voice query processor of a speech recognition system that includes (i) the voice query processor, (ii) a microphone, (iii) an endpointing processor that is configured to endpoint audio data corresponding to an utterance by generating an endpoint signal designating the end of the utterance after speech is not detected for a default pause threshold, and (iv) an automated speech recognizer that is configured to perform automated speech recognition on endpointed audio data having endpoints marked by endpoint signals from the endpoint processor; determining, by the speech recognition system, a value for a variable pause threshold for the particular user based on durations of pauses between words of the voice queries in the collection of voice queries; generating, using the microphone of the speech recognition system, audio data corresponding to an utterance spoken by the particular user; determining, by the endpointing processor of the speech recognition system, that speech of the particular user has not been detected for at least a first period of time equal to the default pause threshold; after determining that speech of the particular user has not been detected for the period of time equal to the default pause threshold: based on the value of the variable pause threshold for the particular user being greater than the default pause threshold, including audio data for audio before the first period and audio after the first period in a single voice query by bypassing (i) endpointing of the audio data by the endpointing processor and (ii) performing automated speech recognition on the audio data by the automated speech recognizer; determining, by the endpointing processor of the speech recognition system, that speech of the particular user has not been detected for at least a second period of time equal to the value of the variable pause threshold determined for the particular user; based on determining that speech of the particular user has not been detected for at least the second period of time equal to the value of the variable pause threshold determined for the particular user, endpointing, by the endpointing processor of the speech recognition system, the audio data by designating a temporal location in the audio data at which to end automated speech recognition system processing for the single voice query; and in response to endpointing the audio data, performing, by the automated speech recognizer of the speech recognition system, automated speech recognition on a portion of the endpointed audio data that (i) includes audio data before the temporal location in the audio data and (ii) excludes the audio data after the temporal location in the audio data.",
    "2. The method of claim 1, comprising: determining, by the speech recognition system, a voice query length based on the voice queries in the collection of voice queries; and modifying, by the speech recognition system, the value for the variable pause threshold for the particular user based on the voice query length.",
    "3. The method of claim 2, wherein the voice query length is an average duration of the voice queries in the collection of voice queries.",
    "4. The method of claim 2, wherein the voice query length is an average number of words in the voice queries in the collection of voice queries.",
    "5. The method of claim 1, comprising: determining, by the speech recognition system, a quantity of voice queries based on quantities of the voice queries in the collection of voice queries that were spoken each day; and modifying, by the speech recognition system, the value for the variable pause threshold for the particular user based on the quantity of voice queries.",
    "6. The method of claim 5, wherein the quantity of voice queries is an average number of the voice queries in the collection of voice queries that were spoken each day.",
    "7. The method of claim 1, comprising: classifying, by the speech recognition system, the voice queries in the collection of voice queries as complete or incomplete; and modifying, by the speech recognition system, the value for the variable pause threshold for the particular user based on the classifications.",
    "8. The method of claim 1, comprising: rounding, by the speech recognition system, the value for the variable pause threshold for the particular user to a nearest duration among a set of predetermined durations.",
    "9. The method of claim 1, wherein the value for the variable pause threshold for the particular user is an average duration of pauses between words of the voice queries in the collection of voice queries.",
    "10. The method of claim 1, wherein: the speech recognition system includes a query length processor, and the value for the variable pause threshold for the particular user is determined by the query length processor of the speech recognition system.",
    "11. The method of claim 1, wherein the speech recognition system includes a natural language processing system.",
    "12. A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: accessing a collection of voice queries that were submitted by a particular user, wherein the collection of voice queries is accessed by a voice query processor of a speech recognition system that includes (i) the voice query processor, (ii) a microphone, (iii) an endpointing processor that is configured to endpoint audio data corresponding to an utterance by generating an endpoint signal designating the end of the utterance after speech is not detected for a default pause threshold, and (iv) an automated speech recognizer that is configured to perform automated speech recognition on endpointed audio data having endpoints marked by endpoint signals from the endpoint processor; determining, by the speech recognition system, a value for a variable pause threshold for the particular user based on durations of pauses between words of the voice queries in the collection of voice queries; generating, using the microphone of the speech recognition system, audio data corresponding to an utterance spoken by the particular user; determining, by the endpointing processor of the speech recognition system, that speech of the particular user has not been detected for at least a first period of time equal to the default pause threshold; after determining that speech of the particular user has not been detected for the period of time equal to the default pause threshold: based on the value of the variable pause threshold for the particular user being greater than the default pause threshold, including audio data for audio before the first period and audio after the first period in a single voice query by bypassing (i) endpointing of the audio data by the endpointing processor and (ii) performing automated speech recognition on the audio data by the automated speech recognizer; determining, by the endpointing processor of the speech recognition system, that speech of the particular user has not been detected for at least a second period of time equal to the value of the variable pause threshold determined for the particular user; based on determining that speech of the particular user has not been detected for at least the second period of time equal to the value of the variable pause threshold determined for the particular user, endpointing, by the endpointing processor of the speech recognition system, the audio data by designating a temporal location in the audio data at which to end automated speech recognition system processing for the single voice query; and in response to endpointing the audio data, performing, by the automated speech recognizer of the speech recognition system, automated speech recognition on a portion of the endpointed audio data that (i) includes audio data before the temporal location in the audio data and (ii) excludes the audio data after the temporal location in the audio data.",
    "13. The system of claim 12, wherein the operations further comprise: determining, by the speech recognition system, a voice query length based on the voice queries in the collection of voice queries; and modifying, by the speech recognition system, the value for the variable pause threshold for the particular user based on the voice query length.",
    "14. The system of claim 12, wherein the operations further comprise: determining, by the speech recognition system, a quantity of voice queries based on quantities of the voice queries in the collection of voice queries that were spoken each day; and modifying, by the speech recognition system, the value for the variable pause threshold for the particular user based on the quantity of voice queries.",
    "15. The system of claim 12, wherein the operations further comprise: classifying, by the speech recognition system, the voice queries in the collection of voice queries as complete or incomplete; and modifying, by the speech recognition system, the value for the variable pause threshold for the particular user based on the classifications.",
    "16. The system of claim 12, wherein the operations further comprise: rounding, by the speech recognition system, the value for the variable pause threshold for the particular user to a nearest duration among a set of predetermined durations.",
    "17. The system of claim 12, wherein the value for the variable pause threshold for the particular user is an average duration of pauses between words of the voice queries in the collection of voice queries.",
    "18. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising: accessing a collection of voice queries that were submitted by a particular user, wherein the collection of voice queries is accessed by a voice query processor of a speech recognition system that includes (i) the voice query processor, (ii) a microphone, (iii) an endpointing processor that is configured to endpoint audio data corresponding to an utterance by generating an endpoint signal designating the end of the utterance after speech is not detected for a default pause threshold, and (iv) an automated speech recognizer that is configured to perform automated speech recognition on endpointed audio data having endpoints marked by endpoint signals from the endpoint processor; determining, by the speech recognition system, a value for a variable pause threshold for the particular user based on durations of pauses between words of the voice queries in the collection of voice queries; generating, using the microphone of the speech recognition system, audio data corresponding to an utterance spoken by the particular user; determining, by the endpointing processor of the speech recognition system, that speech of the particular user has not been detected for at least a first period of time equal to the default pause threshold; after determining that speech of the particular user has not been detected for the period of time equal to the default pause threshold: based on the value of the variable pause threshold for the particular user being greater than the default pause threshold, including audio data for audio before the first period and audio after the first period in a single voice query by bypassing (i) endpointing of the audio data by the endpointing processor and (ii) performing automated speech recognition on the audio data by the automated speech recognizer; determining, by the endpointing processor of the speech recognition system, that speech of the particular user has not been detected for at least a second period of time equal to the value of the variable pause threshold determined for the particular user; based on determining that speech of the particular user has not been detected for at least the second period of time equal to the value of the variable pause threshold determined for the particular user, endpointing, by the endpointing processor of the speech recognition system, the audio data by designating a temporal location in the audio data at which to end automated speech recognition system processing for the single voice query; and in response to endpointing the audio data, performing, by the automated speech recognizer of the speech recognition system, automated speech recognition on a portion of the endpointed audio data that (i) includes audio data before the temporal location in the audio data and (ii) excludes the audio data after the temporal location in the audio data.",
    "19. The medium of claim 18, wherein the operations comprise: determining, by the speech recognition system, a voice query length based on the voice queries in the collection of voice queries; and modifying, by the speech recognition system, the value for the variable pause threshold for the particular user based on the voice query length.",
    "20. The medium of claim 19, wherein the voice query length is an average duration of the voice queries in the collection of voice queries.",
    "21. The medium of claim 19, wherein the voice query length is an average number of words in the voice queries in the collection of voice queries.",
    "22. The medium of claim 18, wherein the operations further comprise: determining, by the speech recognition system, a quantity of voice queries based on quantities of the voice queries in the collection of voice queries that were spoken each day; and modifying, by the speech recognition system, the value for the variable pause threshold for the particular user based on the quantity of voice queries."
  ],
  "description_excerpt": "This disclosure generally relates to speech recognition, and one particular implementation relates to endpointing speech.\n\nNatural language processing systems typically use endpointers to determine when a user has started and finished speaking. Some traditional endpointers evaluate the duration of pauses between words in determining when an utterance begins or ends. For instance, if a user says “what is for dinner,” a traditional endpointer may segment the voice input at the long pause, and may instruct the natural language processing system to attempt to process the incomplete phrase “what is,” instead of the complete phrase “what is for dinner.” If an endpointer designates an incorrect beginning or ending point for a voice input, the result of processing the voice input using the natural language processing system may be inaccurate or undesirable.\n\nDifferent users may have different comfort levels with using voice input on their mobile devices. Some users may use the voice input features frequently and be able to form voice queries without hesitating during speaking. Other users may not use the voice input features as often or quickly formulate queries without hesitating. The more experienced user may benefit from a system that begins to process the user's voice queries after measuring a short pause in the user's speech. The less experienced user may benefit form a system that begins to process the user's voice queries after measuring a longer pause in the user's speech to ensure the less experienced user has finished speaking.",
  "cpc": [
    "G10L 15/05",
    "G06F 16/685",
    "G06F 17/30746",
    "G10L 15/04",
    "G10L 15/065",
    "G10L 15/07",
    "G10L 15/22",
    "G10L 15/26",
    "G10L 2025/783",
    "G10L 25/78"
  ],
  "ipc": [
    "G06F 17/30",
    "G10L 15/05",
    "G10L 15/065",
    "G10L 15/26",
    "G10L 25/78",
    "G10L 15/04",
    "G10L 15/22"
  ],
  "assignees": [
    "Google LLC"
  ],
  "inventors": [
    "Siddhi Tadpatrikar",
    "Michael Buchanan",
    "Pravir Kumar Gupta"
  ],
  "filing_date": "2016-06-29",
  "publication_date": "2019-04-23",
  "grant_date": "2019-04-23",
  "priority_date": "2015-10-19",
  "application_number": "US-201615196663-A",
  "family_id": "56920617",
  "cited_by_count": 4,
  "citations": [
    "JPH01219893A",
    "JPH07104676A",
    "JPH08115093A",
    "JPH10254475A",
    "US20010056344A1",
    "US6321197B1",
    "US20010034601A1",
    "US6324509B1",
    "US6771887B1",
    "WO2001086633A1",
    "US8175876B2",
    "US7177810B2",
    "US20060149558A1",
    "US20050108011A1",
    "US7035807B1",
    "US7665024B1",
    "JP4433704B2",
    "JP2005017932A",
    "US20050171768A1",
    "US20060200350A1",
    "US20060235688A1",
    "JP2008508564A",
    "US8554564B2",
    "US8170875B2",
    "US8165880B2",
    "US20070100626A1",
    "US20070168890A1",
    "US20090149166A1",
    "US20100017209A1",
    "US20080235019A1",
    "US20090063150A1",
    "US20100292989A1",
    "US20100076761A1",
    "US20100131279A1",
    "US20100174533A1",
    "US20110105859A1",
    "US20100280827A1",
    "US20110153309A1",
    "US20110264447A1",
    "US8762150B2",
    "US20120089392A1",
    "US20170354363A1",
    "US20130124646A1",
    "US20150312246A1",
    "US8600746B1",
    "JP2014134675A",
    "US20140337370A1",
    "US9437186B1",
    "US20150106088A1",
    "WO2015073071A1",
    "US20150199966A1",
    "US20150206544A1",
    "US20160063992A1",
    "US20160351196A1",
    "US20160379632A1"
  ]
}