MLchartDataset catalogue

Patent · US10827024B1 · B1 · US

Realtime bandwidth-based communication for assistant systems

(11) Publication number
US10827024B1
(21) Application number
16/222,923
(22) Filing date
2018-12-17
(30) Priority date
2018-04-20
(43) Publication date
2020-11-03
(45) Date of grant
2020-11-03
(51) IPC
G06F 40/00; G06V 10/764; H04L 12/26; H04L 29/08
(52) CPC
  • G06V Image or video recognition or understanding: 10/82, 10/764, 20/10, 40/28
  • G06F Electric digital data processing: 16/176, 16/2255, 16/2365, 16/243, 16/24552, 16/24575, 16/24578, 16/248, 16/3322, 16/3323, 16/3329, 16/3344, 16/338, 16/90332, 16/90335, 16/9038, 16/904, 16/951, 16/9532, 16/9535, 18/2411, 21/6245, 2216/13, 3/011, 3/013, 3/017, 3/167, 40/205, 40/274, 40/295, 40/30, 40/40, 7/14, 8/31, 9/44505, 9/4451, 9/453
  • G06N Computing arrangements based on specific computational models: 20/00, 3/006, 3/0442, 3/045, 3/0464, 3/08, 3/09, 5/022, 5/027, 7/01
  • G06Q Information and communication technology [ICT] specially adapted for administrative, commercial, financial, managerial or supervisory purposes; systems or methods specially adapted for administrative, commercial, financial, managerial or supervisory purposes, not otherwise provided for: 10/00, 10/40, 10/42, 10/48
  • G10L Speech analysis techniques or speech synthesis; speech recognition; speech or voice processing techniques; speech or audio coding or decoding: 13/00, 13/04, 15/02, 15/063, 15/07, 15/16, 15/1815, 15/1822, 15/183, 15/187, 15/22, 15/26, 17/00, 17/06, 17/22, 2015/223, 2015/225
  • H04L Transmission of digital information, e.g. telegraphic communication: 12/2816, 41/20, 41/22, 43/0882, 43/0894, 5/02, 51/02, 51/046, 51/18, 51/216, 51/222, 51/52, 63/102, 67/10, 67/2828, 67/306, 67/53, 67/535, 67/5651, 67/75
  • H04W Wireless communication networks: 12/08
(73) Assignee
Facebook Inc
(72) Inventors
Jason Schissel; Aleksey Sergeyevich Fadeev
(54) Title
Realtime bandwidth-based communication for assistant systems
(57) Abstract

In one embodiment, a method includes, by one or more computing systems of a communication network, receiving, from a first client system associated with a first user during a communication session between the first client system and a second client system associated with a second user, a first user communication in a first modality, wherein the first user communication is to be sent via the network to the second client system. The method further includes determining an available bandwidth of the network with respect to the second client system, determining a second modality to send the first user communication to the second client system based on the available bandwidth, generating a second user communication in the second modality by converting the first user communication to the second modality, and sending, to the second client system the second user communication for presentation to the second user.

Full text
View on Google Patents

Claims (19)

  1. A method comprising, by one or more computing systems of a communication network: receiving, from a first client system associated with a first user during a communication session between the first client system and a second client system associated with a second user, a first user communication in a first modality, wherein the first user communication is to be sent via the communication network to the second client system; determining, by the one or more computing systems, an available bandwidth of the communication network with respect to the second client system; determining, by the one or more computing systems, a second modality in which to send the first user communication to the second client system, wherein the second modality is determined based on the determined available bandwidth of the communication network with respect to the second client system, and wherein the first and second modalities are different modalities; selecting, by the one or more computing systems, a location to generate a second user communication based on the determined available bandwidth, wherein the second user communication is generated at the selected location in the second modality by converting the first user communication to the second modality; and sending, to the second client system responsive to the first user communication, either the first user communication or the second user communication based on the selected location, for presentation, to the second user, in the second modality generated at the selected location.
  2. The method of claim 1, wherein the second client system converts the second user communication to the first modality.
  3. The method of claim 1, wherein the first modality comprises video, audio, or text.
  4. The method of claim 1, wherein the first user communication comprises a video clip, an audio clip, or a text string.
  5. The method of claim 1, wherein determining the second modality comprises: determining whether the available bandwidth is sufficient to send the first user communication to the second client system in the first modality; and when the available bandwidth is determined to be insufficient to send the first user communication to the second client system in the first modality, determining the second modality such that the available bandwidth of the communication network is sufficient to send the second user communication in the second modality.
  6. The method of claim 5, wherein the second user communication in the second modality is smaller in size than the first user communication in the first modality.
  7. The method of claim 5, wherein the first modality is video, and the available bandwidth is insufficient to send the first user communication as video, and (1) the second modality is audio when the available bandwidth is sufficient to send the first user communication as audio, or (2) the second modality is text when the available bandwidth is insufficient to send the first user communication as audio.
  8. The method of claim 1, wherein determining the second modality comprises: determining whether the available bandwidth is sufficient to send the first user communication to the second client system in a modality that represents more information than the first modality; and when the available bandwidth is determined to be sufficient to send the first user communication to the second client system in the modality that represents more information than the first modality, the second modality is the modality that represents more information than the first modality.
  9. The method of claim 8, wherein the second modality uses more bandwidth than the first modality, and wherein the available bandwidth of the communication network is sufficient to send the second user communication.
  10. The method of claim 8, wherein the first modality is text and the second modality is audio, and wherein converting the first user communication to the second modality comprises converting the first user communication from text to audio-encoded speech, wherein the second user communication comprises the audio-encoded speech.
  11. The method of claim 8, wherein the first modality is text and the second modality is video, and wherein converting the first user communication to the second modality comprises converting the first user communication from text to video, the video comprising audio-encoded speech and animation based on the text, wherein the second user communication comprises the video.
  12. The method of claim 11, wherein the animation comprises one or more avatars, emoji, or facial expressions based on the text.
  13. The method of claim 8, wherein the first modality is audio and the second modality is video, and wherein converting the first user communication to the second modality comprises converting the first user communication from audio to video comprising audio-encoded speech and animation based on the audio, wherein the second user communication comprises the video.
  14. The method of claim 1, wherein the available bandwidth comprises an estimate of an amount of bandwidth available on the communications network for use by the first client system.
  15. The method of claim 1, wherein the communication session is associated with a network interface that communicates with the communications network, and wherein the available bandwidth is determined based on one or more attributes of the network interface.
  16. The method of claim 1, wherein the available bandwidth is determined by sending a predetermined quantity of data via the communications network to a destination address, and measuring a rate at which the predetermined quantity of data is sent.
  17. The method of claim 1, wherein the one or more computing systems comprise the first client system, and wherein the second user communication is sent from the first client system to the second client system via the communication network.
  18. One or more computer-readable non-transitory storage media embodying software that is operable when executed to: receive, from a first client system associated with a first user during a communication session between the first client system and a second client system associated with a second user, a first user communication in a first modality, wherein the first user communication is to be sent via the communication network to the second client system; determine an available bandwidth of the communication network with respect to the second client system; determine a second modality in which to send the first user communication to the second client system, wherein the second modality is determined based on the determined available bandwidth of the communication network with respect to the second client system, and wherein the first and second modalities are different modalities; select a location to generate a second user communication based on the determined available bandwidth, wherein the second user communication is generated at the selected location in the second modality by converting the first user communication to the second modality; and send, to the second client system responsive to the first user communication, either the first user communication or the second user communication based on the selected location, for presentation, to the second user, in the second modality generated at the selected location.
  19. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to: receive, from a first client system associated with a first user during a communication session between the first client system and a second client system associated with a second user, a first user communication in a first modality, wherein the first user communication is to be sent via the communication network to the second client system; determine an available bandwidth of the communication network with respect to the second client system; determine a second modality in which to send the first user communication to the second client system, wherein the second modality is determined based on the determined available bandwidth of the communication network with respect to the second client system, and wherein the first and second modalities are different modalities; select a location to generate a second user communication based on the determined available bandwidth, wherein the second user communication is generated at the selected location in the second modality by converting the first user communication to the second modality; and send, to the second client system responsive to the first user communication, either the first user communication or the second user communication based on the selected location, for presentation, to the second user, in the second modality generated at the selected location.

Description

This application claims the benefit, under 35 U.S.C. § 119(e), of U.S. Provisional Patent Application No. 62/660,876, filed 20 Apr. 2018, which is incorporated herein by reference.

This disclosure generally relates to databases and file management within network environments, and in particular relates to hardware and software for smart assistant systems.

An assistant system can provide information or services on behalf of a user based on a combination of user input, location awareness, and the ability to access information from a variety of online sources (such as weather conditions, traffic congestion, news, stock prices, user schedules, retail prices, etc.). The user input may include text (e.g., online chat), especially in an instant messaging application or other applications, voice, images, motion, or a combination of them. The assistant system may perform concierge-type services (e.g., making dinner reservations, purchasing event tickets, making travel arrangements) or provide information based on the user input. The assistant system may also perform management or data-handling tasks based on online information and events without user initiation or interaction. Examples of those tasks that may be performed by an assistant system may include schedule management (e.g., sending an alert to a dinner date that a user is running late due to traffic conditions, update schedules for both parties, and change the restaurant reservation time). The assistant system may be enabled by the combination of computing devices, application programming interfaces (APIs), and the proliferation of applications on user devices.

Citations (70)

  • US20050144284A1
  • US8560564B1
  • US8027451B2
  • US7397912B2
  • US7158678B2
  • US20060294546A1
  • US7124123B1
  • US8677377B2
  • US20080240379A1
  • US20080095187A1
  • US20110161348A1
  • US20100199320A1
  • US10042032B2
  • US20120265528A1
  • US9858925B2
  • US20110153863A1
  • US20130275138A1
  • US20130275164A1
  • US10276170B2
  • US8935192B1
  • WO2012116241A2
  • US20120246191A1
  • EP2530870A1
  • US10241752B2
  • US9154739B1
  • US9418658B1
  • US20130268839A1
  • US9390724B2
  • US9299059B1
  • US9747895B1
  • US9576574B2
  • US8983383B1
  • US20160378861A1
  • US20140164506A1
  • US10199051B2
  • US9659577B1
  • US9875741B2
  • US9304736B1
  • US9472206B2
  • US9338242B1
  • US10134395B2
  • US9479931B2
  • US20150179168A1
  • US9875233B1
  • US20150373565A1
  • US9338493B2
  • US20160378849A1
  • US20160225370A1
  • US9886953B2
  • US20160328096A1
  • WO2016195739A1
  • US9792281B2
  • EP3122001A1
  • WO2017053208A1
  • US20170091168A1
  • US20170132019A1
  • WO2017116488A1
  • US20160255082A1
  • US9990591B2
  • US20170353469A1
  • US20170359707A1
  • AU2017203668A1
  • US20180018562A1
  • US20180018987A1
  • US20180096072A1
  • US20180096071A1
  • US20180107917A1
  • US20180189629A1
  • US9865260B1
  • US20190080698A1
Record as JSON 
{
  "publication_number": "US10827024B1",
  "country": "US",
  "kind": "B1",
  "title": "Realtime bandwidth-based communication for assistant systems",
  "abstract": "In one embodiment, a method includes, by one or more computing systems of a communication network, receiving, from a first client system associated with a first user during a communication session between the first client system and a second client system associated with a second user, a first user communication in a first modality, wherein the first user communication is to be sent via the network to the second client system. The method further includes determining an available bandwidth of the network with respect to the second client system, determining a second modality to send the first user communication to the second client system based on the available bandwidth, generating a second user communication in the second modality by converting the first user communication to the second modality, and sending, to the second client system the second user communication for presentation to the second user.",
  "claims": [
    "1. A method comprising, by one or more computing systems of a communication network: receiving, from a first client system associated with a first user during a communication session between the first client system and a second client system associated with a second user, a first user communication in a first modality, wherein the first user communication is to be sent via the communication network to the second client system; determining, by the one or more computing systems, an available bandwidth of the communication network with respect to the second client system; determining, by the one or more computing systems, a second modality in which to send the first user communication to the second client system, wherein the second modality is determined based on the determined available bandwidth of the communication network with respect to the second client system, and wherein the first and second modalities are different modalities; selecting, by the one or more computing systems, a location to generate a second user communication based on the determined available bandwidth, wherein the second user communication is generated at the selected location in the second modality by converting the first user communication to the second modality; and sending, to the second client system responsive to the first user communication, either the first user communication or the second user communication based on the selected location, for presentation, to the second user, in the second modality generated at the selected location.",
    "2. The method of claim 1, wherein the second client system converts the second user communication to the first modality.",
    "3. The method of claim 1, wherein the first modality comprises video, audio, or text.",
    "4. The method of claim 1, wherein the first user communication comprises a video clip, an audio clip, or a text string.",
    "5. The method of claim 1, wherein determining the second modality comprises: determining whether the available bandwidth is sufficient to send the first user communication to the second client system in the first modality; and when the available bandwidth is determined to be insufficient to send the first user communication to the second client system in the first modality, determining the second modality such that the available bandwidth of the communication network is sufficient to send the second user communication in the second modality.",
    "6. The method of claim 5, wherein the second user communication in the second modality is smaller in size than the first user communication in the first modality.",
    "7. The method of claim 5, wherein the first modality is video, and the available bandwidth is insufficient to send the first user communication as video, and (1) the second modality is audio when the available bandwidth is sufficient to send the first user communication as audio, or (2) the second modality is text when the available bandwidth is insufficient to send the first user communication as audio.",
    "8. The method of claim 1, wherein determining the second modality comprises: determining whether the available bandwidth is sufficient to send the first user communication to the second client system in a modality that represents more information than the first modality; and when the available bandwidth is determined to be sufficient to send the first user communication to the second client system in the modality that represents more information than the first modality, the second modality is the modality that represents more information than the first modality.",
    "9. The method of claim 8, wherein the second modality uses more bandwidth than the first modality, and wherein the available bandwidth of the communication network is sufficient to send the second user communication.",
    "10. The method of claim 8, wherein the first modality is text and the second modality is audio, and wherein converting the first user communication to the second modality comprises converting the first user communication from text to audio-encoded speech, wherein the second user communication comprises the audio-encoded speech.",
    "11. The method of claim 8, wherein the first modality is text and the second modality is video, and wherein converting the first user communication to the second modality comprises converting the first user communication from text to video, the video comprising audio-encoded speech and animation based on the text, wherein the second user communication comprises the video.",
    "12. The method of claim 11, wherein the animation comprises one or more avatars, emoji, or facial expressions based on the text.",
    "13. The method of claim 8, wherein the first modality is audio and the second modality is video, and wherein converting the first user communication to the second modality comprises converting the first user communication from audio to video comprising audio-encoded speech and animation based on the audio, wherein the second user communication comprises the video.",
    "14. The method of claim 1, wherein the available bandwidth comprises an estimate of an amount of bandwidth available on the communications network for use by the first client system.",
    "15. The method of claim 1, wherein the communication session is associated with a network interface that communicates with the communications network, and wherein the available bandwidth is determined based on one or more attributes of the network interface.",
    "16. The method of claim 1, wherein the available bandwidth is determined by sending a predetermined quantity of data via the communications network to a destination address, and measuring a rate at which the predetermined quantity of data is sent.",
    "17. The method of claim 1, wherein the one or more computing systems comprise the first client system, and wherein the second user communication is sent from the first client system to the second client system via the communication network.",
    "18. One or more computer-readable non-transitory storage media embodying software that is operable when executed to: receive, from a first client system associated with a first user during a communication session between the first client system and a second client system associated with a second user, a first user communication in a first modality, wherein the first user communication is to be sent via the communication network to the second client system; determine an available bandwidth of the communication network with respect to the second client system; determine a second modality in which to send the first user communication to the second client system, wherein the second modality is determined based on the determined available bandwidth of the communication network with respect to the second client system, and wherein the first and second modalities are different modalities; select a location to generate a second user communication based on the determined available bandwidth, wherein the second user communication is generated at the selected location in the second modality by converting the first user communication to the second modality; and send, to the second client system responsive to the first user communication, either the first user communication or the second user communication based on the selected location, for presentation, to the second user, in the second modality generated at the selected location.",
    "19. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to: receive, from a first client system associated with a first user during a communication session between the first client system and a second client system associated with a second user, a first user communication in a first modality, wherein the first user communication is to be sent via the communication network to the second client system; determine an available bandwidth of the communication network with respect to the second client system; determine a second modality in which to send the first user communication to the second client system, wherein the second modality is determined based on the determined available bandwidth of the communication network with respect to the second client system, and wherein the first and second modalities are different modalities; select a location to generate a second user communication based on the determined available bandwidth, wherein the second user communication is generated at the selected location in the second modality by converting the first user communication to the second modality; and send, to the second client system responsive to the first user communication, either the first user communication or the second user communication based on the selected location, for presentation, to the second user, in the second modality generated at the selected location."
  ],
  "description_excerpt": "This application claims the benefit, under 35 U.S.C. § 119(e), of U.S. Provisional Patent Application No. 62/660,876, filed 20 Apr. 2018, which is incorporated herein by reference.\n\nThis disclosure generally relates to databases and file management within network environments, and in particular relates to hardware and software for smart assistant systems.\n\nAn assistant system can provide information or services on behalf of a user based on a combination of user input, location awareness, and the ability to access information from a variety of online sources (such as weather conditions, traffic congestion, news, stock prices, user schedules, retail prices, etc.). The user input may include text (e.g., online chat), especially in an instant messaging application or other applications, voice, images, motion, or a combination of them. The assistant system may perform concierge-type services (e.g., making dinner reservations, purchasing event tickets, making travel arrangements) or provide information based on the user input. The assistant system may also perform management or data-handling tasks based on online information and events without user initiation or interaction. Examples of those tasks that may be performed by an assistant system may include schedule management (e.g., sending an alert to a dinner date that a user is running late due to traffic conditions, update schedules for both parties, and change the restaurant reservation time). The assistant system may be enabled by the combination of computing devices, application programming interfaces (APIs), and the proliferation of applications on user devices.",
  "cpc": [
    "G06V 10/82",
    "G06F 16/176",
    "G06F 16/2255",
    "G06F 16/2365",
    "G06F 16/243",
    "G06F 16/24552",
    "G06F 16/24575",
    "G06F 16/24578",
    "G06F 16/248",
    "G06F 16/3322",
    "G06F 16/3323",
    "G06F 16/3329",
    "G06F 16/3344",
    "G06F 16/338",
    "G06F 16/90332",
    "G06F 16/90335",
    "G06F 16/9038",
    "G06F 16/904",
    "G06F 16/951",
    "G06F 16/9532",
    "G06F 16/9535",
    "G06F 18/2411",
    "G06F 21/6245",
    "G06F 2216/13",
    "G06F 3/011",
    "G06F 3/013",
    "G06F 3/017",
    "G06F 3/167",
    "G06F 40/205",
    "G06F 40/274",
    "G06F 40/295",
    "G06F 40/30",
    "G06F 40/40",
    "G06F 7/14",
    "G06F 8/31",
    "G06F 9/44505",
    "G06F 9/4451",
    "G06F 9/453",
    "G06N 20/00",
    "G06N 3/006",
    "G06N 3/0442",
    "G06N 3/045",
    "G06N 3/0464",
    "G06N 3/08",
    "G06N 3/09",
    "G06N 5/022",
    "G06N 5/027",
    "G06N 7/01",
    "G06Q 10/00",
    "G06Q 10/40",
    "G06Q 10/42",
    "G06Q 10/48",
    "G06V 10/764",
    "G06V 20/10",
    "G06V 40/28",
    "G10L 13/00",
    "G10L 13/04",
    "G10L 15/02",
    "G10L 15/063",
    "G10L 15/07",
    "G10L 15/16",
    "G10L 15/1815",
    "G10L 15/1822",
    "G10L 15/183",
    "G10L 15/187",
    "G10L 15/22",
    "G10L 15/26",
    "G10L 17/00",
    "G10L 17/06",
    "G10L 17/22",
    "G10L 2015/223",
    "G10L 2015/225",
    "H04L 12/2816",
    "H04L 41/20",
    "H04L 41/22",
    "H04L 43/0882",
    "H04L 43/0894",
    "H04L 5/02",
    "H04L 51/02",
    "H04L 51/046",
    "H04L 51/18",
    "H04L 51/216",
    "H04L 51/222",
    "H04L 51/52",
    "H04L 63/102",
    "H04L 67/10",
    "H04L 67/2828",
    "H04L 67/306",
    "H04L 67/53",
    "H04L 67/535",
    "H04L 67/5651",
    "H04L 67/75",
    "H04W 12/08"
  ],
  "ipc": [
    "G06F 40/00",
    "G06V 10/764",
    "H04L 12/26",
    "H04L 29/08"
  ],
  "assignees": [
    "Facebook Inc"
  ],
  "inventors": [
    "Jason Schissel",
    "Aleksey Sergeyevich Fadeev"
  ],
  "filing_date": "2018-12-17",
  "publication_date": "2020-11-03",
  "grant_date": "2020-11-03",
  "priority_date": "2018-04-20",
  "application_number": "US-201816222923-A",
  "family_id": "68235956",
  "cited_by_count": 29,
  "citations": [
    "US20050144284A1",
    "US8560564B1",
    "US8027451B2",
    "US7397912B2",
    "US7158678B2",
    "US20060294546A1",
    "US7124123B1",
    "US8677377B2",
    "US20080240379A1",
    "US20080095187A1",
    "US20110161348A1",
    "US20100199320A1",
    "US10042032B2",
    "US20120265528A1",
    "US9858925B2",
    "US20110153863A1",
    "US20130275138A1",
    "US20130275164A1",
    "US10276170B2",
    "US8935192B1",
    "WO2012116241A2",
    "US20120246191A1",
    "EP2530870A1",
    "US10241752B2",
    "US9154739B1",
    "US9418658B1",
    "US20130268839A1",
    "US9390724B2",
    "US9299059B1",
    "US9747895B1",
    "US9576574B2",
    "US8983383B1",
    "US20160378861A1",
    "US20140164506A1",
    "US10199051B2",
    "US9659577B1",
    "US9875741B2",
    "US9304736B1",
    "US9472206B2",
    "US9338242B1",
    "US10134395B2",
    "US9479931B2",
    "US20150179168A1",
    "US9875233B1",
    "US20150373565A1",
    "US9338493B2",
    "US20160378849A1",
    "US20160225370A1",
    "US9886953B2",
    "US20160328096A1",
    "WO2016195739A1",
    "US9792281B2",
    "EP3122001A1",
    "WO2017053208A1",
    "US20170091168A1",
    "US20170132019A1",
    "WO2017116488A1",
    "US20160255082A1",
    "US9990591B2",
    "US20170353469A1",
    "US20170359707A1",
    "AU2017203668A1",
    "US20180018562A1",
    "US20180018987A1",
    "US20180096072A1",
    "US20180096071A1",
    "US20180107917A1",
    "US20180189629A1",
    "US9865260B1",
    "US20190080698A1"
  ]
}

Record 1,912 of 8,000 in Patents full text (MLC-0201). Request the full dataset.