Electronic device, mobile electronic device, and display device, which are controlled on basis of user gaze and input

{
  "publication_number": "US2019324279A1",
  "country": "US",
  "kind": "A1",
  "title": "Electronic device, mobile electronic device, and display device, which are controlled on basis of user gaze and input",
  "abstract": "An electronic device is disclosed. The electronic device comprises a first camera and a second camera, a microphone, a display, and a processor electrically connected to the first camera, the second camera, the microphone, and the display, wherein the processor can be set to display, on the display, a user interface (UI) including a plurality of objects, acquire user gaze information from the first camera, activate, among the plurality of objects, a first object corresponding to the gaze information, determine at least one method of input, corresponding to a type of the activated first object, between a gesture input acquired from the second camera and a voice input acquired by the microphone, and execute the function corresponding to the input for the first object While an activated state of the first object is maintained, if the input of the determined method is applicable to the first object. In addition, various embodiments identified through the specification are possible.",
  "claims": [
    "1. An electronic device comprising: a first camera and a second camera; a microphone; a display; and a processor electrically connected to the first camera, the second camera, the microphone, and the display, wherein the processor is configured to: display a user interface (UI) including a plurality of objects on the display, obtain gaze information of a user from the first camera, activate a first object corresponding to the gaze information among the plurality of objects, determine at least one input method corresponding to a type of the activated first object among a gesture input obtained from the second camera and a speech input obtained by the microphone, and perform a function corresponding to an input on the first object while an activated state of the first object is maintained, when the determined at least one input method is the input applicable to the first object. 2. The electronic device of claim 1, wherein the processor is configured to activate the first object when a gaze of the user is located at a position where the first object is displayed. 3. The electronic device of claim 1, wherein the processor is configured to activate the first object when the user's gaze is kept on a position where the first object is displayed for a predetermined time or more. 4. The electronic device of claim 1, wherein the processor is configured to activate the first object when a gaze of the user is located at a position where the first object is displayed, and maintain the activated state of the first object while the input by the determined at least one input method is obtained. 5. The electronic device of claim 4, wherein the processor is configured to, when the gaze of the user is located at a second object different from the first object among the plurality of objects while the activated state of the first object is maintained and the input is an input applicable to the second object, perform a function corresponding to the input on the second object. 6. The electronic device of claim 1, wherein the processor is configured to determine the at least one input method further based on at least one of the gesture input and the speech input. 7. The electronic device of claim 1, wherein the processor is configured to perform a function corresponding to the gesture input on the first object, when the determined at least one input method is the gesture input and a gesture by the gesture input passes by a gaze of the user. 8. A mobile electronic device coupled to a head mounted display (HMD) including a first camera and a microphone, comprising: a second camera; a display; and a processor functionally connected to the first camera and the microphone and electrically connected to the second camera and the display, wherein the processor is configured to: display a user interface (UI) including a plurality of objects on the display, obtain gaze information of a user from the first camera, activate a first object corresponding to the gaze information among the plurality of objects, determine at least one input method corresponding to a type of the activated first object among a gesture input obtained from the second camera and a speech input obtained by the microphone, and perform a function corresponding to an input on the first object while an activated state of the first object is maintained, when the determined at least one input method is the input applicable to the first object. 9. The mobile electronic device of claim 8, wherein the processor is configured to activate the first object when a gaze of the user is located at a position where the first object is displayed. 10. The mobile electronic device of claim 8, wherein the processor is configured to activate the first object when the user's gaze is kept on a position where the first object is displayed for a predetermined time or more. 11. The mobile electronic device of claim 8, wherein the processor is configured to activate the first object when a gaze of the user is located at a position where the first object is displayed, and maintain the activated state of the first object while the input by the determined at least one input method is obtained. 12. The mobile electronic device of claim 11, wherein the processor is configured to, when a gaze of the user is located at a second object different from the first object among the plurality of objects while the activated state of the first object is maintained, and the input is an input applicable to the second object, perform a function corresponding to the input on the second object. 13. The mobile electronic device of claim 8, wherein the processor is configured to determine the at least one input method further based on at least one of the gesture input and the speech input. 14. The mobile electronic device of claim 8, wherein the processor is configured to determine the at least one input method to be the gesture input, when a noise above a reference value occurs in the speech input. 15. The mobile electronic device of claim 8, wherein the processor is configured to perform a function corresponding to the gesture input on the first object, when the determined at least one input method is the gesture input and a gesture by the gesture input passes by a gaze of the user."
  ],
  "description_excerpt": "Embodiments of the disclosure relate to an electronic device, a mobile electronic device, and a display device which are controlled based on a user's gaze and input.\n\nWith the development of new intelligent devices such as wearable products, various input methods for devices have also been developed. Especially, as the artificial intelligence begins to be attracted to attention, speech input is attracting attention as an interface to replace touch input, and input methods utilizing an eyegazing utilizing human gaze and motion detection recognizing a gesture have been also developed.\n\nIn recent years, an electronic device using a combination of two or more input methods such as gaze information, speech input, or gesture input has been proposed, but such an electronic device merely uses multiple input methods at the same time, and therefore, there is a problem in that the amount of computation of the processor is increased, which is inefficient.\n\nEmbodiments disclosed in the disclosure may provide an electronic device more efficiently using a user's gaze, a speech input, and a gesture input by determining an input method according to a type of an object of a UI, to solve the above problems and the problems raised in the disclosure.",
  "cpc": [
    "G02B 27/0172",
    "G02B 2027/0138",
    "G02B 2027/014",
    "G02B 2027/0187",
    "G02B 27/0101",
    "G02B 27/0176",
    "G06F 3/005",
    "G06F 3/013",
    "G06F 3/017",
    "G06F 3/0484",
    "G06F 3/04842",
    "G06F 3/16",
    "G06F 3/167",
    "H04N 21/4126",
    "H04N 21/4223",
    "H04N 21/44218",
    "H04R 2499/11"
  ],
  "ipc": [
    "G02B 27/01",
    "G06F 3/01",
    "G06F 3/16",
    "H04N 21/41"
  ],
  "assignees": [
    "Samsung Electronics Co Ltd"
  ],
  "inventors": [
    "Sang Woong Hwang",
    "Young Ah SEONG",
    "Say Jang",
    "Seung Hwan Choi"
  ],
  "filing_date": "2017-12-18",
  "publication_date": "2019-10-24",
  "priority_date": "2016-12-19",
  "application_number": "US-201716465239-A",
  "family_id": "62626810",
  "cited_by_count": 8
}