Detection of user interface controls via invariance guided sub-control learning

{
  "publication_number": "US11604663B2",
  "country": "US",
  "kind": "B2",
  "title": "Detection of user interface controls via invariance guided sub-control learning",
  "abstract": "Computerized detection of one or more user interface objects is performed by processing an image file containing one or more user interface objects of a user interface generated by an application program. Sub-control objects can be detected in the image file, where each sub-control object can form a portion of a user interface object that receives user input. Extraneous sub-control objects can be detected. Sub-control objects that overlap with or that are within a predetermined vicinity of an identified set of sub-control objects can be removed. Sub-control objects in the identified set of sub-control objects can be correlated to combine one or more of the sub-control objects in the identified set of sub-control objects to generate control objects that correspond to certain of the user interface objects of the user interface generated by the application program.",
  "claims": [
    "1. A computer-implemented method for detecting one or more user interface objects contained in a screen image of a user interface generated by an application program, the method comprising: receiving an image file containing the one or more user interface objects of the user interface generated by the application program; identifying a set of sub-control objects in the image file, each of the sub-control objects forming a portion of a user interface object of the one or more user interface objects that receives user input; and correlating the sub-control objects in the identified set of sub-control objects to combine one or more of the sub-control objects in the identified set of sub-control objects to form one or more controls that correspond to certain one or more of the one or more user interface objects, wherein the correlating the sub-control objects includes at least: imposing a distance condition to distinguish a sub-control object that is closer in distance to a sub-control object under consideration than another sub-control object; and matching the sub-control object under consideration to the distinguished sub-control object based on at least the distance condition; combining each set of the corresponding matched sub-control objects to generate a corresponding combined control; generating a confidence score for each of the combined controls; determining whether any of the combined controls has a confidence score below a threshold level; and removing any of the combined controls that has a confidence score below the threshold level to generate a final set of controls that corresponds to user interface objects of the user interface generated by the application program.",
    "2. The computer-implemented method of claim 1, wherein the correlating the sub-control objects includes at least imposing an alignment condition to align at least two of the sub-control objects that correspond to a user interface object, and wherein the matching the sub-control object under consideration to the distinguished sub-control object is based on at least the distance condition and the alignment condition.",
    "3. The computer-implemented method of claim 1, wherein the image file contains an image representation of the one or more user interface objects of the user interface generated by the application program; and wherein the detecting of the sub-control objects in the image file comprises detecting the sub-control objects from the image representation of the sub-control objects.",
    "4. The computer-implemented method of claim 1, wherein the method comprises: detecting one or more of the sub-control objects that are extraneous because they overlap with or that are within a predetermined vicinity of the identified set of sub-control objects; and removing the one or more extraneous sub-control objects that are detected as being extraneous.",
    "5. The computer-implemented method of claim 4, wherein the removing extraneous sub-control objects comprises, for at least one given sub-control object in the identified set of sub-control objects: detecting one or more of the sub-control objects that overlap with the given sub-control object in the identified set of sub-control objects; and removing the one or more detected sub-control objects that overlap with the sub-control object in the identified set of sub-control objects.",
    "6. The computer-implemented method of claim 1, wherein the removing extraneous sub-control objects comprises, for at least one given sub-control object in the identified set of sub-control objects: retrieving a confidence score associated with the given sub-control object in the identified set of sub-control objects; and removing the given sub-control object from the identified set of sub-control objects if the confidence score therefor is below a first threshold score.",
    "7. The computer-implemented method of claim 1, wherein removing extraneous sub-control objects comprises, for at least one given sub-control object in the identified set of sub-control objects: identifying one or more of the sub-control objects that overlap with the given sub-control object in the identified set of sub-control objects; removing the identified sub-control objects that overlap with the given sub-control object in the identified set of sub-control objects; determining a confidence score associated with the given sub-control object in the identified set of sub-control objects; and removing the given sub-control object from the identified set of the sub-control objects if the confidence score therefor is below a first threshold score.",
    "8. The computer-implemented method of claim 7, wherein the correlating the sub-control objects includes at least imposing an alignment condition to align at least two of the sub-control objects that correspond to a user interface object, and wherein the matching the sub-control object under consideration to the distinguished sub-control object is based on at least the distance condition and the alignment condition.",
    "9. The computer-implemented method of claim 8, wherein the image file contains an image representation of the one or more user interface objects of the user interface generated by the application program; and wherein the detecting of the sub-control objects in the image file comprises detecting the sub-control objects from the image representation of the sub-control objects.",
    "10. The computer-implemented method of claim 1, wherein the correlating sub-control objects includes at least: accessing a correlation map that provides a mapping between user interface objects of the user interface and sub-control objects associated with each user interface object.",
    "11. The computer-implemented method of claim 1, wherein the generating of a confidence score for each of the combined controls comprises generating the confidence score for each of the combined controls as a function of confidence scores associated with each of the sub-control objects that form the corresponding combined control.",
    "12. A computer-implemented method for detecting one or more user interface objects contained in a screen image of a user interface generated by an application program, the method comprising: receiving an image file containing one or more user interface objects of a user interface generated by an application program; identifying a set of sub-control objects in the image file, each of the sub-control objects in the identified set of sub-control objects forming a portion of a user interface object of the one or more user interface objects that receives user input; and correlating the sub-control objects in the identified set of sub-control objects to combine one or more of the sub-control objects in the identified set of sub-control objects to form one or more controls that correspond to certain one or more user interface objects, wherein the correlating the sub-control objects includes at least: imposing an alignment condition to align at least two of the sub-control objects that correspond to a user interface object; imposing a confidence value on each of the sub-control objects and removing those of the sub-control objects having a confidence value that falls below a threshold level; and matching the sub-control object under consideration to another sub-control object of the sub-control objects based on at least the alignment condition.",
    "13. The computer-implemented method of claim 12, wherein the method comprises: combining each set of corresponding matched sub-control objects to generate a corresponding combined control; generating a confidence score for each of the combined controls; and removing at least one of the combined controls that has a confidence score below a threshold level to generate a final set of controls that corresponds to user interface objects of the user interface generated by the application program.",
    "14. The computer-implemented method of claim 13, wherein the generating of a confidence score for each of the combined controls comprises generating the confidence score for each of the combined controls as a function of confidence scores associated with each of the sub-control objects that form the corresponding combined control.",
    "15. The computer-implemented method of claim 12, wherein the method comprises: detecting one or more of the sub-control objects that are extraneous because they overlap with or that are within a predetermined vicinity of the identified set of sub-control objects; and removing the one or more extraneous sub-control objects that are detected as being extraneous.",
    "16. The computer-implemented method of claim 12, wherein the correlating the sub-control objects includes at least: determining a distance condition to distinguish a sub-control object that is closer in distance to a sub-control object under consideration to than one or more other of the sub-control objects, and wherein the matching the sub-control object under consideration to the another sub-control object is based on at least the distance condition and the alignment condition.",
    "17. The computer-implemented method of claim 12, wherein the image file contains an image representation of the one or more user interface objects of the user interface generated by the application program; and wherein the detecting of the sub-control objects in the image file comprises detecting the sub-control objects from the image representation of the sub-control objects.",
    "18. The computer-implemented method of claim 12, wherein removing extraneous sub-control objects comprises, for at least one sub-control object in the identified set of sub-control objects: identifying sub-control objects that overlap with the sub-control object in the identified set of sub-control objects; removing sub-control objects that overlap with the sub-control object in the identified set of sub-control objects; and retrieving a confidence score associated with the sub-control object in the identified set of sub-control objects and removing sub-control objects having a confidence score below a first threshold score.",
    "19. The computer-implemented method of claim 12, wherein correlating sub-control objects includes at least: accessing a correlation map that provides a mapping between user interface objects of the user interface and sub-control objects associated with each user interface object.",
    "20. A computer system, comprising: data storage having stored thereupon a plurality of image files, each image file containing one or more user interface objects of a user interface generated by an application program; and a processor, programmed with instructions that cause the processor to detect one or more user interface objects contained in a screen image of a user interface generated by an application program by way of invariance guided sub-control learning, the instructions when executed by the processor operate to at least: receive an image file containing the one or more user interface objects of the user interface generated by the application program; identify a set of sub-control objects in the image file, each of the sub-control objects forming a portion of a user interface object of the one or more user interface objects that receives user input; and correlate the sub-control objects in the identified set of sub-control objects to combine one or more of the sub-control objects in the identified set of sub-control objects to form one or more controls that correspond to certain one or more of the one or more user interface objects, wherein the correlating the sub-control objects includes at least: impose a distance condition to cause a sub-control object that is closer in distance to a sub-control object under consideration than another of the sub-control objects; and match the sub-control object under consideration to the another of the sub-control object based on at least the distance condition; combine each set of corresponding matched sub-control objects to generate a corresponding combined control; generate a confidence score for each of the combined controls; and remove at least one of the combined controls that has a confidence score below a threshold level to generate a final set of controls that corresponds to user interface objects of the user interface generated by the application program.",
    "21. A tangible storage medium, having stored thereupon computer program code for execution on a computer system, the computer program code executing on a server processor to cause the computer system to perform a computer-implemented method for detecting one or more user interface objects contained in a screen image of a user interface generated by an application program, the computer-implemented method comprising: receiving an image file containing one or more user interface objects of a user interface generated by an application program; identifying a set of sub-control objects in the image file, each of the sub-control objects in the identified set of sub-control objects forming a portion of a user interface object of the one or more user interface objects that receives user input; and correlating the sub-control objects in the identified set of sub-control objects to combine one or more of the sub-control objects in the identified set of sub-control objects to form one or more controls that correspond to certain one or more user interface objects, wherein the correlating the sub-control objects includes at least: imposing an alignment condition to align at least two of the sub-control objects that correspond to a user interface object; and matching the sub-control object under consideration to the another sub-control object based on at least the alignment condition.",
    "22. A tangible storage medium of claim 21, wherein the computer-implemented method comprises: imposing a confidence value on each of the sub-control objects and removing those of the sub-control objects having a confidence value that falls below a threshold level."
  ],
  "description_excerpt": "This disclosure relates generally to the field of data processing systems and more particularly to detection of objects in images.\n\nRobotic process automation (RPA) is the application of technology that allows workers in an organization to configure computer software, known as a “robot” or “bot” to capture, interpret, and execute actions or commands in existing third-party applications for processing a transaction, manipulating data, triggering responses and communicating with other digital systems. The conventional RPA systems employ the robots or bots to interpret the user interface of such third-party applications and to execute actions or commands as a human worker or user would.\n\nIn particular, as the demand for automation increases, it is imperative to recognize user interface (UI) controls in legacy application programs which do not provide programmatic access, in order to automate usage of such applications. For websites, the code is available in one form or another so detection of controls and their type on the website is relatively straightforward. However, many licensed applications do not allow access to their code. Moreover, in certain situations, applications may be used by an automation user by way of remote desktop type software where only the screen image is available to the user. Automated detection of UI controls on such applications for automation is a challenge.\n\nIn such scenarios, since the application cannot provide information about the location and property of the UI controls, techniques to infer such information from image or video screenshots of the application are sometimes utilized.",
  "cpc": [
    "G06F 9/451",
    "G06F 3/0481",
    "G06F 3/04855",
    "G06N 20/10",
    "G06N 3/045",
    "G06N 3/0464",
    "G06N 3/082",
    "G06N 3/09"
  ],
  "ipc": [
    "G06F 3/0481",
    "G06F 9/451"
  ],
  "assignees": [
    "Automation Anywhere Inc"
  ],
  "inventors": [
    "Sudhir Kumar Singh",
    "Virinchipuram J Anand"
  ],
  "filing_date": "2021-11-16",
  "publication_date": "2023-03-14",
  "grant_date": "2023-03-14",
  "priority_date": "2020-02-21",
  "application_number": "US-202117528119-A",
  "family_id": "78703517",
  "cited_by_count": 4,
  "citations": [
    "US20040153649A1",
    "US5983001A",
    "US9444844B2",
    "US5949999A",
    "US6133917A",
    "US6496979B1",
    "US6226407B1",
    "US6427234B1",
    "US6389592B1",
    "US6473794B1",
    "US6957186B1",
    "US20070156677A1",
    "US6704873B1",
    "US6640244B1",
    "US20050257214A1",
    "US6954747B1",
    "US7571427B2",
    "US20040210885A1",
    "US7441007B1",
    "US20030033590A1",
    "US20030101245A1",
    "US20030159089A1",
    "US6898764B2",
    "US7246128B2",
    "US20080027769A1",
    "US7091898B2",
    "US20040083472A1",
    "US20050234699A1",
    "US20040172526A1",
    "US7846023B2",
    "US20040243994A1",
    "US7805710B2",
    "US20070112574A1",
    "US7568109B2",
    "US20050188357A1",
    "US20050204343A1",
    "US7398469B2",
    "US7810070B2",
    "US7765525B1",
    "US20060095276A1",
    "US20060150188A1",
    "US20090313229A1",
    "US7805317B2",
    "US8464240B2",
    "US20090103769A1",
    "US9104294B2",
    "US20070101291A1",
    "US8832048B2",
    "US8396890B2",
    "US20080005086A1",
    "US7533096B2",
    "US20080028392A1",
    "US8056092B2",
    "US9278284B2",
    "US20100235433A1",
    "US20080209392A1",
    "US20080222454A1",
    "US8028269B2",
    "US8095910B2",
    "US20080263024A1",
    "US20100138015A1",
    "US8209738B2",
    "US8132156B2",
    "US20100251163A1",
    "US20090037509A1",
    "US8245215B2",
    "US20090116071A1",
    "US20090172814A1",
    "US20090199160A1",
    "US20090217309A1",
    "US20090249297A1",
    "US8234622B2",
    "US20110145807A1",
    "US20090320002A1",
    "US20100023933A1",
    "US20100023602A1",
    "US20100100605A1",
    "US20100106671A1",
    "US8943493B2",
    "US8769482B2",
    "US8443291B2",
    "US8438558B1",
    "US8713003B2",
    "US20110276568A1",
    "US8352464B2",
    "US20110022578A1",
    "US8874685B1",
    "US9621584B1",
    "US20120330940A1",
    "US20110197121A1",
    "US20110276946A1",
    "US9462042B2",
    "US8504803B2",
    "US20160019049A1",
    "US20160078368A1",
    "US20110302570A1",
    "US20120011458A1",
    "US9213625B1",
    "US20120042281A1",
    "US8498473B2",
    "US20130173648A1",
    "US20120124062A1",
    "US20120131456A1",
    "US9032314B2",
    "US20120324333A1",
    "US8682083B2",
    "US8631458B1",
    "US9571332B2",
    "US20130290318A1",
    "US20140036290A1",
    "US20140045484A1",
    "US20140189576A1",
    "US9946233B2",
    "US20140181705A1",
    "US8965905B2",
    "US8819241B1",
    "US20140282125A1",
    "US20150082280A1",
    "US9600519B2",
    "US20150310268A1",
    "US20150347284A1",
    "US20160034441A1",
    "US20180113781A1",
    "US20180275835A1",
    "US20190126463A1",
    "US20190141596A1",
    "US20190188462A1",
    "US10043255B1",
    "US10282280B1",
    "US20190317803A1",
    "US11182178B1"
  ]
}