Robotic process automation (RPA) comprising automatic document scrolling

{
  "publication_number": "US12118043B2",
  "country": "US",
  "kind": "B2",
  "title": "Robotic process automation (RPA) comprising automatic document scrolling",
  "abstract": "In some embodiments, a robotic process automation (RPA) robot is configured to search for a target element within a first part of a document currently exposed within a user interface. When the search fails, the robot may automatically actuate a scroll control of the respective UI to cause it to bring another part of the respective document into view. The robot may then continue searching for the RPA target within the newly revealed part of the document. In some embodiments, the robot automatically determines whether the respective document is scrollable, and identifies the scroll control according to a type of target application (e.g., spreadsheet vs. web browser).",
  "claims": [
    "1. A method comprising employing at least one hardware processor of a computer system to expose a robotic process automation (RPA) robot design interface to a user, the robot design interface configured to: receive a user input indicating a target element of a user interface (UI) displaying an electronic document, the user input further indicating an RPA activity to be performed on the target element; display a control enabling the user to indicate whether an automatic scroll option is active; and output an RPA script including an indicator of whether the automatic scroll option is active, wherein executing the RPA script causes an RPA robot to: perform an attempt to identify a runtime instance of the target element within a runtime instance of the UI, the runtime instance of the UI currently displaying an initial part of the electronic document; in response to the attempt failing, determine according to the RPA script whether the automatic scroll option is active; in response to determining that the automatic scroll option is active, activate a scroll control of the runtime instance of the UI to cause a display of a subsequent part of the electronic document; in response to activating the scroll control, perform another attempt to identify the runtime instance of the target element within the subsequent part of the electronic document; and in response to a successful identification of the runtime instance of the target element, carry out the RPA activity on the runtime instance of the target element.",
    "2. The method of claim 1, wherein executing the RPA script further causes the RPA robot to automatically identify the scroll control according to a type of application exposing the runtime instance of the UI.",
    "3. The method of claim 1, wherein executing the RPA script further causes the RPA robot to automatically identify the scroll control, and wherein identifying the scroll control comprises: selecting a candidate UI element from the runtime instance of the UI according to whether the candidate UI element is responsive; and determining whether the candidate UI element comprises the scroll control.",
    "4. The method of claim 1, wherein the electronic document comprises an item selected from a group consisting of a spreadsheet, a webpage, and a list of email messages.",
    "5. The method of claim 1, wherein the electronic document includes a hierarchical structure comprising a plurality of nodes, and wherein activating the scroll control causes the runtime instance of the UI to expand a selected node of the plurality of nodes into a set of sub-nodes.",
    "6. The method of claim 1, further comprising employing the at least one hardware processor to identify the scroll control according to a snapshot of at least one region of the runtime UI.",
    "7. A computer system comprising at least one hardware processor configured to expose a robotic process automation (RPA) robot design interface to a user, the robot design interface configured to: receive a user input indicating a target element of a user interface (UI) displaying an electronic document, the user input further indicating an RPA activity to be performed on the target element; display a control enabling the user to indicate whether an automatic scroll option is active; and output an RPA script including an indicator of whether the automatic scroll option is active, wherein executing the RPA script causes an RPA robot to: perform an attempt to identify a runtime instance of the target element within a runtime instance of the UI, the runtime instance of the UI currently displaying an initial part of the electronic document; if the attempt fails, determine according to the RPA script whether the automatic scroll option is active; in response to determining that the automatic scroll option is active, activate a scroll control of the runtime instance of the UI to cause a display of a subsequent part of the electronic document; in response to activating the scroll control, perform another attempt to identify the runtime instance of the target element within the subsequent part of the electronic document; and in response to a successful identification of the runtime instance of the target element, carry out the RPA activity on the runtime instance of the target element.",
    "8. The computer system of claim 7, wherein executing the RPA script further causes the RPA robot to automatically identify the scroll control according to a type of application exposing the runtime instance of the UI.",
    "9. The computer system of claim 7, wherein executing the RPA script further causes the RPA robot to automatically identify the scroll control, and wherein identifying the scroll control comprises: selecting a candidate UI element from the runtime instance of the UI according to whether the candidate UI element is responsive; and determining whether the candidate UI element comprises the scroll control.",
    "10. The computer system of claim 7, wherein the electronic document comprises an item selected from a group consisting of a spreadsheet, a webpage, and a list of email messages.",
    "11. The computer system of claim 7, wherein the electronic document includes a hierarchical structure comprising a plurality of nodes, and wherein activating the scroll control causes the runtime instance of the UI to expand a selected node of the plurality of nodes into a set of sub-nodes.",
    "12. The computer system of claim 7, wherein the at least one hardware processor is further configured to identify the scroll control according to a snapshot of at least one region of the runtime UI.",
    "13. A non-transitory computer-readable medium storing instructions which, when executed by at least one hardware processor of a computer system, cause the computer system to expose a robotic process automation (RPA) robot design interface to a user, the robot design interface configured to: receive a user input indicating a target element of a user interface (UI) displaying an electronic document, the user input further indicating an RPA activity to be performed on the target element; display a control enabling the user to indicate whether an automatic scroll option is active; and output an RPA script including an indicator of whether the automatic scroll option is active, wherein executing the RPA script causes an RPA robot to: perform an attempt to identify a runtime instance of the target element within a runtime instance of the UI, the runtime instance of the UI currently displaying an initial part of the electronic document; if the attempt fails, determine according to the RPA script whether the automatic scroll option is active; in response to determining that the automatic scroll option is active, activate a scroll control of the runtime instance of the UI to cause a display of a subsequent part of the electronic document; in response to activating the scroll control, perform another attempt to identify the runtime instance of the target element within the subsequent part of the electronic document; and in response to a successful identification of the runtime instance of the target element, carry out the RPA activity on the runtime instance of the target element.",
    "14. The method of claim 1, wherein the RPA script comprises a specification of a sequence of RPA activities to be carried out by the RPA robot, the sequence including the indicated RPA activity, and wherein the robot design interface enables the user to indicate whether the automatic scroll option is active independently for each RPA activity of the sequence.",
    "15. The computer system of claim 7, wherein the RPA script comprises a specification of a sequence of RPA activities to be carried out by the RPA robot, the sequence including the indicated RPA activity, and wherein the robot design interface enables the user to indicate whether the automatic scroll option is active independently for each RPA activity of the sequence.",
    "16. The method of claim 1, wherein executing the RPA script causes the RPA robot, in preparation for activating the scroll control, to determine whether to activate the scroll control according to a type of the electronic document.",
    "17. The computer system of claim 7, wherein executing the RPA script causes the RPA robot, in preparation for activating the scroll control, to determine whether to activate the scroll control according to a type of the electronic document.",
    "18. The method of claim 16, wherein executing the RPA script causes the RPA robot to not activate the scroll control if the electronic document is a webpage.",
    "19. The computer system of claim 17, wherein executing the RPA script causes the RPA robot to not activate the scroll control if the electronic document is a webpage.",
    "20. The method of claim 1, wherein executing the RPA script causes the RPA robot, in preparation for activating the scroll control, to determine whether to activate the scroll control according to a device type of an RPA host executing the RPA robot.",
    "21. The computer system of claim 7, wherein executing the RPA script causes the RPA robot, in preparation for activating the scroll control, to determine whether to activate the scroll control according to a device type of an RPA host executing the RPA robot."
  ],
  "description_excerpt": "The invention relates to robotic process automation (RPA), and in particular to facilitating the automatic processing of large documents, such as substantial spreadsheets and webpages.\n\nRPA is an emerging field of information technology aimed at improving productivity by automating repetitive computing tasks, thus freeing human operators to perform more intellectually sophisticated and/or creative activities. Notable tasks targeted for automation include extracting structured data from documents and interacting with user interfaces, for instance to fill in forms, among others. However, automating the interaction with a user interface poses substantial technical challenges, for instance unambiguously identifying a target UI element such as a button, a form field, or a spreadsheet cell. Furthermore, RPA applications may fail due to changes in the appearance of the interface (e.g., positioning of various elements, color scheme, fonts, etc.) occurring between the design of the respective software robot and runtime.\n\nA distinct prong of RPA development is directed at simplifying the programming and management of software robots, with the ultimate goal of extending the reach of RPA technology to users that lack advanced programming skills or training. One way of making RPA more accessible is the development of RPA-oriented integrated development environments (IDEs) which allow the programming of robots via graphical user interface (GUI) tools, instead of coding per se.",
  "cpc": [
    "G06F 16/93",
    "B25J 9/1628",
    "G05B 19/0426",
    "G05B 2219/23258",
    "G05B 2219/24071",
    "G06F 11/36",
    "G06F 11/3696",
    "G06F 3/0483",
    "G06F 3/0485",
    "G06F 40/174",
    "G06F 8/34",
    "G06F 9/451",
    "G06F 9/45512",
    "G06F 9/45529"
  ],
  "ipc": [
    "B25J 9/16",
    "G05B 19/042",
    "G06F 11/36",
    "G06F 16/93",
    "G06F 9/451"
  ],
  "assignees": [
    "UiPath Inc"
  ],
  "inventors": [
    "Iulian MIRCEA",
    "Lev KUSHNIR"
  ],
  "filing_date": "2021-05-06",
  "publication_date": "2024-10-15",
  "grant_date": "2024-10-15",
  "priority_date": "2021-05-06",
  "application_number": "US-202117302560-A",
  "family_id": "80001338",
  "cited_by_count": 0,
  "citations": [
    "US5600789A",
    "US5874961A",
    "US20100058226A1",
    "US20130263090A1",
    "US20150339213A1",
    "US20180173614A1",
    "US10354225B2",
    "US10235192B2",
    "US20190354720A1",
    "WO2019172485A1",
    "US20190324781A1",
    "US20200034399A1",
    "US20200073686A1",
    "US10949225B2",
    "US20200285353A1",
    "US10713068B1",
    "US20210110233A1",
    "WO2021130500A1"
  ]
}