GUI: Mouse Actions
Description
GUI: Mouse Actions plugin step will perform mouse actions.
Prerequisites:
- If you want to use Java technology in this step, ensure you have Java 1.8 32-bit installed on your system. For details on installing Java 1.8 and setting up access bridge, see Appendix: Java Access Bridge Installation.
- If you want to use Web technology in this step, ensure valid extensions are added to corresponding Web browser.
Limitations:
The step will not work for Java and Windows technologies on Linux. For Java technology:
- The scaling factor of the machine must be 100% only.
- Resolution and scaling of workflow recorder machine and workflow runner step must be same.
- This release does not support minimizing frames or maximizing internal frames. Designer can use GUI: Universal Actions step for robot handling
- Max Timeout in step configuration is only applicable for Java and Windows technology.
- Recorder is not supported for java/applet.
Known issues:
For both, Java and Windows technologies:
- In the Mouse Actions step, the Scroll action does not work correctly. Workaround: To capture the Scroll action, use either of the following steps: Special Key Actions and Universal Actions. For Java technology:
- When the access bridge is configured to run the GUI Java automation steps, then SAHI automation will not work.
Configurations
| No. | Field Name | Description |
|---|---|---|
| 1 | Step Name | Name of the step. The name must be unique in a single workflow.The data type is string, and the field is mandatory. |
| 2 | Technology | Select the technology used by the target application: Java, Web, or Windows.The data type is string, and the field is mandatory. |
| 3 | Action Information | Click to open the action information dialog.Select the field details, as required. Click OK, and the selected Action Information details appear in the tabular format.Note: The fields will change depending on the selected technology. |
| For Windows/Java Technology | Provide the following field details in the Action Information dialog | |
| I | Action Type | Select an Action Type from the following list: - Click - Double Click - Drag Drop - Hover - Right Click - Scroll - Toggle - Middle Click – Only for Java Technology These are extra options available for Windows technology. - Expand - Collapse Note: - Hover, Click, Double Click, and Right Click actions are available with all element types. - Expand and Collapse are useful for tree items and dropdown element. If expand is not available, you may use Click Action Type. - Toggle is useful for setting value to checkbox. Scroll is available for scrollable elements. You need to detect the immediate window containing the scroll bar, possibly by hovering on the intersection of the right and bottom scroll bars. If action type is Drag Drop, first Spy the Source Element and generate step. In the step, click on any field in the Target Element and click Edit Destination Element to spy the destination element. Note: In Mouse Actions, if drag-drop is not working you may use the Set Value step. |
| In Action Type | Select Scroll, then depending on the technology the following fields are available for: Windows technology: Horizontal Amount and Vertical Amount. Java technology: Scroll Amount. For field option details, see the description of Horizontal Amount and Vertical Amount fields. | |
| a. | Horizontal Amount | -LargeDecrement / SmallDecrement: Scroll left as per the predefined value for scrolling. -SmallIncrement / LargeIncrement: Scroll right as per the predefined value for scrolling. - NoAmount: Left or Right scrolling will be unavailable. Note: The NoAmount option is not available for Scroll amount. |
| b. | Vertical Amount | - LargeDecrement / SmallDecrement: Scroll up as per the predefined value for scrolling - NoAmount: Up or down scrolling will be unavailable. SmallIncrement / LargeIncrement: Scroll down as per the predefined value for scrolling. |
| In Action Type, if you select Drag Drop, Provide following field details: | ||
| Source Element Fields: | ||
| 1 | Window Path | Window Path is the relative\absolute path of the Windows application on which element exists. It is captured by GUI Spy in the Generate Step dialog. It is then populated in the step configuration upon Step Generation. You may also enter or modify Windows path manually on the Step Generation window or step configuration. Windows path may be provided as a String or regex. Sample of Windows Path as string: w[Desktop]/w[Application Name], w[Desktop]/w[/Application N/], w[Desktop]/w[/Application .*/]. If you are providing partial application name then put it as a regex between / and /. Also, if the Application name changes dynamically, it is a good idea to provide it as a regex. Java: Provide the Window Name on which automation actions should be performed. For example, Windows Name: AutomationEdge Applet App Users can also utilize a regex pattern to match the window name, instead of specifying the exact window name of Java Application. For example: Suppose the Java application window title is, “Oracle Apps 2024-11-26“, a user can provide: - Oracle Apps \d4-\d2-\d2 - ^Oracle - Any other valid regex |
| 2 | Element Index(Starts with 0) | For Multiple elements with same criteria, specify index of the element you want to locate. Provide index 0 if you want first element from the located elements, increment index by 1 for the next elements. Default Value:0 |
| 3 | Criteria | Select the criteria from the list to locate an element. ‘Criteria’ is the criterion to locate element on Windows \Java application. It is captured by Windows Spy in the Generate Step dialog. Upon Step Generation it is populated in the step configuration. You may change criteria by choosing from the list in Step Generation window or step configuration. For list and description see Criteria Table. The field is Mandatory Field. |
| 4 | Criteria Value | Specify relevant criteria value for the criteria provided above. |
| 5 | Position | Select source element position from the list: -Left -Right -Bottom Right - Bottom Left - Top Right - Top Left - Bottom - Top - Center |
| Destination Element: | ||
| I | Window Path | Window Path is the relative\absolute path of the Windows application on which element exists. It is captured by GUI Spy in the Generate Step dialog. It is then populated in the step configuration upon Step Generation. You may also enter or modify Windows path manually on the Step Generation window or step configuration. Windows path may be provided as a String or regex. Sample of Windows Path as string: w[Desktop]/w[Application Name], w[Desktop]/w[/Application N/], w[Desktop]/w[/Application .*/]. If you are providing partial application name then put it as a regex between / and /. Also, if the Application name changes dynamically, it is a good idea to provide it as a regex. Java: Provide the Window Name on which automation actions should be performed. For example, Windows Name: AutomationEdge Applet App Users can also utilize a regex pattern to match the window name, instead of specifying the exact window name of Java Application. For example: Suppose the Java application window title is, “Oracle Apps 2024-11-26“, a user can provide: - Oracle Apps \d4-\d2-\d2 - ^Oracle - Any other valid regex |
| II | Element Index(Starts with 0) | For Multiple elements with same criteria, specify index of the element you want to locate. Provide index 0 if you want first element from the located elements, increment index by 1 for the next elements. |
| III | Criteria | ‘Criteria’ is the criterion to locate element on Windows Desktop. It is captured by Windows Spy in the Generate Step dialog.Upon Step Generation it is populated in the step configuration. You may change criteria by choosing from the list in ‘Step Generation’ window or step configuration. |
| IV | Criteria Value | Specify relevant criteria value for the criteria provided above. |
| V | Position | Select source element position from the list: -Left -Right -Bottom Right - Bottom Left - Top Right - Top Left - Bottom - Top - Center |
| For Web Technology | Provide the following field details in the Action Information dialog: | |
| I | Action Type | Select an Action Type from the following list: - Single Click - Right Click -Double Click -Mouse Up - Mouse Down It is a mandatory field. |
| II | Web Browser Name | Select any one Web Browser Name from the given list.Note: Select the same Web Browser Name, which was selected in Start Browser Step. |
| III | Element Index(Starts with 0) | Specify the index of the element you want to locate. Note: One can also specify First or Last. First, if you want the first element from the located element or specify Last if you want the last element. Default Value: 0 |
| IV | Criteria | ‘Criteria’ is the criterion to locate elements on Web GUI. GUI Spy can capture it in the Generate Step dialog. Upon Step Generation, it is populated in the step configuration. You may change criteria by choosing from the drop-down list in the ‘Step Generation’ window or step configuration. Note: For list and description of criteria, See Criteria Table. It is a mandatory field. |
| V | Criteria Value | Specify relevant criteria value for the criteria provided above.It is a mandatory field. |
| VI | Iframe Indices | Specify Iframe Indices value. Specifying iframe indices ensures that the target element is identified correctly within the nested frame structure.You can provide comma-separated values.For e.g. 0,1,2. |
| VII. | Iframe Xpath | Iframe Xpath is a uniquely generated path that represents the location of an <iframe> or <frame> relative to the top-level window. Iframe Xpath identifies each frame using a unique attribute such as id, name, or title if available, for example, iframe[@id='mainFrame']. If no such attributes exist, the frame is identified by its position among sibling frames, like iframe[2] for the second iframe. For nested structures, each frame is joined in sequence to form a full path. For example, if an element is inside an iframe[@id='outer'], which contains a frameset with a frame[@name='middle'], and that frame contains iframe[@title='inner'], the generated path would be: iframe[@id='outer'];frame[@name='middle'];iframe[@title='inner']. Note: If an iframe or frame has no id, name, or title, its origin (i.e., the src URL) will be used as a fallback in the path for unique identification. For example: iframe[@origin='https://example.com']. |
| VIII. | Frame ID | Provide the current Iframe Id to target a specific iframe to perform the action. Default Value: frameID |
| 5 | Max Timeout (In Second) | Specify a Timeout value for the action to be performed. |
| 6 | Step Description | Describe the intended action for better step-row clarity.It is a mandatory field. |
| Common Buttons: | ||
| 1 | Edit Source Element/ Edit Destination Element | Click to start spying. Either Edit Source Element or Edit Destination Element button is visible depending on any field chosen on ‘Source element’ or Destination element’ group. Opening spy from step is restricted to the step, i.e. Only relevant elements are detected from active spy. Use Hotkeys (‘Shift’ or ’~’) to detect the element. We may call it as a ‘Step Restricted Spy’ for understanding purposes. E.g. is restricted to elements that support ‘Mouse Action’. All elements that are supported will be highlighted in blue color and others will be highlighted in ‘Red’ color upon hover. Step dropdown is disabled and value selected is ‘Mouse Action’. Note: Option is visible for Java Technology or Windows Technology. |