Surface

Surface plugin has been designed with image recognition capabilities to capture images/regions on a screen. The captured image can be used for image search, mouse Actions on the image, image labeling and optical character recognition (OCR).

Surface Automation is particularly useful on remote machines where we do not have any handle to the content and everything is an image or pattern. It can be useful to work on remote environments such as Citrix.

Prerequisite:

• For surface plugin steps, Microsoft Visual C++ version 2015-2019 Redistributable (x64) and Microsoft Visual C++ version 2015-2019 Redistributable (x86) should be installed.

  • This is available at the following URL,

  • https://support.microsoft.com/en-in/help/2977003/the-latest-supported-visual-c- downloads

• Download the following two executable:

  • x86: vc_redist.x86.exe

  • x64: vc_redist.x64.exe

• Surface spy is required to build workflows with Surface plugin steps. For a description on how to use Surface Spy refer the following,

Additional Prerequisite for Windows Server OS:

Surface Plugins use some of Windows Media Service DLL's (ex.mfplat.dll) for image recognition. You need to enable "Desktop Experience" windows feature to get the required DLL's. Following are the steps to install the required Windows Media Service - DLL's

1. On Windows Start menu search for 'Turn windows features on or off'

2. Skip the roles screen and directly go to Feature screen

3. Select "Desktop Experience" under "User Interfaces and Infrastructure"

4. Restart the system.

Limitation: Any type of UI Automation with Surface plugin steps is not supported in Linux based OS.

References:

For a description on how to use Surface Spy refer to Appendix 4: Surface Spy Tutorial.

Surface Actions

Description

Surface Actions step is used to perform multiple actions in a single step.

Prerequisites:

• Use machines with similar scale and resolution at the time of workflow creation and workflow execution.

• Open the remote or virtual machine in full-screen mode.

Limitation:

Surface Actions step is not supported on Linux.

Configurations

Field Name	Description
Step Name	Name of the step. The name must be unique in a single workflow.
Template Match Pattern	Select the pattern option from the list. - Retrieve Single closest match: It searches for and retrieves a single closest match. If the image is not found or multiple matches are found, then it throws an error. - Retrieve Multiple close matches: It searches for matching patterns and retrieves the closest matches. - Retrieve All matches: : It searches for and retrieves all matches (not below 50% or above 150% of the original size). The field is mandatory. The data type is String.
Template Image Index	Specify the image index. The image index determines which image to use when multiple similar images are found and starts with 0. The field is mandatory. The data type is Integer.
Retry Count	The field allows the system to automatically reattempt the image-matching process up to the specified count.
Template Image	Displays the template image with respect to which the user will perform actions.The field is mandatory.
Add / Edit Image	Click to show the Surface Spy window. You can perform capture and locate functions and change editable step properties.The field is mandatory.
Table Fields: Action Fields
Action	Click on the row to select the action details. For details, see Appendix: Surface Actions: How to guide. The field is mandatory. The data type is String.
Wait For(in seconds)	Provide the Wait For to set a specific duration in seconds before the performing action.The field is mandatory. The data type is Integer.
Match Pattern	Select the pattern option from the list. - Single Closest Match: It searches for and retrieves a single closest match. If the image is not found or multiple matches are found, then it throws an error. - Multiple close matches: It searches for matching patterns and retrieves the closest matches. - All matches: It searches for and retrieves all matches (not below 50% or above 150% of the original size). Note: The field works the same as the field Template Match Pattern and applies to table rows only. The field is mandatory. The data type is String.
Image Index	Specify the image index. The image index determines which image to use when multiple similar images are found and starts with 0.Note: The field works the same as the field Template Image Index and applies to table rows only. The field is mandatory. The data type is Integer.
Field Information	Use the field to provide additional information about the action just like providing comments.