Unique Rows
Description
Unique Rows removes duplicate rows from a sorted input stream by comparing consecutive rows on specified fields. Use this step after a Sort Rows step when you need to de-duplicate data — for example, ensuring each customer appears only once in a mailing list or removing repeated log entries. The input must be sorted on the comparison fields; for unsorted data, use the Unique Rows (HashSet) step instead. Duplicate rows can optionally be redirected to a separate error hop for review or logging.
Configurations
| Field Name | Description |
|---|---|
| Step name | Name of the step as it appears in the workflow workspace. This name has to be unique in a single workflow. |
| Add counter to output? | Check this option to add a counter field to the stream. |
| Counter field | Define the counter field name. |
| Redirect duplicate row | Processes duplicate rows as an error and redirect rows to the error stream of the step. Requires you to set error handling for this step. |
| Error Description | Sets the error handling description to display when duplicate rows are detected. Only available when Redirect duplicate row is checked. |
| Fields to compare table | Specify the field names on which you want to force uniqueness or click Get to insert all fields from the input stream(s) You can choose to ignore case by setting the Ignore case flag to Y. For example: Tea, TEA, tea are the same if the compare is performed as case-insensitive. In this instance, the f irst occurrence (Tea) is passed to the next step(s). |