Sort Rows
Description
Sort Rows orders the data stream by one or more specified fields in ascending or descending order. Use this step whenever downstream steps require sorted input — for example, before a Group By, Merge Join, or Unique Rows step. For large datasets that exceed memory capacity, the step automatically uses temporary files for disk-based sorting. You can also configure it to remove duplicate rows during the sort operation and compress temporary files to save disk space.
Configurations
| Field Name | Description |
|---|---|
| Step name | Name of the step as it appears in the workflow workspace. This name has to be unique in a single workflow. |
| Sort directory | The directory in which the temporary files are stored in case when needed; the default is the standard temporary directory for the system |
| TMP-file prefix | Choose an easily recognized prefix so you can identify the files when they show up in the temp directory. |
| Sort size | The more rows you store in memory, the faster the sorting process because fewer temporary files must be used and less I/O is generated. |
Free memory threshold (in %) | If the sort algorithm finds that it has less available free memory than the indicated number, it will start to page data to disk. Note: This is not exact science, because: - This is checked every 1000 rows. Depending on the row size and other steps within complex workflows this could still lead to an OutOfMemoryError. - In a Java Virtual Machine it's not possible to know the exact amount of free memory. As such it is not recommended to use this step for very complex workflows with other steps and processes that use up a lot of memory. |
| Compress TMP Files | Compresses temporary files when they are needed to complete the sort. |
| Only pass unique rows? | Enable if you want to pass unique rows only to the output stream(s). |
| Fields table | Specify the fields and direction (ascending/descending) to sort. You can specify whether to perform a case sensitive sort (optional) |
| Get Fields | Click to retrieve a list of all fields coming in on the stream(s). |