Incremental Pipeline Reference
When an incremental schema is added to an ETL job, a clock icon (
) is displayed when hovering over the component in the job. For example:
Clicking the clock icon opens the Incremental Load dialog box, which lists the Incremental Column Name, Value, and Comparator from the schema query. For example:
Publishing the job for this example will onboard only the records for which the EmployeeID is greater than 5. When the job is finished, Anzo adjusts the incremental load value to list the last value that was onboarded for the incremental column. Every time the pipeline is published, Anzo changes the incremental load value parameter to the highest or lowest value for the column, depending on the Comparator.
For example, viewing the Incremental Load dialog box after running the job above shows that the last EmployeeID value that was onboarded was 14:
The next time this job is run, Anzo will onboard only the records where EmployeeID is greater than 14. To view the number of rows processed after running a job, you can search the System Datasource for the following predicate on the Find tab in the Query Builder:
<http://cambridgesemantics.com/ontologies/2015/08/SDIService#rowsProcessed>
The Object column shows the number of rows processed each time the pipeline was run. For example:
