Schedule Automated Data Updates

There are often data update operations that must be performed on a regular or periodic basis, such as retrieving updates from external data sources or exporting data. AnzoGraph DB provides a CRON-like mechanism to automatically perform these repetitive operations. These operations are managed entirely within the database rather than being controlled by the configuration of external control files.

There are two primary aspects to creating and configuring automated or scheduled operations within AnzoGraph DB:

1. Create and define the contents of one or more Cron graphs, each of which specify the database operations to perform for one or more Cron jobs. Each Cron graph is defined as a collection of RDF triples, with each triple specifying a particular scheduled job attribute or parameter. The Cron graph includes configuration settings that control other aspects of each scheduled job, such as a job's scheduled execution time (particular dates and times or intervals), retry options, error handling policies, and so on.
2. Update the scheduled Cron graph job settings in the AnzoGraph DB settings.conf file to include the Cron graphs you want to execute. The settings.conf file contains two settings to control the scheduling and execution of Cron graphs, cron_graphs and cron_graphs_recheck.

This topic provides instructions for setting up automated database operations and describes the configuration options and best practices available to control the scheduling, prioritization, error handling, and other aspects of running jobs.

• Create a Cron Graph
• Load a Cron Graph
• Configure AnzoGraph DB to Run Cron Jobs
• Monitor Job Execution and Errors

Create a Cron Graph

A Cron graph is defined in a TTL file that contains a collection of RDF triples that define configuration and scheduling information for one or more Cron jobs. A Cron graph can contain any number of Cron jobs, and each job can have custom scheduling and error-handling policies.

The content below shows the syntax for a Cron graph file. Descriptions of each job parameter are provided below.

# filename.ttl

PREFIX azg: <http://www.anzograph.com/> .
 
<job_name> azg:Schedule | Delay "<duration_value>"^^xsd:duration ;
           azg:Statement "<statement>" ;
           azg:ErrorPolicy "<policy>" ;
           [ azg:BaseTime "<datetime_value>"^^xsd:dateTime ; ]
           [ azg:RetryInterval "<duration_value>"^^xsd::duration ; ]
           [ azg:RetryCount <integer_value> ; ]
           [ azg:RunAfterStartup "true | false" ] .
 
[ <job2>   azg:Schedule | Delay "<duration_value>"^^xsd:duration ;
           azg:Statement "<statement>" ;
           azg:ErrorPolicy "<policy>" ;
           ...
]

Example Cron Graph File

The following content provides a simple example of a Cron graph file, named cron1.ttl, which schedules two jobs in the same graph:

PREFIX azg: <http://www.anzograph.com/> .
<job1> azg:BaseTime "2020-04-07:11:32"^^xsd:dateTime .
<job1> azg:Schedule "1Day"^^xsd:duration .
<job1> azg:ErrorPolicy "AbortDatabase" .
<job1> azg:RetryInterval "1Hour"^^xsd::duration .
<job1> azg:RetryCount 23 .
<job1> azg:Statement "REFRESH VIEW <testView1>" .
<job2> azg:BaseTime "2020-07-08:00:00"^^xsd:dateTime .
<job2> azg:Schedule "1Day"^^xsd:duration .
<job2> azg:ErrorPolicy "Ignore" .
<job2> azg:Statement "REFRESH VIEW <testView2>" .

In this example, the subject defines the job names: job1 for scheduling and configuration of one scheduled job, and job2 for the scheduling and configuration of a second job. Each predicate specifies a particular attribute or parameter of a scheduled job.

Load a Cron Graph

Once you have created a Cron graph file, you load the Cron graph into AnzoGraph DB using the following LOAD command:

LOAD <file:/<path>/<filename>.ttl> INTO GRAPH <graph_name>

For example:

LOAD <file:/tmp/cron1.ttl> INTO GRAPH <CronGraph1>

In this example, the triples stored in the cron1.ttl file are loaded into a graph named CronGraph1. It is this name, CronGraph1, that is added to the cron_graphs setting in <install_path>/config/settings.conf to run the scheduled jobs defined in CronGraph1. More details about configuring AnzoGraph DB to run Cron jobs are included in Configure AnzoGraph DB to Run Cron Jobs.

PREFIX azg: <http://www.anzograph.com/> .

GRAPH <CronGraph1> {

  <job1> azg:BaseTime "2020-04-07:11:32"^^xsd:dateTime .
  ...
  <job1> azg:Statement "REFRESH VIEW <testView1>" .
  <job2> azg:BaseTime "2020-07-08:00:00"^^xsd:dateTime .
  ...
  <job2> azg:Statement "REFRESH VIEW <testView2>" .
}

LOAD <file:/path/cron1.ttl>

Configure AnzoGraph DB to Run Cron Jobs

To configure AnzoGraph DB to run the jobs within Cron graphs, edit the <install_path>/config/settings.conf configuration file to specify values for the following two settings:

• cron_graphs: A comma-separated list of the Cron graph names to enable. For example, cron_graphs=CronGraph1, CronGraph2.

• cron_graphs_recheck: The interval (in number of seconds) to wait before re-checking the cron_graphs value to determine if there are changes, i.e, new or deleted graph names. For example, cron_graphs_recheck=300.

  If a Cron graph is non-existent or empty, the associated Cron thread periodically checks at the specified interval whether the named graph is now loaded and has new jobs.

After changing settings.conf, restart AnzoGraph DB to apply the configuration changes.

Monitor Job Execution and Errors

Details about scheduled job run are logged to the following AnzoGraph DB system tables.

You can query AnzoGraph DB's system tables using SPARQL queries in the following format:

SELECT * | list_of_variables
WHERE { table 'table_name' }

For example:

SELECT *
WHERE { table 'sth_cron_events' }
LIMIT 100

Cron Graph Errors

The table below lists the errors that are returned for errors related to Cron job processing.