Ingesting Data into Anzo

Follow the instructions below to ingest data for a schema that does not have an associated data dictionary.

1. In the Anzo console, expand the Onboard menu and click Structured Data. Anzo displays the Data Sources screen, which lists the available data sources. For example:

   

2. On the Data Sources screen, click the name of the data source for which you want to ingest data. Anzo displays the Tables screen for the source. For example:

   

3. Click the Ingest button. If the source has more than one schema, Anzo displays the select schema dialog box. In the drop-down list, select the schema to use, and then click OK. For example:

   

   Anzo opens the Ingest dialog box and automatically populates the data source as well as the Anzo data store if only one store is configured. For example:

   

4. If necessary, click the Anzo Data Store field and select the data store for this pipeline. For information about creating an Anzo data store, see Creating an Anzo Data Store.
5. Click the Auto Map Engine Config field and select the ETL engine to use for this new pipeline.
6. By default, Anzo enables the Select all tables radio button to ingest the data for all tables in the schema. If you do not want to add all tables, click the Custom select radio button and then select each of the tables to add.
7. Click Advanced to view additional configuration options. If this source has not been ingested previously, the workflow defaults to Create New Model. If the source has been ingested, the workflow defaults to Use Existing Model. Click a description below to view instructions for configuring the workflow to create a new model or use an existing one:
   Create a new model

   Follow the instructions below if you want to create a new model.

   1. Select the Create New Model radio button. Additional options become available:

      

   2. If you want to define the model, class, or property URIs, type the desired prefixes in the Schema Ontology URI, Schema Class Prefix, and Schema Property Prefix fields:
      • Schema Ontology URI: The URI for the data model. When the Schema Ontology URI field is blank, Anzo generates the model URI for this schema with the following format:

        http://cambridgesemantics.com/ont/autogen/xx/schema_name

        Where xx is a hash snippet based on the model's globally unique identifier (GUID). If you want Anzo to use a specific URI format when creating the model for this schema, you can type that URI into the Schema Ontology URI field. For example, you can specify a URI such as http://mycompany.com.ontology/movies. This format results in an ontology URI of http://mycompany.com.ontology/movies.

        Important: Make sure that you type a unique Schema Ontology URI for each schema. Schemas with the same model URI overwrite each other when the projects run.

      • Schema Class Prefix: The URI prefix format to use for classes in the data model. When the Schema Class Prefix field is blank, Anzo generates the ontology's class URIs using the following format:

        http://cambridgesemantics.com/ont/autogen/xx/schema_name#class_name

        Where xx is a hash snippet based on the model's GUID. If you want Anzo to use a specific URI format when creating classes in the model, you can type that URI into the Schema Class Prefix field. For example, you can specify a URI such as http://mycompany.com.ontology/class. This format results in class URIs like http://mycompany.com.ontology/class#className.

        Tip: To generate models from multiple sources and use the same class URIs across models, you can set the Schema Class Prefix to the same value across schemas.

      • Schema Property Prefix: The URI prefix format to use for properties in the data model. When the Schema Property Prefix field is blank, Anzo generates the ontology's property URIs using the following format:

        http://cambridgesemantics.com/ont/autogen/xx/schema_name#class_name_property_name

        Where xx is a hash snippet based on the model's GUID. If you want Anzo to use a specific URI format when creating properties in the model, you can type that URI into the Schema Property Prefix field. For example, you can specify a URI such as http://mycompany.com.ontology/property. This format results in property URIs like http://mycompany.com.ontology/property#className_propertyName.

        Tip: To generate models from multiple sources and use the same property URIs across models, you can set the Schema Property Prefix to the same value across schemas.

   3. If you want to transform property names to upper or lower case letters, select the Transform Property Names checkbox. Then select the To lowercase radio button if you want to convert property names to lowercase or select the To UPPERCASE radio button if you want to convert property names to uppercase.
   Use an existing model for this workflow

   Follow the instructions below if you want to associate this source with an existing model.

   1. Select the Use Existing Model radio button. Additional options become available:

      

   2. Select the model to use from the Model drop-down list.

   3. Specify whether to regenerate artifacts and which artifacts to regenerate. The list below describes each of the options:

      • Regenerate Entire Model: Selecting this option means that Anzo deletes all entities from the existing model and recreates them. The model that results from the current ingestion process will contain only the data from the current process. For example, if a previous run generated a model that contains classes A, B, and C, and the current data contains Classes C, D, and E, selecting Regenerate Entire Model results in a model that contains only classes C, D, and E. If Regenerate Entire Model is NOT selected, the resulting model will contain classes A, B, C, D, and E.
      • Regenerate Mappings and Jobs: Selecting this option means that Anzo deletes all entities from the existing mappings and jobs and recreates them. The artifacts that result from the current ingestion process will contain only the data from the current process. For example, if a previous run generated mappings and jobs that contain tables A and B and the current run is ingesting tables C and D, selecting Regenerate Mappings and Jobs results in artifacts that contain only tables C and D. If Regenerate Mappings and Jobs is NOT selected, the resulting artifacts contain tables A, B, C, and D.
   4. Specify what Anzo should do if there is a property type mismatch between the existing model and the new schema:
      • Merge types using most permissive: Anzo looks at the inferred types in both schemas and chooses the type that covers all inputs. In most cases Anzo sets the type to String.
      • Add a new property with a different type: If Anzo encounters a type mismatch, it adds a new property with the new type to the existing model.

   Note: When associating column names in the new schema with the existing model, the match is case-insensitive. Anzo matches the names based on the spelling. For example, "myInt" matches "MYint."

8. Click Save. Anzo creates a pipeline (or updates the existing one) and generates or updates the model and mappings according to the options you specified.
9. In the main navigation menu under Onboard, click Structured Data. Then click the Pipelines tab.
10. Click the name of the pipeline to run. Anzo displays the pipeline overview screen. For example:

    

11. If you would like to see the jobs that Anzo created for this data source, click the Jobs tab. The jobs are listed on the left side of the screen. A job exists for each of the tables that were imported. For example:

    

12. To run all of the jobs, click the Publish button at the top of the screen. To publish a subset of the jobs, select the checkbox next to each job that you want to run and then click the Publish button below the list of jobs. Anzo runs the pipeline and generates the resulting RDF files in a new subdirectory under the specified base directory for the data store.

When the pipeline finishes, you can generate metrics on the graph data to start exploring the data. See Generating Graph Data Metrics for more information. You can also add the new data set to a graphmart and load it to AnzoGraph so that you can explore and analyze the data using Hi-Res Analytics. See Creating a Graphmart for instructions.

Follow the instructions below to ingest data for a schema that has an associated data dictionary.

1. In the Anzo console, expand the Onboard menu and click Structured Data. Anzo displays the Data Sources screen, which lists the available data sources. For example:

   

2. On the Data Sources screen, click the name of the data source for which you want to ingest data. Anzo displays the Tables screen for the source. For example:

   

3. Click the Ingest With Dictionary button to build the model and mappings from the data dictionary. If the source has more than one schema, Anzo displays the select schema dialog box. In the drop-down list, select the schema to use, and then click OK. For example:

   

   Anzo opens the Ingest With Dictionary dialog box, which lists the dictionary to use and automatically populates the Anzo data store if only one store is configured. For example:

   

4. If necessary, click the Anzo Data Store field and select the Anzo data store for this pipeline. For information about creating an Anzo data store, see Creating an Anzo Data Store.
5. Click the ETL Engine field and select the ETL engine to use for this new pipeline.
6. By default, Anzo enables the Select all tables radio button to ingest the data for all tables in the schema. If you do not want to add all tables, click the Custom select radio button and then select each of the tables to add.
7. To view model and dictionary options for this pipeline, expand the Advanced section of the dialog box.
   • If this source has not been previously ingested and no model exists, Anzo displays the following advanced options:

     

     • Create New Model: Anzo will create a new model and you have the option to specify a custom Schema Ontology URI.
     • Schema Ontology URI: The URI for the data model. When the Schema Ontology URI field is blank, Anzo generates the model URI for this schema with the following format:

       http://cambridgesemantics.com/ont/autogen/xx/schema_name

       Where xx is a hash snippet based on the model's globally unique identifier (GUID). If you want Anzo to use a specific URI format when creating the model for this schema, you can specify that URI in the Schema Ontology URI field. For example, you can specify a URI such as http://mycompany.com.ontology/movies, which results in a model URI of http://mycompany.com.ontology/movies.

       Important: Make sure that you type a unique Schema Ontology URI for each schema. Schemas with the same model URI overwrite each other when the pipelines are run.

     • Include Unmatched Concepts: This option specifies whether to ingest new data that does not map to the concepts that are defined in the dictionary. For example, the dictionary includes a class concept with 10 properties that map to columns in the schema. The new data contains 15 columns, 5 of which are not mapped to properties in the dictionary. If Include Unmatched Concepts is enabled, Anzo will ingest the data for all 15 columns and update the model to include the 5 unmatched properties. The 5 new properties are added to a new subclass of the class that is defined in the dictionary. If Include Unmatched Concepts is disabled, Anzo will ingest only the 10 columns that match the concepts in the dictionary and will not add new properties to the model.
   • If this source has been previously ingested and a model exists, Anzo displays the following advanced options:

     

     • Create New Model: Select this option if you want Anzo to create a new model. Selecting Create New Model presents the option to specify a custom Schema Ontology URI.
     • Schema Ontology URI: The URI for the data model. When the Schema Ontology URI field is blank, Anzo generates the model URI for this schema with the following format:

       http://cambridgesemantics.com/ont/autogen/xx/schema_name

       Where xx is a hash snippet based on the model's globally unique identifier (GUID). If you want Anzo to use a specific URI format when creating the model for this schema, you can specify that URI in the Schema Ontology URI field. For example, you can specify a URI such as http://mycompany.com.ontology/movies, which results in a model URI of http://mycompany.com.ontology/movies.

       Important: Make sure that you type a unique Schema Ontology URI for each schema. Schemas with the same model URI overwrite each other when the pipelines are run.

     • Use Existing Model: Select this option if you want to reuse the existing model. Anzo displays the name of the model that will be used and presents additional model-related options at the bottom of the screen.
     • Include Unmatched Concepts: This option specifies whether to ingest new data that does not map to the concepts that are defined in the dictionary. For example, the dictionary includes a class concept with 10 properties that map to columns in the schema. The new data contains 15 columns, 5 of which are not mapped to properties in the dictionary. If Include Unmatched Concepts is enabled, Anzo will ingest the data for all 15 columns and update the model to include the 5 unmatched properties. The 5 new properties are added to a new subclass of the class that is defined in the dictionary. If Include Unmatched Concepts is disabled, Anzo will ingest only the 10 columns that match the concepts in the dictionary and will not add new properties to the model.
     • Replace Entire Ontology: Selecting this option means that Anzo deletes all entities from the existing model and recreates them. The model that results from the current ingestion process will contain only the data from the current process. For example, if a previous run generated a model that contains classes A, B, and C, and the current data contains Classes C, D, and E, selecting Replace Entire Ontology results in a model that contains only classes C, D, and E. If Replace Entire Ontology is NOT selected, the resulting model will contain classes A, B, C, D, and E.
     • Regenerate Mappings and Jobs: Selecting this option means that Anzo deletes all entities from the existing mappings and jobs and recreates them. The artifacts that result from the current ingestion process will contain only the data from the current process. For example, if a previous run generated mappings and jobs that contain tables A and B and the current run is ingesting tables C and D, selecting Regenerate Mappings and Jobs results in artifacts that contain only tables C and D. If Regenerate Mappings and Jobs is NOT selected, the resulting artifacts contain tables A, B, C, and D.
8. Click Save. Anzo creates a pipeline (or updates the existing one) and generates or updates the model and mappings according to the options you specified.
9. In the main navigation menu under Onboard, click Structured Data. Then click the Pipelines tab.
10. Click the name of the pipeline to run. Anzo displays the pipeline overview screen. For example:

    

11. If you would like to see the jobs that Anzo created for this data source, click the Jobs tab. The jobs are listed on the left side of the screen. A job exists for each of the tables that were imported. For example:

    

12. To run all of the jobs, click the Publish button at the top of the screen. To publish a subset of the jobs, select the checkbox next to each job that you want to run and then click the Publish button below the list of jobs. Anzo runs the pipeline and generates the resulting RDF files in a new subdirectory under the specified base directory for the data store.

When the pipeline finishes, you can generate metrics on the graph data to start exploring the data. See Generating Graph Data Metrics for more information. You can also add the new data set to a graphmart and load it to AnzoGraph so that you can explore and analyze the data using Hi-Res Analytics. See Creating a Graphmart for instructions.