Creating an Unstructured Pipeline

Follow the instructions below to create a new Unstructured Pipeline.

  1. In the Anzo application, expand the Onboard menu and click Unstructured Data. Anzo displays the Pipeline screen, which lists any existing unstructured pipelines. For example:

  2. Click the Add Unstructured Pipeline button and select Distributed Unstructured Pipeline. Anzo opens the Create Distributed Unstructured Pipeline dialog box. For example:

  3. In the Title field, type a name for the Pipeline.

    This Title serves as a key to identify this Pipeline and its corpus in multiple contexts. Specify a Title that is unique and stable. The Pipeline's corpus Dataset name is derived from this Title.

  4. Type an optional description for the pipeline in the Description field.
  5. If necessary, click the Target Anzo Data Store field and select the Data Store for this Pipeline. For instructions on creating an Anzo Data Store, see Creating an Anzo Data Store.
  6. If the environment is configured for dynamic Kubernetes-based deployments of the Anzo Unstructured infrastructure, select the Deploy Unstructured Infrastructure Dynamically checkbox and leave the Static Elasticsearch Config field blank.
  7. If necessary, click the Static Elasticsearch Config field and select the Elasticsearch connection to use for this Pipeline. If you use dynamic deployments to deploy Elasticsearch instances on-demand, leave this field blank. Anzo will prompt the user to choose a Cloud Location when the Pipeline is run. For instructions on creating a static Elasticsearch connection, see Connecting to Elasticsearch.
  8. Click Save to create the Pipeline. Anzo displays the pipeline Overview screen. For example:

    A Pipeline saves automatically and constantly undergoes validation to make sure that it is is valid based on the current configuration. Anzo displays validation issues in red on the top of the screen. The warnings will disappear as you add components to the Pipeline.

  9. If necessary, click Advanced to configure the advanced Pipeline settings.
  10. Click the Crawlers tab and follow the substeps below to add a Crawler to the Pipeline:
    1. Click Add Input. Anzo opens the Add Component dialog box. The New tab lists the default crawlers and the Existing Components tab lists crawlers that have been previously configured for other Pipelines.

    2. To add a new Crawler, select the Crawler. To add an existing Crawler to the pipeline, click the Existing Components tab and select a Crawler. The list below describes each of the Crawlers:
      • File Based Dataset Crawler: Include this crawler to process data from a file-based linked data set (FLDS) on a file store.
      • Filesystem Crawler: Include this crawler to process documents, such as email messages, PDF, XML, PowerPoint, Excel, OneNote, or Word files, and images, that are available on a file store.
      • Graphmart RDF Crawler: Include this crawler to process RDF in an online Graphmart or specific Data Layer.
      • Local Volume Dataset Crawler: Include this crawler to process RDF data that is stored as a linked data set (LDS) in an Anzo journal.
    3. After selecting a Crawler, click OK. Anzo opens the Create dialog box for that Crawler so that you can configure it. The list below provides details about the settings for each Crawler. Click a Crawler name to view the details for that component:

      This section describes the settings that are available on the Create File Based Dataset Crawler screen:

      • Title: Required field that specifies the unique name for this Crawler.
      • Description: Optional field that provides a description of this Crawler.
      • Backing Dataset: Required field that specifies the Anzo Dataset to crawl.
      • Backing Ontology: Required field that specifies the Model for the backing Dataset.
      • RDF Resource Type: Required field that specifies the resource type or class of data to target with this Crawler.
      • Link Property: Optional field that specifies any link properties to crawl. A link property is a property whose value identifies the location of a linked document. When linked properties are specified, the Crawler will crawl the linked documents. For example, in the triples below, fileLocation is a link property:
        <urn://someUnstructuredDocument> <urn://someProperty> "file metadata" ;
<urn://fileLocation> "/path/to/file.pdf" .

        In typical use cases, this Crawler is configured to define either a Link Property or a Content Property but not both.

      • Content Property: Optional field that identifies any content properties to crawl. A content property is a property whose value is a string literal, and you want the Crawler to crawl and annotate those strings. For example, in the triples below, longDescription is a content property:
        <urn://someUnstructuredDocument> <urn://someProperty> "file metadata" ;
        <urn://longDescription> "this is some interesting, likely long, unstructured text
with a lot of information, and I want it to be annotated" .
      • Base Path Connection: Required field whose value depends on whether you specified a Link Property or a Content Property:
        • If a Link Property was specified, the Base Path Connection is the base path to use for resolving relative file paths in the Link Property values. For example, using the example triples: 
          <urn://someUnstructuredDocument> <urn://someProperty> "file metadata" ;
<urn://fileLocation> "/path/to/file.pdf" .

          The <urn://fileLocation> value of /path/to/file.pdf could be a relative path to a location like s3://location/bucket/path/to/file.pdf or /opt/anzoshare/data/path/to/file.pdf. Therefore, the Base Path needs to be specified to resolve any relative paths and locate the linked documents.

        • If a Content Property was specified, the Base Path Connection is a directory on the File Store where the Crawler can save a copy of the Content Property strings for the Anzo Unstructured Worker instances. Saving the content to a shared file location avoids the overhead of sending the strings to the workers over the network.

      This section describes the settings that are available on the Create Filesystem Crawler screen:

      • Title: Required field that specifies the unique name for this crawler.
      • Description: Optional field that provides a description of this crawler.
      • VFS Crawl Location: Required field that specifies the virtual file system crawl location. Click the field to open the File Location dialog box:

        On the left side of the screen, select the storage location for the files to crawl. On the right side of the screen, navigate to the directory that contains the files. Select a directory, and then click OK.

      • Crawl subfolders: Optional field that specifies whether to crawl the subdirectories under the VFS Crawl Location. To crawl the subdirectories, select the Crawl subfolders checkbox. To ignore subdirectories, clear the Crawl subfolders checkbox.

      This section describes the settings that are available on the Create Graphmart RDF Crawler screen:

      • Title: Required field that specifies the unique name for this Crawler.
      • Description: Optional field that provides a description of this Crawler.
      • Backing Graphmart: Optional field that specifies the Graphmart to crawl. To configure the Crawler to crawl at the Graphmart level, select one or more Graphmarts in the Backing Graphmart field and leave the Backing Layer field blank.
      • Backing Layer: Optional field that specifies the Data Layer or Layers that you want the Pipeline to crawl. To crawl specific Data Layers and not an entire Graphmart, make sure that you leave the Backing Graphmart field blank and select the Layers to crawl in the Backing Layer field. If you specify both a Backing Graphmart and a Backing Layer, the Backing Graphmart value supersedes Backing Layer value, resulting in the entire Graphmart being crawled.
      • Backing Ontology: Required field that specifies the Model for the Backing Graphmart or Data Layer.
      • RDF Resource Type: Required field that specifies the resource type or class of data to target with this Crawler.
      • Link Property: Optional field that specifies any link properties to crawl. A link property is a property whose value identifies the location of a linked document. When linked properties are specified, the Crawler will crawl the linked documents. For example, in the triples below, fileLocation is a link property:
        <urn://someUnstructuredDocument> <urn://someProperty> "file metadata" ;
<urn://fileLocation> "/path/to/file.pdf" .

        In typical use cases, this Crawler is configured to define either a Link Property or a Content Property but not both.

      • Content Property: Optional field that identifies any content properties to crawl. A content property is a property whose value is a string literal, and you want the Crawler to crawl and annotate those strings. For example, in the triples below, longDescription is a content property:
        <urn://someUnstructuredDocument> <urn://someProperty> "file metadata" ;
        <urn://longDescription> "this is some interesting, likely long, unstructured text
with a lot of information, and I want it to be annotated" .
      • Base Path Connection: Required field whose value depends on whether you specified a Link Property or a Content Property:
        • If a Link Property was specified, the Base Path Connection is the base path to use for resolving relative file paths in the Link Property values. For example, using the example triples: 
          <urn://someUnstructuredDocument> <urn://someProperty> "file metadata" ;
<urn://fileLocation> "/path/to/file.pdf" .

          The <urn://fileLocation> value of /path/to/file.pdf could be a relative path to a location like s3://location/bucket/path/to/file.pdf or /opt/anzoshare/data/path/to/file.pdf. Therefore, the Base Path needs to be specified to resolve any relative paths and locate the linked documents.

        • If a Content Property was specified, the Base Path Connection is a directory on the File Store where the Crawler can save a copy of the Content Property strings for the Anzo Unstructured Worker instances. Saving the content to a shared file location avoids the overhead of sending the strings to the workers over the network.

      This section describes the settings that are available on the Create Local Volume Dataset Crawler screen:

      • Title: Required field that specifies the unique name for this Crawler.
      • Description: Optional field that provides a description of this Crawler.
      • Backing Dataset: Required field that specifies the Anzo Dataset to crawl.
      • Backing Ontology: Required field that specifies the Model for the backing Dataset.
      • RDF Resource Type: Required field that specifies the resource type or class of data to target with this Crawler.
      • Link Property: Optional field that specifies any link properties to crawl. A link property is a property whose value identifies the location of a linked document. When linked properties are specified, the Crawler will crawl the linked documents. For example, in the triples below, fileLocation is a link property:
        <urn://someUnstructuredDocument> <urn://someProperty> "file metadata" ;
<urn://fileLocation> "/path/to/file.pdf" .

        In typical use cases, this Crawler is configured to define either a Link Property or a Content Property but not both.

      • Content Property: Optional field that identifies any content properties to crawl. A content property is a property whose value is a string literal, and you want the Crawler to crawl and annotate those strings. For example, in the triples below, longDescription is a content property:
        <urn://someUnstructuredDocument> <urn://someProperty> "file metadata" ;
        <urn://longDescription> "this is some interesting, likely long, unstructured text
with a lot of information, and I want it to be annotated" .
      • Base Path Connection: Required field whose value depends on whether you specified a Link Property or a Content Property:
        • If a Link Property was specified, the Base Path Connection is the base path to use for resolving relative file paths in the Link Property values. For example, using the example triples: 
          <urn://someUnstructuredDocument> <urn://someProperty> "file metadata" ;
<urn://fileLocation> "/path/to/file.pdf" .

          The <urn://fileLocation> value of /path/to/file.pdf could be a relative path to a location like s3://location/bucket/path/to/file.pdf or /opt/anzoshare/data/path/to/file.pdf. Therefore, the Base Path needs to be specified to resolve any relative paths and locate the linked documents.

        • If a Content Property was specified, the Base Path Connection is a directory on the File Store where the Crawler can save a copy of the Content Property strings for the Anzo Unstructured Worker instances. Saving the content to a shared file location avoids the overhead of sending the strings to the workers over the network.
    4. When you have finished configuring the crawler, click Save. Anzo adds the crawler to the pipeline and returns to the Crawlers screen. For example:

    5. If you want to change the crawler configuration, click the Edit icon () for the crawler and modify the settings as needed. If you want to add another crawler to the pipeline, repeat substeps a – d.
  11. Click the Annotators tab and follow the substeps below to add an annotator to the pipeline:
    1. Click Add Output to select an annotator. Anzo opens the Add Component dialog box.

      In the Add Component dialog box, the New tab lists the default annotators and the Existing Components tab lists annotators that have been previously configured for other pipelines.

    2. To add a new annotator to the pipeline, click the annotator name to select it. To add an existing annotator to the pipeline, click the Existing Components tab, and then select an annotator. The list below describes each of the default annotators:
      • Custom Relationship Annotator: Include this annotator to map relationships between annotations based on the number of characters between the annotations.
      • External Service Annotator: Include this annotator to hit an HTTP endpoint that provides annotations.
      • Keyword and Phrase Annotator: Include this annotator to create annotations based on the phrases that you specify.
      • Knowledgebase Annotator: Include this annotator to link structured and unstructured data by finding instances in data layers, graphmarts, or Anzo linked datasets. Based on the names and aliases of entities present or patterns that are indicative of the entities, this annotator marks up the documents with the structured entities linked.
      • Regex Annotator: Include this annotator to use regular expression rules to identify entities such as email addresses, URLs, phone numbers, or any other entity that can be matched using a regular expression.
      • Semantria Annotator: Include this annotator to use the Semantria web service to find entities, sentiment, and topics in documents. It requires an Semantria API access key from Lexalytics.
      • Significant Phrases Annotator: Include this annotator to annotate statistically significant words and phrases.
    3. After selecting an annotator, click OK. Anzo opens the Create dialog box for the component. Complete the fields to configure the annotator. The list below provides details about the settings for the annotators that are typically used in pipelines. Click an annotator name to view the details for that component:

      This section describes the settings that are available on the Create External Service Annotator screen:

      • Title: Required field that specifies the unique name for this annotator.
      • Description: Optional field that provides a description of this annotator.
      • HTTP Request Config: Required field that specifies the HTTP source object that contains the URL and method to use when sending data for annotations.
      • Document ID Response Path: Required field that specifies where to find the document ID in the response.
      • Entity Class Path: Required field that specifies the class URI for an annotation.
      • Entity Name Path: Required field that specifies the annotation object name path.

      This section describes the settings that are available on the Create Knowledgebase Annotator screen:

      • Title: Required field that specifies the unique name for this annotator.
      • Description: Optional field that provides a description of this annotator.
      • Backing Layer: Optional field that specifies the data layer or layers to annotate.

        The Backing Layer and Backing Graphmart fields are treated independently. Layers that you select do not have to be part of the graphmart that you specify in Backing Graphmart. And specifying a layer does not mean that you must select a Backing Graphmart. However, any layers or graphmarts that you select must contain classes and properties from the Backing Ontology or the data will not be annotated.

      • Backing Graphmart: Optional field that specifies the graphmart or graphmarts to annotate.

        If you want the annotator to run against a linked dataset or Anzo knowledgebase instead of a data layer or graphmart, leave the Backed Layer and Backed Graphmart fields blank. After saving the pipeline, you can edit the pipeline and specify a Backed Dataset at that time.

      • Backing Ontology: Required field that specifies the model for the backing data layers and/or graphmart. Click the field and select a model from the drop-down list.
      • Term Class: Required field that specifies the class of data for the annotation.
      • Term Label Property: Required field that lists the property for which to find entities.
      • Term Identifying Properties: Required field that specifies the properties that contain names, aliases, or other identifiers by which you want to find entities.

      This section describes the settings that are available on the Create Regex Annotator screen:

      • Title: Required field that specifies the unique name for this annotator.
      • Description: Optional field that provides a description of this annotator.
      • Regular Expression Rule: Required field the lists the regular expression rules for this annotator. To add a rule, click the plus icon () in the field. Anzo opens the Create Regular Expression Rule dialog box where you can define the rule:

        • Title: Required field that specifies the name of the rule.
        • Description: Optional field that describes the rule.
        • Regular Expression: Required field that specifies the regular expression to use for finding matching entities.
        • Class Structure: Required field that specifies the class structure for the entities in the format group_number:class_name. For example, 0:person,1:Company.

      For information about the options that are presented when you edit a Regex Annotator, refer to the Field Summary section in the Java Regex Compiler documentation.

    4. When you have finished configuring the annotator, click Save. Anzo adds the annotator to the pipeline and returns to the Annotators screen. For example:

    5. If you want to change the annotator configuration, click the Edit icon () for the annotator and modify the settings as needed. If you want to add another annotator to the pipeline, repeat substeps a – d.
  12. When you have finished adding crawlers and annotators to the pipeline, click the Run Pipeline button to run the pipeline.

The process can take several minutes to complete. You can click the Progress tab to view details such as the pipeline status, runtime, number of documents processed, and errors. For example:

When the Pipeline finishes, this run of the Pipeline becomes the Managed Edition. The Managed Edition always contains the latest successfully published data for all of the jobs in the Pipeline. If one or more of the jobs failed, those jobs are excluded from the Edition. If you publish the failed jobs at a later date or you create and publish additional jobs in the Pipeline, the data from those jobs is also added to the Managed Edition. For more information about Editions, see Managing Dataset Editions.

The new Dataset also becomes available in the Dataset catalog. From the catalog, you can generate graph Data Profiles and create Graphmarts. See Blending Data for next steps.

Related Topics