Generating a Knowledge Graph from Non-RDF Sources

With no mapping required, the Graph Data Interface (GDI) can automatically generate a graph and an ontology for a data source. By running a relatively simple SPARQL query to invoke the RDF and Ontology Generators, the GDI determines the structure of a data source and automatically generates the necessary RDF statements.

Invoking the Generators is preferable to producing a hand-written query, especially when the structure of the data is very complex, such as a JSON data source with many inner repeating structures or a database with many tables and keys. When the source contains complex structures, the GDI will generate only the required statements and avoid cross-products, optimizing query execution and memory usage. In addition, the GDI Generator parallelizes the load across the AnzoGraph DB cluster so that a data source (such as a database) can be ingested with a single query.

This topic provides details about invoking the GDI RDF and Ontology Generators. The Generators can be used with all of the supported data source types.

• GDI Generator Query Syntax
• GDI Generator Example Queries

GDI Generator Query Syntax

The following query syntax shows the structure of a GDI Generator query. The clauses, patterns, and placeholders that are links are described below.

# PREFIX Clause
PREFIX s:    <http://cambridgesemantics.com/ontologies/DataToolkit#>
PREFIX owl:  <http://www.w3.org/2002/07/owl#>
PREFIX rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX xsd:  <http://www.w3.org/2001/XMLSchema#>

#Result Clause
INSERT {
   GRAPH <target_graph> {
     ?s ?p ?o .
  }
}
WHERE
{
	SERVICE <http://cambridgesemantics.com/services/DataToolkit>
    {
      ?data a s:source_type ;      
# Based on the source_type, additional connection and input parameters are
# available. The options below are valid for all sources. For source-related
# options, see Source-Specific Properties in the GDI property reference. 
        s:url "string" ;
        [ s:model "class_name_for_this_source" ; ]
        [ s:username "string" ; ]
        [ s:password "string" ; ]
        [ s:timeout int ; ]
        [ s:maxConnections int ; ]
        [ s:batching boolean | int ; ]
        [ s:concurrency int | [ list_of_properties ] ; ]
        [ s:rate int | "string" ; ]
        [ s:locale "string" ; ]
        [ s:sampling int ; ]
        [ s:selector "string" | [ list ] ; ]
        [ s:key ("string") ; ]
        [ s:reference [ s:model "string" ; s:using ("string") ]
        [ s:formats [ datatype_formatting_options ] ; ]
        [ s:normalize boolean | [ source_normalization_rules ] ; ]
        [ s:count ?variable ; ]
        [ s:offset int ; ]
        [ s:limit int ; ] .

      # Multiple data sources can be merged if they project a similar set
      # of output variables. Make sure each source has a unique subject variable.
  
        [ ?unique_variable a s:source_type ;
          ...
        . ]

      ?rdf a s:RdfGenerator, s:OntologyGenerator ;
        s:as (?s ?p ?o);
        s:ontology ontology_uri ;
        s:base base_uri ;
        [ s:normalize boolean | [ global_normalization_rules ] ; ]
        .
      # Additional clauses such as BIND, FILTER
   }
}

PREFIX s:    <http://cambridgesemantics.com/ontologies/DataToolkit#>
PREFIX owl:  <http://www.w3.org/2002/07/owl#>
PREFIX rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX xsd:  <http://www.w3.org/2001/XMLSchema#>

INSERT {
   GRAPH <target_graph> {
     ?s ?p ?o .
  }
}

?data a s:DbSource ;
  s:url "{{@Somedb.url}}" ;
  s:username "{{@Somedb.user}}" ;
  s:password "{{@Somedb.password}}" ;

s:concurrency [
  s:limit 24 ;
  s:nodes 4 ;
  s:executorsPerNode 8 ;
] ;

s:rate "90/minute" ;
s:rate "90 per minute" ;
s:rate "200000 every week" ;
s:rate "10000 every 6 hours" ;

GDI Generator Example Queries

This section includes sample queries that may be useful as a starting point for writing your own RDF and Ontology Generator queries.

• Basic Query that Generates RDF and Ontology for a JSON File
• Basic Query that Generates an Ontology for a Directory of CSV Files
• Query that Normalizes and Generates RDF and Ontology for a Database
• Query with Query Context that Normalizes and Generates RDF and Ontology for a Database
• Query for Multiple Sources that Generates RDF and Ontology with Resource Templates and Object Properties

Basic Query that Generates RDF and Ontology for a JSON File

PREFIX s: <http://cambridgesemantics.com/ontologies/DataToolkit#>

INSERT {
   GRAPH <http://anzograph.com/people> {
     ?s ?p ?o .
  }
}
WHERE {
   SERVICE <http://cambridgesemantics.com/services/DataToolkit> {

      ?data a s:FileSource ;
         s:model "People" ;
         s:url "/opt/shared-files/json/people.json" .

      ?rdf a s:RdfGenerator , s:OntologyGenerator ;
         s:as (?s ?p ?o) ;
         s:ontology <http://anzograph.com/ontologies/People> ;
         s:base <http://anzograph.com/data/> .
  }
}

Basic Query that Generates an Ontology for a Directory of CSV Files

PREFIX s:  <http://cambridgesemantics.com/ontologies/DataToolkit#>

INSERT {
   GRAPH <http://anzograph.com/sales> {
     ?s ?p ?o .
  }
}
WHERE {
   SERVICE <http://cambridgesemantics.com/services/DataToolkit> {

      ?data a s:FileSource ;
         s:model "Sales" ;
         s:url "/opt/shared-files/csv/sales" ;
         s:format [
            s:delimiter "," ;
            s:headers true ;
            s:comment "#" ;
            s:quote "\"" ;
            s:maxColumns 22 ;
         ] .

      ?rdf a s:OntologyGenerator ;
         s:as (?s ?p ?o) ;
         s:ontology <http://anzograph.com/ontologies/Sales> ;
         s:base <http://anzograph.com/data/> .
  }
}

Query that Normalizes and Generates RDF and Ontology for a Database

PREFIX s: <http://cambridgesemantics.com/ontologies/DataToolkit#>

INSERT {
   GRAPH <http://anzograph.com/emr> {
      ?s ?p ?o .
  }
}
WHERE {
   SERVICE <http://cambridgesemantics.com/services/DataToolkit> {

      ?data a s:DbSource ;
      s:url "jdbc:mysql://10.11.12.9/emrdbbig" ;
      s:username "root" ;
      s:password "sql1@#" ;
      s:normalize [
         s:model [
            s:removeStart "emr_" ;
            s:words "activity 'patient complaint' medication observation patient specialty study" ;
         ] ;
         s:field [ 
            s:removePartialPrefix true ;
            s:words "provider description start end drug complaint date medication normal code
                     observation product active dose generic route admin strength collection
                     activity home first last status first year birth death directed complex
                     period age flag gender language" ;
         ] ;
      ] .

      ?rdf a s:RdfGenerator , s:OntologyGenerator ;
      s:as (?s ?p ?o) ;
      s:ontology <http://anzograph.com/ontologies/EMR> ;
      s:base <http://anzograph.com/EMR> .
  }
}

Query with Query Context that Normalizes and Generates RDF and Ontology for a Database

The query below references a Query Context to supply the username and password for the database connection.

PREFIX s: <http://cambridgesemantics.com/ontologies/DataToolkit#>

INSERT {
   GRAPH <http://anzograph.com/adventureworks> {
      ?s ?p ?o .
  }
}
WHERE {
   SERVICE <http://cambridgesemantics.com/services/DataToolkit> {

      ?data a s:DbSource ;
         s:url "jdbc:sqlserver://localhost;databaseName=AdventureWorks2012" ;
         s:username "{{@adventureworksdb.username}}" ;
         s:password "{{@adventureworksdb.password}}" ;
         s:schema "Production", "HumanResources", "Person", "Sales", "Purchasing" ;
         s:normalize [ 
            s:model [
               s:localNamePrefix "C_" ;
               s:localNameSeparator "_" ;
               s:match [ s:pattern "(.+)Enlarged" ; s:replace "$1" ] ;
            ] ;
            s:field [
               s:localNamePrefix "P_" ;
               s:localNameSeparator "_" ;
               s:ignore "rowguid ModifiedDate" ;
               s:match (
                  [ s:pattern "(.+)GUID$" ; s:replace "$1" ]
                  [ s:pattern "(.+)ID$" ; s:replace "$1" ]
               ) ;
            ] ;
         ] .

      ?rdf a s:RdfGenerator, s:OntologyGenerator ;
         s:as (?s ?p ?o) ;
         s:ontology <http://anzograph.com/ontologies/AdventureWorks> ;
         s:base <http://anzograph.com/AdventureWorks> .
  }
}

Query for Multiple Sources that Generates RDF and Ontology with Resource Templates and Object Properties

This query also includes global normalization rules for normalizing the data across all sources.

PREFIX s: <http://cambridgesemantics.com/ontologies/DataToolkit#>

INSERT {
   GRAPH <http://anzograph.com/tickets> {
      ?s ?p ?o .
  }
}
WHERE { 
   SERVICE <http://cambridgesemantics.com/services/DataToolkit> {

      ?event a s:FileSource ;
         s:model "event" ;
         s:url  "/opt/shared-files/csv/events.csv" ;
         s:key ("EVENT_ID") .

      ?listing a s:FileSource ;
         s:model "listing" ;
         s:url " /opt/shared-files/csv/listings.csv" ;
         s:key ("LIST_ID") ;
         s:reference [ s:model "event" ; s:using ("EVENT_ID") ; s:key ("EVENT_ID") ] .

      ?date a s:FileSource ;
         s:model "date" ;
         s:url  "/opt/shared-files/csv/event_dates.csv" ;
         s:key ("DATE_ID") ;
         s:reference [ s:model "event" ; s:using ("EVENT_ID") ; s:key ("EVENT_ID") ] .

      ?venue a s:FileSource ;
         s:model "venue" ;
         s:url " /opt/shared-files/csv/venues.csv" ;
         s:key ("VENUE_ID") ;
         s:reference [ s:model "event" ; s:using ("EVENT_ID") ; s:key ("EVENT_ID") ] .
     
      ?sale a s:FileSource ;
         s:model "sale" ;
         s:url " /opt/shared-files/csv/sales.csv" ;
         s:key ("SALE_ID") ;
         s:reference [ s:model "event" ; s:using ("EVENT_ID") ; s:key ("EVENT_ID") ] ;
         s:reference [ s:model "listing" ; s:using ("LIST_ID") ; s:key ("LIST_ID") ] .

      ?rdf a s:RdfGenerator, s:OntologyGenerator ;
         s:as (?s ?p ?o) ;
         s:ontology <http://anzograph.com/tickets> ;
         s:base <http://anzograph.com/data> ;
         s:normalize [ 
            s:all [
               s:casing s:UPPER ;
               s:localNameSeparator "_" ;
            ] ;
         ] .
  }
}