Connecting to an ETL Engine

The default Anzo installation includes a pre-configured local Spark ETL engine and Sparkler ETL compiler. Sparkler is Cambridge Semantics' SPARQL-driven ETL compiler that supports the ingestion of wide CSV files with a large number of columns and increases performance over the Spark Scala-based compiler in many cases. The topics in this section provide instructions for changing the configuration of the local engines or connecting to an alternate Spark ETL engine or Sparkler compiler.

Note Before you can configure Informatica or Pentaho ETL engines, you must import the appropriate Cambridge Semantics bundle. Contact Cambridge Semantics Support to obtain the bundles.
  1. In the Anzo console, expand the Administration menu and click ETL Engine Config.
  2. On the ETL Engine Config screen, click the Create button and select the type of ETL engine that you want to configure. Anzo displays the Create ETL Engine Config screen. For example:

  3. On the Create ETL Engine Config screen, type a Title and optional Description for the engine. Then click Save. Anzo displays the Details view for the new engine. For example:

  4. The options that appear depend on the type of ETL engine you chose. Enter the required details for the engine. Hover the pointer over an options and click the Edit icon () to modify any of the options. Click the check mark icon () to save changes to an option, or click the X icon () to clear the value for an option.

    Click an engine type below to see a description of the settings for that engine.

    • Repo Server Host: Leave this field blank.
    • Repo Server Port: Leave this field blank.
    • Admin Username: Not currently used.
    • Admin Password: Not currently used.
    • Compile tab: The Compile settings control where Anzo saves the compiled Scala .jar files for the Spark job.
      • Remote Server: The host name or IP address of the server where the compilation will be performed.
      • Target Folder: The path and directory on the server where Anzo can stage temporary artifacts created during the compilation and upload process. The location must be a valid path on the Anzo server that the user running the ETL job has access to.
    • Deploy tab: The Deploy step is performed after the job is compiled locally and before the job is submitted to Spark. The Deploy settings control how and where the job's .jar files will be copied from the Anzo server to a file system that Spark can access.
      • Deployment Working Dir: The directory that the Anzo server should use when executing the deploy commands.
      • Deploy Command: The command line script that the deploy step should run.
    • Run tab:
      • Job Runner Endpoint: The HTTP endpoint used to reach the Livy server. For example, when using the local Anzo Spark engine, the endpoint is localhost:8998.
      • SDI Jobs Dir: The file system location where the Spark engine will look for the compiled .jar files.
      • SDI Dependencies Dir: The file system location where the Spark engine will look for the dependency .jar files, sdi-full-deps.jar and sdi-deps.jar. These files can be copied to the Spark master node from the installation_path/Anzo/Server/data/sdiScripts/Spark_version/compile/dependencies-lib directory on the Anzo server.
      • Additional Jars: For relational database sources, the file system location for the JDBC driver .jar file or files that are used to connect to the databases. For multiple jar files, specify a comma-separated list. Do not include a space after the commas.
      • Execute Locally: Select this option for local Spark engines on the Anzo server. Make sure this option is not selected when using a remote Spark server.
      • Do Callback: Select this option when you want Anzo to create a new data set in the Dataset catalog and generate load files for the graph source.
      • Run with Yarn: Employs the Spark YARN cluster manager when running ETL jobs.
      • Callback URL: When Do Callback is selected, enter one of the following URLs: 
        http://Anzo_server_hostname_or_IP:Anzo_application_HTTP_port/anzoclient/call
        https://Anzo_server_hostname_or_IP:Anzo_application_HTTPS_port/anzoclient/call
    • Publish tab: This tab controls the action of the Publish All button when running jobs.
    • Sharing tab: Enables you to share or restrict access to this ETL engine.
    • Description: Description of this engine.
    • Run tab:
      • Remote Server Name: The host name or IP address of the server where the compilation will be performed.
      • Job Runner Endpoint: The HTTP endpoint used to reach the Livy server. For example, when using the local Anzo Sparkler engine, the endpoint is localhost:8998.
      • Target Folder Name: The path and directory on the host where temporary artifacts can be created during the compilation and upload process. The location must be a valid path on the server that the user running the ETL job has access to.
      • Sparkler Home: The path and directory where the Sparkler compiler is installed on the host server.
      • SDI Dependencies Dir: The file system location where the Sparkler engine will look for the dependency .jar files, sdi-full-deps.jar and sdi-deps.jar. These files can be copied to the Sparkler host server from the install_path/Anzo/Server/data/sdiScripts/Spark_version/compile/dependencies-lib directory on the Anzo server.
      • Additional Jars: For relational database sources, the file system location for the JDBC driver .jar file or files that are used to connect to the databases. For multiple jar files, specify a comma-separated list. Do not include a space after the commas.
      • Execute Locally: Select this option for local Sparkler engines on the Anzo server. Make sure this option is not selected when using a remote Sparkler server.
      • Do Callback: Select this option when you want Anzo to create a new data set in the Dataset catalog and generate load files for the graph source.
      • Run with Yarn: Employs the Spark YARN cluster manager when running ETL jobs.
      • Callback URL: When Do Callback is selected, enter one of the following URLs: 
        http://Anzo_server_hostname_or_IP:Anzo_application_HTTP_port/anzoclient/call
        https://Anzo_server_hostname_or_IP:Anzo_application_HTTPS_port/anzoclient/call
    • Advanced tab: The options on this tab enable users with advanced Spark expertise to customize the values that are passed to Spark.
      • Enable CSV Error Reporting: Controls whether detailed CSV errors are displayed in the Anzo user interface.
      • Run As User: Specifies the user to impersonate when starting the Livy session.
      • Max Graph Output File Size Default (Bytes): The maximum number of bytes to limit graph output files to.
      • Max Input File Partition Size (Bytes): The maximum number of bytes to pack into a partition when reading files. Maps to the spark.files.maxPartitionBytes Spark configuration setting.
      • Spark Job Driver Cores: The number of cores to use for the driver process. Maps to the spark.driver.cores Spark configuration setting.
      • Spark Job Driver Memory: The amount of memory to use for the driver process. Maps to the spark.driver.memory Spark configuration setting.
      • Number of Executors Per Spark Job: The number of executors to request per Spark job. Maps to the spark.executor.instances Spark configuration setting.
      • Spark Job Cores Per Executor: The number of cores to use on each executor. Maps to the spark.executor.cores Spark configuration setting.
      • Spark Job Memory Per Executor: The amount of memory to use per executor process. Maps to the spark.executor.memory Spark configuration setting.
      • Off Heap Size (Bytes): The amount of memory in bytes that can be used for off-heap allocation. Maps to the spark.memory.offHeap.size Spark configuration setting.
      • Job Dependencies (Maven Package Coordinate): The comma-separated list of Maven jar coordinates to include on the driver and executor classpaths. Maps to the spark.jars.packages Spark configuration setting.
      • Maven Package Excludes: To avoid dependency conflicts, this is the comma-separated list of groupId:artifactId to exclude while resolving the dependencies listed in spark.jars.packages. Maps to the spark.jars.excludes Spark configuration setting.
      • Maven Repositories: A comma-separated list of additional remote repositories to search for the maven coordinates from the Job Dependencies setting. Maps to the spark.jars.repositories Spark configuration setting.
      • Spark Job Deploy Mode (Livy Config has Precedence): The deploy mode of the Spark driver program. If this value is set in the Livy configuration, the Livy value takes precedence. Maps to the spark.submit.deployMode Spark configuration setting.
    • Publish tab: This tab controls the action of the Publish All button when running jobs.
    • Sharing tab: Enables you to share or restrict access to this Sparkler engine.
    NoteBy default, the Sparkler Livy RSC client is configured to use a range of 10 internal ports for running jobs. Cambridge Semantics recommends that you increase the range of ports by 100. To do so, modify the livy.rsc.launcher.port.range value in the livy-client.conf file. If you use the embedded Anzo Sparkler compiler, the file is in the install_path/Server/spark/csi-livy-spark/conf directory. Set livy.rsc.launcher.port.range = 10000~10110 and restart the Livy server.

When the configuration is complete, Anzo provides this ETL engine as a choice to select when ingesting data and configuring pipelines.