Managing the Automatic Restart Feature

If AnzoGraph shuts down unexpectedly, the system manager automatically restarts the database and evaluates the queries that were running at the time of the shutdown. This topic describes the process that occurs when AnzoGraph automatically restarts and provides information about the configuration settings that control the functionality.

Automated Restart Procedure

The steps below describe what occurs during the automatic restart process after AnzoGraph has crashed:

  1. The system manager restarts the database in safe mode. In safe mode, AnzoGraph is locked to users and returns the following message if a user runs a query: "AnzoGraph is running in safe-mode. Cannot execute query." In addition, running azgctl -status to check the status of the database returns the message "AnzoGraph is running in safe-mode." If persistence is enabled, the data that was in memory at the time of the crash is reloaded into memory.
  2. While in safe mode, AnzoGraph runs any queries that were inflight at the time of the crash. By executing the queries that were running, AnzoGraph tries to determine if the crash was directly caused by one of the inflight queries.
  3. Depending on the outcome of running the inflight queries, AnzoGraph does the following:
    • If all inflight queries run to completion in safe mode, they are all added to the grey list. In addition, each query is copied to a file named <query_ID>.txt in the <install_path>/internal/auto_restart/<timestamp>/grey_listed directory.
      NoteWhen all inflight queries complete successfully, that means it is unlikely that any one of the queries on its own is the culprit for the crash. However, all of the queries are grey listed because it is possible that the combination of queries run concurrently could have caused the crash.
    • If any of the inflight queries fail or crash the database in safe mode, those queries are added to the black list. In addition, each query is copied to a file named <query_ID>.txt in the <install_path>/internal/auto_restart/<timestamp>/black_listed directory.

      If an inflight query fails, none of the inflight queries are grey-listed. Instead, the failed queries are added to the black list.

    • If AnzoGraph runs a query in safe mode and cannot determine if it should be added to the grey or black list, those queries are copied to a file named <query_ID>.txt in the <install_path>/internal/auto_restart/<timestamp>/unanalyzed_queries directory.
    NoteSaving copies of the grey- or black-listed queries is purely informational. It helps users view which queries were running at the time of the crash. A future AnzoGraph release will automatically block black-listed queries from running.
  4. After the inflight queries have been run, AnzoGraph restarts the database, loads the persisted data back into memory, and returns the system to normal operation.

Automated Restart System Settings

The automatic restart feature is controlled by the following three settings in <install_path>/config/settings.conf:

  • auto_restart_max_attempts: This setting specifies the number of times the system manager should attempt to start the database after a crash. The default value is 5, which means the system manager will attempt to restart the database a maximum of 5 times. Changing auto_restart_max_attempts to 0 disables the auto-restart feature.
  • auto_restart_time: This setting specifies the number of seconds to spend attempting to restart the database. If all attempts fail and this time limit is reached, the system manager stops trying to restart the database. The default value is 600, which means that the system manager will attempt to restart the database for a maximum of 600 seconds (10 minutes).
  • auto_restart_directory: This setting specifies the base location of the auto_restart directory, which contains the black_listed, grey_listed, and unanalyzed_queries directories. The default value is <install_path>/internal.
Important Changing any of the auto_restart settings above requires a restart of the system management daemon, azgmgrd, as well as the database. See Starting and Stopping AnzoGraph for instructions.
Related Topics