Historical Data

Singularity can optionally persist all task and deployment historical information into a MySQL or PostgreSQL database. This is useful because Mesos does not necessarily keep state forever, nor does it provide a deploy-focused interface for viewing that state. The Singularity API and web application will return historical information from both ZooKeeper and the database. Singularity will periodically dump stale state into the databaw.

Configuration

The database section of the Singularity configuration file must be populated in order for Singularity to persist task and deploy history information. Here's an example using MySQL:

database:
  driverClass: com.mysql.jdbc.Driver
  user: USERNAME
  password: PASSWORD
  url: jdbc:mysql://HOSTNAME:3306/DB_NAME

A PostgreSQL configuration would be similar, but use a driverClass of org.postgresql.Driver and an appropriate url - for example jdbc:postgresql://HOSTNAME:5432/DB_NAME

Schema Changes

Singularity uses the dropwizard-migrations bundle (which in turn uses liquibase) for managing and applying database schema changes.

To check the status of your database, run the db status task:

java -jar SingularityService/target/SingularityService-.jar db status ./config.yaml --migrations migrations.sql
INFO  [2013-12-23 18:41:33,620] liquibase: Reading from singularity.DATABASECHANGELOG
INFO  [2013-12-23 18:41:33,668] liquibase: Reading from singularity.DATABASECHANGELOG
1 change sets have not been applied to root@localhost@jdbc:mysql://localhost:3306/singularity

Note: The appropriate migrations.sql differs depending on database flavor and is provided in the postgres or mysql directory of the Singularity distribution.

To apply pending migrations, run the db migrate task:

java -jar SingularityService/target/SingularityService-.jar db migrate ./config.yaml --migrations migrations.sql
INFO  [2013-12-23 18:42:08,469] liquibase: Successfully acquired change log lock
INFO  [2013-12-23 18:42:10,206] liquibase: Creating database history table with name: singularity.DATABASECHANGELOG
INFO  [2013-12-23 18:42:10,237] liquibase: Reading from singularity.DATABASECHANGELOG
INFO  [2013-12-23 18:42:10,239] liquibase: Reading from singularity.DATABASECHANGELOG
INFO  [2013-12-23 18:42:10,327] liquibase: ChangeSet migrations.sql::1::tpetr ran successfully in 57ms

More information about db tasks can be found in the dropwizard-migrations docs, and more information about the migration file syntax can be found in the liquibase docs.

Purging Old Tasks

You can optionally purge old task data from the db by specifying historyPurging configuration. In the configuration for SingularityService, you can have a section similar to the following (default values shown):

historyPurging:
  deleteTaskHistoryAfterDays: 365                  # purge tasks older than this
  deleteTaskHistoryAfterTasksPerRequest: 1000      # How many tasks per request before purge
  deleteTaskHistoryBytesInsteadOfEntireRow: true   # Keep the row, just delete data to save space
  checkTaskHistoryEveryHours: 24                   # how often to check for tasks to purge
  enabled: false                                   # determines if we should run the purge