Upgrading Singularity to use Mesos
Starting with release version
0.18.0, Singularity will use the mesos http api to communicate with the mesos master via the mesos rx-java library. Documentation on upgrading mesos itself can be found on the mesos website. In order to upgrade Singularity there are a few things to keep in mind beyond the scope of a normal release:
Mesos Version Selection
As of mesos
1.2, the mesos master will no longer accept registrations from mesos agents running
0.x.x versions. As a result, we have chosen to release Singularity
0.18.0 built against mesos
1.1.2, allowing for a smoother upgrade path for users.
1.x version upgrades, less overall change should be needed due to the fact that we are now using the http api and do not depend on native libraires being installed.
Singularity Executor Updates
If you are running the custom Singularity Executor, we recomend updating mesos on your agents before updating the Singularity Executor. We have found in our testing that the older executor (built against
0.x) can run smoothly on mesos
1.1, but the inverse is not always true.
Singularity Service Configuration Updates
The configuration to connect to the mesos master is the only field that has changed with the 1.x upgrade. The new
mesos.master field in the configuration yaml is now a comma seaprated list of mesos master
host:port vaules. Singularity will randomly select from the list when searching for a master (1.x masters will automatically redirect requests to the leading master), trying other hosts in the list if it is not successful.
For example an old configuration of:
mesos: master: zk://my-zk-hostname.com:2181/mesos/singularity
Would now become:
mesos: master: my-mesos-master-host.com:5050
As part of the mesos 1 update, the
org.apache.mesos:mesos library is now pulling in a newer version of protobuf. This can cause issues for users using any other protobuf version. As a result, we have refactored the models in
SingularityBase such that
SingularityBase and the
SingularityClient no longer have a dependency on
For users of the java client, this means that a few of the previously accessible methods on the
SingularityTask object may not be present. All information from the mesos
TaskInfo protos is still being saved as json for later usage, but only the parts needed by Singularity internals are mapped to POJO fields, with the remainder being caught by jackson's
@JsonAnySetter. Extra fields on objects are available as a
getAllOtherFields on the objects.
See #1648 for more details.
Other Mesos Considerations
--work_dirflag must be set on all mesos agents or they will error on startup
- internally the agent -> agent rename is being used, but all apis endpoints and fields still reference agent as they did before. Singularity will tackle the agent -> agent rename in future versions