Maintenance

When the application needs to be switched off for maintenance, you can use the maintenance switch.

Maintenance Switch in GitLab

In order to switch maintenance on or off, go to gitlab feature flags and look for flags named <environment>-maintenance.

The daemon will make sure that the change is intercepted within 5 seconds (depending on its setting).

Maintenance Daemon

The daemon periodically checks the status of the flag in GitLab (and possibly anything implementing MaintenanceChecker.php interface) and if at least one checker reports that the service should go down, maintenance key is set in Redis.

If the maintenance is already set and at least one checker does not respond, the maintenance is kept up. Only if all checkers report that he maintenance should be off again, it is switched off.

Detailed implementation of the daemon and its configuration can be found - MaintenanceCheckerCommand.php the command - symfony-command.yaml the helm values for the command - deploy-symfony-command.sh deployment script for kubernetes

The daemon does not log anything when nothing noteworthy happens, but you can send the SIGUSR1 signal to log current stats. The easiest way is to make a k9s shell and run kill -SIGUSR1 1.

You can temporarily switch the checking of status off by sending the SIGUSR2 signal.

Manual Switch from Inside the Cluster

If you need to switch on maintenance manually (gitlab is down or whatever), you can do so by running

./bin/console maintenance:switch on

in k9s shell.

To turn the maintenance off, you can run:

./bin/console maintenance:switch off

Make sure that the daemon is NOT running, since it might interfere with the manual action.

Ignoring Maintenance in Commands and Controllers

By default, all calls done under maintenance will produce a 503 Service Unavailable message and all commands will fail with return code 3 while emitting a warning level message.

This behaviour can be circumvented with the use of IgnoresMaintenance.php attribute - simply add it to your controller or command class.

A few notable endpoints and commands that already circumvent maintenance are - /ch1/liveness-probe which is used to determine the liveness of container - /ch2/maintenance/status which is used to determine maintenance status from the outside - maintenance:switch command, which is used to switch maintenance on or off - maintenance:check command, which is the maintenance daemon itself