[SECURITY-360] introduce a system switch to kill CLI

This basically is a convenient version of
https://github.com/jenkinsci-cert/SECURITY-218.

During the course of discussing how to fix SECURITY-360, it was agreed
by the CERT team that we provide this switch.