Remove background job if the server accepted to ask for the shared secret

If we don't remove it the server will later ask the remote server to ask for
the shared secret which will result in a error log message on the remote server
and in some circumstances maybe even to a failure