Make sure we do not remove files that are replaced by a symlink

As we ignore symlink, we should not remove on the server the files that
are ignored.

Change tested in t4.pl

Relates to #1299