Fix to not destroy draggable when no draggable was set

When a dir has no delete permission, the draggable isn't initialized on
files. This fix makes sure we don't try to destroy a draggable when it
wasn't inited in the first place.

Fixes #6254