Do not remove speaker and video for own peer

The speaker and video are never added for the own peer, so there is no
need to remove them.

Signed-off-by: Daniel Calviño Sánchez <danxuliu@gmail.com>