Cache the public key tokens

Sometimes (esp with token auth) we query the same token multiple times.
While this is properly indexed and fast it is still a bit of a waste.

Right now it is doing very stupid caching. Which gets invalidate on any
update.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>