The trick is to have multiple SSH keys -- spread out over several physical devices -- all of which are in your server's authorized_keys.