When we're rejecting invites, ignore the backoff data, so that we have a better chance of not getting the room out of sync.