Fixes #9542

This fixes the case when a server is shutdown right after it Accept'ed
a connection. The socket needs to be closed right before the endpoint
is destroyed.