Allow asynchronous call destruction in some cases

If there's a chance we're being called via a callback, we need to delay destruction until later. Otherwise, for performance, destroy it inline.