Move parent removal from final destruction to destruction request

We don't need the list for cancel propagation after that point, and
doing it early avoids a bug whereby the call is reffed after reaching a
zero ref when the parent is destroyed first.