This allows us to safely asynchronously add FDs in the
possibly-promoting unary add case.

Also fix the unary add async path to properly handle more of the extra
cases it needs to handle.