Bluetooth: BAP: Do not send PA term request on local remove

If the receive state is locally removed, then we should not
request permission from the application, as that is implicit
when removing the source locally.

Signed-off-by: Emil Gydesen <emil.gydesen@nordicsemi.no>