drivers: can: mcux: flexcan: fix handling of RTR frames

When installing a RX filter, the driver uses "filter->rtr &
filter->rtr_mask" for setting the filter mask. It should just be using
filter->rtr_mask, otherwise filters for non-RTR frames will match RTR
frames as well.

When transmitting a RTR frame, the hardware automatically switches the
mailbox used for TX to RX in order to receive the reply. This, however,
does not match the Zephyr CAN driver model, where mailboxes are dedicated
to either RX or TX. Attempting to reuse the TX mailbox (which was
automatically switched to an RX mailbox by the hardware) fails on the first
call, after which the mailbox is reset and can be reused for TX. To
overcome this, the driver must abort the RX mailbox operation when the
hardware performs the TX to RX switch.

Fixes: #47902

Signed-off-by: Henrik Brix Andersen <hebad@vestas.com>
(cherry picked from commit 3e5aaf83)