>the Listener rejects any handshake from an unknown key.

It never responds, there's no indication of rejection.

The client can't even be sure a service is actually there or they've hit a default drop firewall rule.

Why stop there? Give use Wireguard certificate auth already!