Use link for accept invite banner only with access

Only show a link when the user has access to the given group (or
the group is public).