Fix loading of ancestors for signed out users

See merge request https://gitlab.com/gitlab-org/gitlab/-/merge_requests/138209



Merged-by: Aaron Huntsman <ahuntsman@gitlab.com>
Approved-by: Joseph Wambua <jjoshua@gitlab.com>
Reviewed-by: Nicolas Dular <ndular@gitlab.com>
Co-authored-by: Nicolas Dular <ndular@gitlab.com>