creation denied from custom hooks

Summary

用户环境较为复杂。

无状态和组件服务运行在 k8s 集群中（例如 webservice/sidekiq 等）
有状态的组件单独使用 omnibus 方式进行部署（例如 gitaly/redis/postgresql 等）

目前出现的问题是存在个别时候创建分支提示creation denied from custom hooks，经过沟通和确认项目仅设置了分支规则branch rules，并没有设置推送规则，删除分支规则后分支可以正常创建。但并不是保护分支规则创建后就会出现此错误，用户反馈是使用一段时间后才会出现这个报错，原来只有一个项目出现这个问题，现在会有多个项目都出现这个问题。目前的 workaround 方案也是删除保护分支规则后重新创建，重建后大约 10 天左右会复现以上问题。

目前只有分支创建会出现类似的报错，暂无反馈其他相关报错。

Steps to reproduce

未复现

Example Project

What is the current bug behavior?

创建匹配分支规则的分支时提示creation denied from custom hooks

What is the expected correct behavior?

可以正常创建匹配保护分支规则的保护分支。

Relevant logs and/or screenshots

相关日志：

{
    "msg": "finished unary call with code PermissionDenied",
    "grpc.time_ms": "136.549",
    "pid": "8412",
    "grpc.meta.deadline_type": "regular",
    "grpc.meta.client_name": "gitlab-web",
    "grpc.code": "PermissionDenied",
    "catfile.request_object_count": "1",
    "command.real_time_ms": "2",
    "remote_ip": "10.160.229.122",
    "grpc.meta.auth_version": "v2",
    "peer.address": "10.88.3.94:38026",
    "catfile.request_object_ms": "0",
    "level": "warning",
    "command.cpu_time_ms": "1",
    "limit.concurrency_queue_ms": "0",
    "grpc.meta.method_type": "unary",
    "grpc.request.repoStorage": "gitaly-1",
    "command.oublock": "0",
    "__TAG__": {
        "inst_name": "gitlab-ha-gitaly-prod"
    },
    "command.count": "2",
    "system": "grpc",
    "user_id": "6169",
    "catfile.read_object_ms": "0",
    "catfile.flush_count": "1",
    "catfile.flush_ms": "0",
    "grpc.start_time": "2023-07-26T13:06:56.277",
    "grpc.request.repoPath": "@cluster/repositories/7a/3c/11046",
    "command.maxrss": "2236696",
    "grpc.service": "gitaly.OperationService",
    "grpc.request.glProjectPath": "SCMS/dms.server",
    "command.system_time_ms": "1",
    "catfile.read_object_count": "1",
    "error": "creation denied by custom hooks",
    "grpc.method": "UserCreateBranch",
    "grpc.request.fullMethod": "/gitaly.OperationService/UserCreateBranch",
    "grpc.request.glRepository": "project-1490",
    "grpc.request.payload_bytes": "181",
    "span.kind": "server",
    "grpc.response.payload_bytes": "0",
    "command.user_time_ms": "0",
    "grpc.request.deadline": "2023-07-26T13:07:51.062",
    "command.majflt": "0",
    "command.inblock": "0",
    "correlation_id": "01H689FMJ4WF370M6GT8ETPYHA",
    "catfile.duration_ms": "0",
    "command.spawn_token_wait_ms": "0",
    "time": "2023-07-26T05:06:56.413Z",
    "command.minflt": "816",
    "username": "ning.xiang",
    "__SOURCE__": "10.88.3.28",
    "__FILENAME__": "/data/logs/gitlab/gitaly/current",
    "__HOSTNAME__": "p-qcbj6-gitlab-ha-gitaly-01"
}

看起来是鉴权错误，但是奇怪的是为什么只有分支创建时会出现鉴权错误。需要特别说明的是用户 k8s 环境和 omnibus 环境都有配置 NTP。

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

由马翔编辑于 8月 01, 2023