ERROR: null value in column "group" violates not-null constraint

Summary

Steps to reproduce

fresh install gitlab-jh-14.4.0
execute gitlab-ctl reconfigure
error occurs

Example Project

What is the current bug behavior?

migrate gitlab-rails database partial failed.

What is the expected correct behavior?

pg_dump execute successfully

Relevant logs and/or screenshots

================================================================================
      Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
      ================================================================================

      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Expected process to exit with [0], but received '1'
      ---- Begin output of "bash"  "/tmp/chef-script20211102-1905-pxk8ia" ----
      STDOUT: psql:/opt/gitlab/embedded/service/gitlab-rails/db/structure.sql:9: NOTICE:  extension "btree_gist" already exists, skipping
      psql:/opt/gitlab/embedded/service/gitlab-rails/db/structure.sql:11: NOTICE:  extension "pg_trgm" already exists, skipping
      rake aborted!
      ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  null value in column "group" violates not-null constraint
      DETAIL:  Failing row contains (1, null, Memory Usage (Total), avg(sum(container_memory_usage_bytes{container!="POD",pod=~"^{{c..., Total Memory Used (GB), GB, Total (GB), null, 2021-11-02 04:04:33.99797+00, 2021-11-02 04:04:33.99797+00, t, system_metrics_kubernetes_container_memory_total, null).
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:60:in `block (2 levels) in <class:ConnectionProxy>'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:20:in `block in execute'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:58:in `process_metric_details'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:48:in `block in process_panel'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:47:in `map'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:47:in `process_panel'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:38:in `block in process_group'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:37:in `map'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:37:in `process_group'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:28:in `block in process_content'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:27:in `map'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:27:in `process_content'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:18:in `execute'
      (eval):3:in `block (2 levels) in run_file'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:271:in `block in transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:270:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:66:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:23:in `load'
      /opt/gitlab/embedded/bin/bundle:23:in `<main>'

      Caused by:
      PG::NotNullViolation: ERROR:  null value in column "group" violates not-null constraint
      DETAIL:  Failing row contains (1, null, Memory Usage (Total), avg(sum(container_memory_usage_bytes{container!="POD",pod=~"^{{c..., Total Memory Used (GB), GB, Total (GB), null, 2021-11-02 04:04:33.99797+00, 2021-11-02 04:04:33.99797+00, t, system_metrics_kubernetes_container_memory_total, null).
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:60:in `block (2 levels) in <class:ConnectionProxy>'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:20:in `block in execute'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:58:in `process_metric_details'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:48:in `block in process_panel'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:47:in `map'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:47:in `process_panel'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:38:in `block in process_group'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:37:in `map'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:37:in `process_group'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:28:in `block in process_content'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:27:in `map'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:27:in `process_content'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database_importers/common_metrics/importer.rb:18:in `execute'
      (eval):3:in `block (2 levels) in run_file'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:271:in `block in transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:270:in `transaction'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:66:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:23:in `load'
      /opt/gitlab/embedded/bin/bundle:23:in `<main>'
      Tasks: TOP => db:seed_fu
      (See full trace by running task with --trace)

database logs

2021-11-02_04:04:33.99924 ERROR:  null value in column "group" violates not-null constraint
2021-11-02_04:04:33.99925 DETAIL:  Failing row contains (1, null, Memory Usage (Total), avg(sum(container_memory_usage_bytes{container!="POD",pod=~"^{{c..., Total Memory Used (GB), GB, Total (GB), null, 2021-11-02 04:04:33.99797+00, 2021-11-02 04:04:33.99797+00, t, system_metrics_kubernetes_container_memory_total, null).
2021-11-02_04:04:33.99926 STATEMENT:  /*application:web,db_config_name:main*/ INSERT INTO "prometheus_metrics" ("title", "query", "y_label", "unit", "legend", "created_at", "updated_at", "common", "identifier") VALUES ('Memory Usage (Total)', 'avg(sum(container_memory_usage_bytes{container!="POD",pod=~"^{{ci_environment_slug}}-(.*)",namespace="{{kube_namespace}}"}) by (job)) without (job)  /1024/1024/1024     OR      avg(sum(container_memory_usage_bytes{container_name!="POD",pod_name=~"^{{ci_environment_slug}}-(.*)",namespace="{{kube_namespace}}"}) by (job)) without (job)  /1024/1024/1024', 'Total Memory Used (GB)', 'GB', 'Total (GB)', '2021-11-02 04:04:33.997970', '2021-11-02 04:04:33.997970', TRUE, 'system_metrics_kubernetes_container_memory_total') RETURNING "id"

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info


(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)
```
System information
System:
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.7.4p191
Gem Version:	3.1.4
Bundler Version:2.1.4
Rake Version:	13.0.6
Redis Version:	6.0.14
Git Version:	2.33.0.
Sidekiq Version:6.2.2
Go Version:	unknown

GitLab information
Version:	14.4.0-jh
Revision:	aff11a16a83
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.7
URL:		http://gitlab.example.com
HTTP Clone URL:	http://gitlab.example.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.example.com:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	13.21.1
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git
```

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of: sudo gitlab-rake gitlab:check SANITIZE=true)

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.21.1 ? ... OK (13.21.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: FAILED - Internal API unreachable
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... no
  Try fixing it:
  sudo -u git -H RAILS_ENV=production bin/background_jobs start
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  see log/sidekiq.log for possible errors
  Please fix the error above and rerun the checks.

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/1 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.4)
Git version >= 2.33.0 ? ... yes (2.33.0)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

(For installations from source run and paste the output of: sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)

(we will only investigate if the tests are passing)

$ gitlab-ctl status
run: gitaly: (pid 3118) 2965s; run: log: (pid 2726) 3088s
run: logrotate: (pid 2620) 3107s; run: log: (pid 2631) 3104s
run: postgresql: (pid 2878) 3078s; run: log: (pid 2903) 3076s
run: redis: (pid 2662) 3101s; run: log: (pid 2671) 3097s

Possible fixes

由 Qian Zhang (Gary) 编辑于 11月 02, 2021