index.md

stage: Monitor
group: Respond
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments

Log system (FREE SELF)

GitLab has an advanced log system where everything is logged, so you can analyze your instance using various system log files. The log system is similar to audit events.

System log files are typically plain text in a standard log file format. This guide talks about how to read and use these system log files.

Read more about the log system and using the logs:

• Customize logging on Linux package installations including adjusting log retention, log forwarding, switching logs from JSON to plain text logging, and more.
• How to parse and analyze JSON logs.

Log Levels

Each log message has an assigned log level that indicates its importance and verbosity. Each logger has an assigned minimum log level. A logger emits a log message only if its log level is equal to or above the minimum log level.

The following log levels are supported:

Level	Name
0	DEBUG
1	INFO
2	WARN
3	ERROR
4	FATAL
5	UNKNOWN

GitLab loggers emit all log messages because they are set to DEBUG by default.

Override default log level

You can override the minimum log level for GitLab loggers using the GITLAB_LOG_LEVEL environment variable. Valid values are either a value of 0 to 5, or the name of the log level.

Example:

GITLAB_LOG_LEVEL=info

For some services, other log levels are in place that are not affected by this setting. Some of these services have their own environment variables to override the log level. For example:

Service	Log level	Environment variable
GitLab API	INFO
GitLab Cleanup	INFO	DEBUG
GitLab Doctor	INFO	VERBOSE
GitLab Export	INFO	EXPORT_DEBUG
GitLab Geo	INFO
GitLab Import	INFO	IMPORT_DEBUG
GitLab QA Runtime	INFO	QA_LOG_LEVEL
Google APIs	INFO
Rack Timeout	ERROR
Sidekiq (server)	INFO
Snowplow Tracker	FATAL
gRPC Client (Gitaly)	WARN	GRPC_LOG_LEVEL
LLM	INFO	LLM_DEBUG

Log Rotation

The logs for a given service may be managed and rotated by:

• logrotate
• svlogd (runit's service logging daemon)
• logrotate and svlogd
• Or not at all

The following table includes information about what's responsible for managing and rotating logs for the included services. Logs managed by svlogd are written to a file called current. The logrotate service built into GitLab manages all logs except those captured by runit.

Log type	Managed by logrotate	Managed by svlogd/runit
Alertmanager logs	{dotted-circle} No	{check-circle} Yes
crond logs	{dotted-circle} No	{check-circle} Yes
Gitaly	{check-circle} Yes	{check-circle} Yes
GitLab Exporter for Linux package installations	{dotted-circle} No	{check-circle} Yes
GitLab Pages logs	{check-circle} Yes	{check-circle} Yes
GitLab Rails	{check-circle} Yes	{dotted-circle} No
GitLab Shell logs	{check-circle} Yes	{dotted-circle} No
Grafana logs	{dotted-circle} No	{check-circle} Yes
LogRotate logs	{dotted-circle} No	{check-circle} Yes
Mailroom	{check-circle} Yes	{check-circle} Yes
NGINX	{check-circle} Yes	{check-circle} Yes
PgBouncer logs	{dotted-circle} No	{check-circle} Yes
PostgreSQL logs	{dotted-circle} No	{check-circle} Yes
Praefect logs	{dotted-circle} Yes	{check-circle} Yes
Prometheus logs	{dotted-circle} No	{check-circle} Yes
Puma	{check-circle} Yes	{check-circle} Yes
Redis logs	{dotted-circle} No	{check-circle} Yes
Registry logs	{dotted-circle} No	{check-circle} Yes
Workhorse logs	{check-circle} Yes	{check-circle} Yes

production_json.log

This file is located at:

• /var/log/gitlab/gitlab-rails/production_json.log on Linux package installations.
• /home/git/gitlab/log/production_json.log on self-compiled installations.

It contains a structured log for Rails controller requests received from GitLab, thanks to Lograge. Requests from the API are logged to a separate file in api_json.log.

Each line contains JSON that can be ingested by services like Elasticsearch and Splunk. Line breaks were added to examples for legibility:

{
  "method":"GET",
  "path":"/gitlab/gitlab-foss/issues/1234",
  "format":"html",
  "controller":"Projects::IssuesController",
  "action":"show",
  "status":200,
  "time":"2017-08-08T20:15:54.821Z",
  "params":[{"key":"param_key","value":"param_value"}],
  "remote_ip":"18.245.0.1",
  "user_id":1,
  "username":"admin",
  "queue_duration_s":0.0,
  "gitaly_calls":16,
  "gitaly_duration_s":0.16,
  "redis_calls":115,
  "redis_duration_s":0.13,
  "redis_read_bytes":1507378,
  "redis_write_bytes":2920,
  "correlation_id":"O1SdybnnIq7",
  "cpu_s":17.50,
  "db_duration_s":0.08,
  "view_duration_s":2.39,
  "duration_s":20.54,
  "pid": 81836,
  "worker_id":"puma_0"
}

This example was a GET request for a specific issue. Each line also contains performance data, with times in seconds:

• duration_s: Total time to retrieve the request
• queue_duration_s: Total time the request was queued inside GitLab Workhorse
• view_duration_s: Total time inside the Rails views
• db_duration_s: Total time to retrieve data from PostgreSQL
• cpu_s: Total time spent on CPU
• gitaly_duration_s: Total time by Gitaly calls
• gitaly_calls: Total number of calls made to Gitaly
• redis_calls: Total number of calls made to Redis
• redis_cross_slot_calls: Total number of cross-slot calls made to Redis
• redis_allowed_cross_slot_calls: Total number of allowed cross-slot calls made to Redis
• redis_duration_s: Total time to retrieve data from Redis
• redis_read_bytes: Total bytes read from Redis
• redis_write_bytes: Total bytes written to Redis
• redis_<instance>_calls: Total number of calls made to a Redis instance
• redis_<instance>_cross_slot_calls: Total number of cross-slot calls made to a Redis instance
• redis_<instance>_allowed_cross_slot_calls: Total number of allowed cross-slot calls made to a Redis instance
• redis_<instance>_duration_s: Total time to retrieve data from a Redis instance
• redis_<instance>_read_bytes: Total bytes read from a Redis instance
• redis_<instance>_write_bytes: Total bytes written to a Redis instance
• pid: The worker's Linux process ID (changes when workers restart)
• worker_id: The worker's logical ID (does not change when workers restart)

User clone and fetch activity using HTTP transport appears in the log as action: git_upload_pack.

In addition, the log contains the originating IP address, (remote_ip), the user's ID (user_id), and username (username).

Some endpoints (such as /search) may make requests to Elasticsearch if using advanced search. These additionally log elasticsearch_calls and elasticsearch_call_duration_s, which correspond to:

• elasticsearch_calls: Total number of calls to Elasticsearch
• elasticsearch_duration_s: Total time taken by Elasticsearch calls
• elasticsearch_timed_out_count: Total number of calls to Elasticsearch that timed out and therefore returned partial results

ActionCable connection and subscription events are also logged to this file and they follow the previous format. The method, path, and format fields are not applicable, and are always empty. The ActionCable connection or channel class is used as the controller.

{
  "method":null,
  "path":null,
  "format":null,
  "controller":"IssuesChannel",
  "action":"subscribe",
  "status":200,
  "time":"2020-05-14T19:46:22.008Z",
  "params":[{"key":"project_path","value":"gitlab/gitlab-foss"},{"key":"iid","value":"1"}],
  "remote_ip":"127.0.0.1",
  "user_id":1,
  "username":"admin",
  "ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0",
  "correlation_id":"jSOIEynHCUa",
  "duration_s":0.32566
}

NOTE: Starting with GitLab 12.5, if an error occurs, an exception field is included with class, message, and backtrace. Previous versions included an error field instead of exception.class and exception.message. For example:

{
  "method": "GET",
  "path": "/admin",
  "format": "html",
  "controller": "Admin::DashboardController",
  "action": "index",
  "status": 500,
  "time": "2019-11-14T13:12:46.156Z",
  "params": [],
  "remote_ip": "127.0.0.1",
  "user_id": 1,
  "username": "root",
  "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0",
  "queue_duration": 274.35,
  "correlation_id": "KjDVUhNvvV3",
  "queue_duration_s":0.0,
  "gitaly_calls":16,
  "gitaly_duration_s":0.16,
  "redis_calls":115,
  "redis_duration_s":0.13,
  "correlation_id":"O1SdybnnIq7",
  "cpu_s":17.50,
  "db_duration_s":0.08,
  "view_duration_s":2.39,
  "duration_s":20.54,
  "pid": 81836,
  "worker_id": "puma_0",
  "exception.class": "NameError",
  "exception.message": "undefined local variable or method `adsf' for #<Admin::DashboardController:0x00007ff3c9648588>",
  "exception.backtrace": [
    "app/controllers/admin/dashboard_controller.rb:11:in `index'",
    "ee/app/controllers/ee/admin/dashboard_controller.rb:14:in `index'",
    "ee/lib/gitlab/ip_address_state.rb:10:in `with'",
    "ee/app/controllers/ee/application_controller.rb:43:in `set_current_ip_address'",
    "lib/gitlab/session.rb:11:in `with_session'",
    "app/controllers/application_controller.rb:450:in `set_session_storage'",
    "app/controllers/application_controller.rb:444:in `set_locale'",
    "ee/lib/gitlab/jira/middleware.rb:19:in `call'"
  ]
}

production.log

This file is located at:

• /var/log/gitlab/gitlab-rails/production.log on Linux package installations.
• /home/git/gitlab/log/production.log on self-compiled installations.

It contains information about all performed requests. You can see the URL and type of request, IP address, and what parts of code were involved to service this particular request. Also, you can see all SQL requests performed, and how much time each took. This task is more useful for GitLab contributors and developers. Use part of this log file when you're reporting bugs. For example:

Started GET "/gitlabhq/yaml_db/tree/master" for 168.111.56.1 at 2015-02-12 19:34:53 +0200
Processing by Projects::TreeController#show as HTML
  Parameters: {"project_id"=>"gitlabhq/yaml_db", "id"=>"master"}

  ... [CUT OUT]

  Namespaces"."created_at" DESC, "namespaces"."id" DESC LIMIT 1 [["id", 26]]
  CACHE (0.0ms) SELECT  "members".* FROM "members"  WHERE "members"."source_type" = 'Project' AND "members"."type" IN ('ProjectMember') AND "members"."source_id" = $1 AND "members"."source_type" = $2 AND "members"."user_id" = 1  ORDER BY "members"."created_at" DESC, "members"."id" DESC LIMIT 1  [["source_id", 18], ["source_type", "Project"]]
  CACHE (0.0ms) SELECT  "members".* FROM "members"  WHERE "members"."source_type" = 'Project' AND "members".
  (1.4ms) SELECT COUNT(*) FROM "merge_requests"  WHERE "merge_requests"."target_project_id" = $1 AND ("merge_requests"."state" IN ('opened','reopened')) [["target_project_id", 18]]
  Rendered layouts/nav/_project.html.haml (28.0ms)
  Rendered layouts/_collapse_button.html.haml (0.2ms)
  Rendered layouts/_flash.html.haml (0.1ms)
  Rendered layouts/_page.html.haml (32.9ms)
Completed 200 OK in 166ms (Views: 117.4ms | ActiveRecord: 27.2ms)

In this example, the server processed an HTTP request with URL /gitlabhq/yaml_db/tree/master from IP 168.111.56.1 at 2015-02-12 19:34:53 +0200. The request was processed by Projects::TreeController.

api_json.log

This file is located at:

• /var/log/gitlab/gitlab-rails/api_json.log on Linux package installations.
• /home/git/gitlab/log/api_json.log on self-compiled installations.

It helps you see requests made directly to the API. For example:

{
  "time":"2018-10-29T12:49:42.123Z",
  "severity":"INFO",
  "duration":709.08,
  "db":14.59,
  "view":694.49,
  "status":200,
  "method":"GET",
  "path":"/api/v4/projects",
  "params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],
  "host":"localhost",
  "remote_ip":"::1",
  "ua":"Ruby",
  "route":"/api/:version/projects",
  "user_id":1,
  "username":"root",
  "queue_duration":100.31,
  "gitaly_calls":30,
  "gitaly_duration":5.36,
  "pid": 81836,
  "worker_id": "puma_0",
  ...
}

This entry shows an internal endpoint accessed to check whether an associated SSH key can download the project in question by using a git fetch or git clone. In this example, we see:

• duration: Total time in milliseconds to retrieve the request
• queue_duration: Total time in milliseconds the request was queued inside GitLab Workhorse
• method: The HTTP method used to make the request
• path: The relative path of the query
• params: Key-value pairs passed in a query string or HTTP body (sensitive parameters, such as passwords and tokens, are filtered out)
• ua: The User-Agent of the requester

NOTE: As of Grape Logging v1.8.4, the view_duration_s is calculated by duration_s - db_duration_s. Therefore, view_duration_s can be affected by multiple different factors, like read-write process on Redis or external HTTP, not only the serialization process.

application.log (deprecated)

Deprecated in GitLab 15.10.

This file is located at:

• /var/log/gitlab/gitlab-rails/application.log on Linux package installations.
• /home/git/gitlab/log/application.log on self-compiled installations.

It contains a less structured version of the logs in application_json.log, like this example:

October 06, 2014 11:56: User "Administrator" (admin@example.com) was created
October 06, 2014 11:56: Documentcloud created a new project "Documentcloud / Underscore"
October 06, 2014 11:56: Gitlab Org created a new project "Gitlab Org / Gitlab Ce"
October 07, 2014 11:25: User "Claudie Hodkiewicz" (nasir_stehr@olson.co.uk)  was removed
October 07, 2014 11:25: Project "project133" was removed

application_json.log

Introduced in GitLab 12.7.