Add counter metric for deferred jobs

doc/administration/monitoring/prometheus/gitlab_metrics.md

@@ -234,6 +234,7 @@ configuration option in `gitlab.yml`. These metrics are served from the
 | `sidekiq_mem_total_bytes`                      | Gauge     | 15.3 | Number of bytes allocated for both objects consuming an object slot and objects that required a malloc'|                                                                   |
 | `sidekiq_concurrency_limit_queue_jobs`         | Gauge     | 17.3 | Number of Sidekiq jobs waiting in the concurrency limit queue|  `worker`                                                             |
 | `sidekiq_concurrency_limit_max_concurrent_jobs` | Gauge     | 17.3 | Max number of concurrent running Sidekiq jobs |   `worker`                                                           |
+| `sidekiq_concurrency_limit_deferred_jobs_total` | Counter     | 17.3 | Total number of deferred Sidekiq jobs |   `worker`                                                           |
 | `geo_db_replication_lag_seconds`               | Gauge   | 10.2  | Database replication lag (seconds) | `url` |
 | `geo_repositories`                             | Gauge   | 10.2  | Total number of repositories available on primary | `url` |
 | `geo_lfs_objects`                              | Gauge   | 10.2  | Number of LFS objects on primary | `url` |

doc/development/sidekiq/worker_attributes.md

@@ -372,12 +372,18 @@ With the `concurrency_limit` property, you can limit the worker's concurrency. I
 a separate `LIST` and re-enqueued when it falls under the limit. `ConcurrencyLimit::ResumeWorker` is a cron
 worker that checks if any throttled jobs should be re-enqueued.
 
-The first job that crosses the defined concurency limit initiates the throttling process for all other jobs of this class.
+The first job that crosses the defined concurrency limit initiates the throttling process for all other jobs of this class.
 Until this happens, jobs are scheduled and executed as usual.
 
 When the throttling starts, newly scheduled and executed jobs will be added to the end of the `LIST` to ensure that
 the execution order is preserved. As soon as the `LIST` is empty again, the throttling process ends.
 
+Prometheus metrics are exposed to monitor workers using concurrency limit middleware:
+
+- `sidekiq_concurrency_limit_deferred_jobs_total`
+- `sidekiq_concurrency_limit_queue_jobs`
+- `sidekiq_concurrency_limit_max_concurrent_jobs`
+
 WARNING:
 If there is a sustained workload over the limit, the `LIST` is going to grow until the limit is disabled or
 the workload drops under the limit.

lib/gitlab/sidekiq_middleware/concurrency_limit/concurrency_limit_service.rb

@@ -33,6 +33,8 @@ def add_to_queue!(args, context)
           with_redis do |redis|
             redis.rpush(redis_key, serialize(args, context))
           end
+
+          deferred_job_counter.increment({ worker: worker_name })
         end
 
         def queue_size
@@ -96,6 +98,11 @@ def next_batch_from_queue(redis, limit:)
         def remove_processed_jobs(redis, limit:)
           redis.ltrim(redis_key, limit, -1)
         end
+
+        def deferred_job_counter
+          @deferred_job_count ||= ::Gitlab::Metrics.counter(:sidekiq_concurrency_limit_deferred_jobs_total,
+            'Count of jobs deferred by the concurrency limit middleware.')
+        end
       end
     end
   end

spec/lib/gitlab/sidekiq_middleware/concurrency_limit/concurrency_limit_service_spec.rb

@@ -45,6 +45,15 @@ def self.name
 
       add_to_queue!
     end
+
+    it 'reports prometheus metrics' do
+      deferred_job_count_double = instance_double(Prometheus::Client::Counter)
+      expect(Gitlab::Metrics).to receive(:counter).with(:sidekiq_concurrency_limit_deferred_jobs_total, anything)
+        .and_return(deferred_job_count_double)
+      expect(deferred_job_count_double).to receive(:increment).with({ worker: worker_class_name })
+
+      add_to_queue!
+    end
   end
 
   describe '.has_jobs_in_queue?' do