Move Google IP fetch into a separate service with rate limiting
Previously the initial merge request in the Google CDN code https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96336 would fetch the list of Google IPs within the same thread as the process that attempts to serve out a Google CDN-frontend URL. However, there are several issues with this: 1. If the HTTP endpoint is slow or down for some reason, this will delay the response handling. 2. There is no throttling to prevent excessive requests to this endpoint. To improve the reliability, we move the fetching of this URL to a background Sidekiq job that has an application rate limit to prevent too many requests. The initial limit is 10 requests in a minute, which should should be adequate since we only need one worker to populate the cache for a whole day. Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/373115 Changelog: changed
显示
- app/services/google_cloud/fetch_google_ip_list_service.rb 89 个添加, 0 个删除app/services/google_cloud/fetch_google_ip_list_service.rb
- app/uploaders/object_storage/cdn/google_cdn.rb 3 个添加, 73 个删除app/uploaders/object_storage/cdn/google_cdn.rb
- app/uploaders/object_storage/cdn/google_ip_cache.rb 60 个添加, 0 个删除app/uploaders/object_storage/cdn/google_ip_cache.rb
- app/workers/all_queues.yml 9 个添加, 0 个删除app/workers/all_queues.yml
- app/workers/google_cloud/fetch_google_ip_list_worker.rb 17 个添加, 0 个删除app/workers/google_cloud/fetch_google_ip_list_worker.rb
- config/sidekiq_queues.yml 2 个添加, 0 个删除config/sidekiq_queues.yml
- lib/gitlab/application_rate_limiter.rb 2 个添加, 1 个删除lib/gitlab/application_rate_limiter.rb
- spec/services/google_cloud/fetch_google_ip_list_service_spec.rb 73 个添加, 0 个删除...ervices/google_cloud/fetch_google_ip_list_service_spec.rb
- spec/uploaders/object_storage/cdn/google_cdn_spec.rb 3 个添加, 45 个删除spec/uploaders/object_storage/cdn/google_cdn_spec.rb
- spec/uploaders/object_storage/cdn/google_ip_cache_spec.rb 84 个添加, 0 个删除spec/uploaders/object_storage/cdn/google_ip_cache_spec.rb
- spec/workers/google_cloud/fetch_google_ip_list_worker_spec.rb 15 个添加, 0 个删除.../workers/google_cloud/fetch_google_ip_list_worker_spec.rb
加载中
想要评论请 注册 或 登录