Skip to content
代码片段 群组 项目
提交 85a2f651 编辑于 作者: Rodrigo Tomonari's avatar Rodrigo Tomonari
浏览文件

Merge branch '434087-update-refresh-jid-github-docs' into 'master'

Updating documentation for github stuckImports

See merge request https://gitlab.com/gitlab-org/gitlab/-/merge_requests/138854



Merged-by: default avatarRodrigo Tomonari <rtomonari@gitlab.com>
Approved-by: default avatarRodrigo Tomonari <rtomonari@gitlab.com>
Co-authored-by: default avatarMax Fan <mfan@gitlab.com>
No related branches found
No related tags found
无相关合并请求
...@@ -209,7 +209,8 @@ When you schedule a job, `AdvanceStageWorker` ...@@ -209,7 +209,8 @@ When you schedule a job, `AdvanceStageWorker`
is given a project ID, a list of Redis keys, and the name of the next is given a project ID, a list of Redis keys, and the name of the next
stage. The Redis keys (produced by `Gitlab::JobWaiter`) are used to check if the stage. The Redis keys (produced by `Gitlab::JobWaiter`) are used to check if the
running stage has been completed or not. If the stage has not yet been running stage has been completed or not. If the stage has not yet been
completed `AdvanceStageWorker` reschedules itself. After a stage finishes completed `AdvanceStageWorker` reschedules itself. After a stage finishes,
or if more jobs have been finished after the last invocation.
`AdvanceStageworker` refreshes the import JID (more on this below) and `AdvanceStageworker` refreshes the import JID (more on this below) and
schedule the worker of the next stage. schedule the worker of the next stage.
...@@ -221,18 +222,19 @@ also reduces pressure on the system as a whole. ...@@ -221,18 +222,19 @@ also reduces pressure on the system as a whole.
## Refreshing import job IDs ## Refreshing import job IDs
GitLab includes a worker called `Gitlab::Import::StuckProjectImportJobsWorker` GitLab includes a worker called `Gitlab::Import::StuckProjectImportJobsWorker`
that periodically runs and marks project imports as failed if they have been that periodically runs and marks project imports as failed if they have not been
running for more than 24 hours. For GitHub projects, this poses a bit of a refreshed for more than 24 hours. For GitHub projects, this poses a bit of a
problem: importing large projects could take several hours depending on how problem: importing large projects could take several days depending on how
often we hit the GitHub rate limit (more on this below), but we don't want often we hit the GitHub rate limit (more on this below), but we don't want
`Gitlab::Import::StuckProjectImportJobsWorker` to mark our import as failed because of this. `Gitlab::Import::StuckProjectImportJobsWorker` to mark our import as failed because of this.
To prevent this from happening we periodically refresh the expiration time of To prevent this from happening we periodically refresh the expiration time of
the import process. This works by storing the JID of the import job in the the import. This works by storing the JID of the import job in the
database, then refreshing this JID TTL at various stages throughout the import database, then refreshing this JID TTL at various stages throughout the import
process. This is done by calling `ProjectImportState#refresh_jid_expiration`. By process. This is done either by calling `ProjectImportState#refresh_jid_expiration`,
refreshing this TTL we can ensure our import does not get marked as failed so or by using the RefreshImportJidWorker and passing in the current worker's jid.
long we're still performing work. By refreshing this TTL we can ensure our import does not get marked as failed so
long as we're still performing work.
## GitHub rate limit ## GitHub rate limit
......
0% 加载中 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册