Add deprecations and removals to named folders with version subdir

Add a removals template and example yml

Reformat all removals yml to match expected format

Add removal_milestone to historic removals

Update tooling naming to reflect hierarchy

Compile deprecations and removals

Deprecation announcements in announced milestone

Updating CI job to ensure deps/rems are in sync

Combine the check_* rake tasks to reduce pipeline time

Indicate that removal is Milestone

Use **/* to check all files and subdirs

This file needed to be updated; it's causing a test to fail. I can't
guarantee it'll fix all of the pipeline failures but it'll fix
this one.

Add deprecations and removals to named folders with version subdir

Compile Deprecations with 14.6 updates

Add spec to deprecation handling

.gitlab/ci/docs.gitlab-ci.yml

@@ -77,15 +77,16 @@ ui-docs-links lint:
   script:
     - bundle exec haml-lint -i DocumentationLinks
 
-docs-lint deprecations:
+docs-lint deprecations-and-removals:
   variables:
     SETUP_DB: "false"
   extends:
     - .default-retry
     - .rails-cache
     - .default-before_script
-    - .docs:rules:deprecations
+    - .docs:rules:deprecations-and-removals
   stage: lint
   needs: []
   script:
     - bundle exec rake gitlab:docs:check_deprecations
+    - bundle exec rake gitlab:docs:check_removals

.gitlab/ci/rules.gitlab-ci.yml

@@ -171,12 +171,13 @@
   - ".markdownlint.yml"
   - "scripts/lint-doc.sh"
 
-.docs-deprecations-patterns: &docs-deprecations-patterns
+.docs-deprecations-and-removals-patterns: &docs-deprecations-and-removals-patterns
   - "doc/update/deprecations.md"
-  - "data/deprecations/*.yml"
-  - "data/deprecations/templates/_deprecation_template.md.erb"
+  - "doc/update/removals.md"
+  - "data/deprecations/**/*"
+  - "data/removals/**/*"
+  - "tooling/docs/**/*"
   - "lib/tasks/gitlab/docs/compile_deprecations.rake"
-  - "tooling/deprecations/docs.rb"
 
 .bundler-patterns: &bundler-patterns
   - '{Gemfile.lock,*/Gemfile.lock,*/*/Gemfile.lock}'
@@ -541,10 +542,10 @@
       changes: *docs-patterns
       when: on_success
 
-.docs:rules:deprecations:
+.docs:rules:deprecations-and-removals:
   rules:
     - <<: *if-default-refs
-      changes: *docs-deprecations-patterns
+      changes: *docs-deprecations-and-removals-patterns
 
 ##################
 # GraphQL rules  #

data/deprecations/templates/_deprecation_template.md.erb

@@ -4,7 +4,7 @@ group: none
 info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines"
 ---
 
-# Deprecated features
+# Deprecations by milestone
 
 DISCLAIMER:
 This page contains information related to upcoming products, features, and functionality.
@@ -38,13 +38,12 @@ For deprecation reviewers (Technical Writers only):
 <% if milestones.any? -%>
   <%- milestones.each do |milestone| %>
 ## <%= milestone %>
-    <%- deprecations.select{|d| d["removal_milestone"] == milestone}.each do |deprecation| %>
+    <%- entries.select{|d| d["announcement_milestone"] == milestone}.each do |deprecation| %>
 ### <%= deprecation["name"]%>
 
 <%= deprecation["body"] -%>
 
-Announced: <%= deprecation["announcement_date"]%>
-Planned removal: <%= deprecation["removal_date"]%>
+Planned removal milestone: <%= deprecation["removal_milestone"]%> (<%= deprecation["removal_date"]%>)
     <%- end -%>
   <%- end -%>
 <%- else -%>

data/removals/14_0/14_0-ds-deprecations.yml

+- name: "Deprecations for Dependency Scanning"
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  reporter: nicoleschwartz
+  body: |
+    As mentioned in [13.9](https://about.gitlab.com/releases/2021/02/22/gitlab-13-9-released/#deprecations-for-dependency-scanning) and [this blog post](https://about.gitlab.com/blog/2021/02/08/composition-analysis-14-deprecations-and-removals/) several removals for Dependency Scanning take effect.
+
+    Previously, to exclude a DS analyzer, you needed to remove it from the default list of analyzers, and use that to set the `DS_DEFAULT_ANALYZERS` variable in your project’s CI template. We determined it should be easier to avoid running a particular analyzer without losing the benefit of newly added analyzers. As a result, we ask you to migrate from `DS_DEFAULT_ANALYZERS` to `DS_EXCLUDED_ANALYZERS` when it is available. Read about it in [issue #287691](https://gitlab.com/gitlab-org/gitlab/-/issues/287691).
+
+    Previously, to prevent the Gemnasium analyzers to fetch the advisory database at runtime, you needed to set the `GEMNASIUM_DB_UPDATE` variable. However, this is not documented properly, and its naming is inconsistent with the equivalent `BUNDLER_AUDIT_UPDATE_DISABLED` variable. As a result, we ask you to migrate from `GEMNASIUM_DB_UPDATE` to `GEMNASIUM_UPDATE_DISABLED` when it is available. Read about it in [issue #215483](https://gitlab.com/gitlab-org/gitlab/-/issues/215483).

data/removals/14_0/14_0-lc-deprecations.yml

+- name: "Removals for License Compliance"
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  reporter: nicoleschwartz
+  body: |
+    In 13.0, we deprecated the License-Management CI template and renamed it License-Scanning. We have been providing backward compatibility by warning users of the old template to switch. Now in 14.0, we are completely removing the License-Management CI template. Read about it in [issue #216261](https://gitlab.com/gitlab-org/gitlab/-/issues/216261) or [this blog post](https://about.gitlab.com/blog/2021/02/08/composition-analysis-14-deprecations-and-removals/).

data/removals/14_0/change_default_branch_name_to_main.yml

+- name: "Default branch name for new repositories now `main`"
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  reporter: sarahwaldner
+  body: |
+    Every Git repository has an initial branch, which is named `master` by default. It's the first branch to be created automatically when you create a new repository. Future [Git versions](https://lore.kernel.org/git/pull.656.v4.git.1593009996.gitgitgadget@gmail.com/) will change the default branch name in Git from `master` to `main`. In coordination with the Git project and the broader community, [GitLab has changed the default branch name](https://gitlab.com/gitlab-org/gitlab/-/issues/223789) for new projects on both our SaaS (GitLab.com) and self-managed offerings starting with GitLab 14.0. This will not affect existing projects.
+
+    GitLab has already introduced changes that allow you to change the default branch name both at the [instance level](https://docs.gitlab.com/ee/user/project/repository/branches/default.html#instance-level-custom-initial-branch-name) (for self-managed users) and at the [group level](https://docs.gitlab.com/ee/user/group/#use-a-custom-name-for-the-initial-branch) (for both SaaS and self-managed users). We encourage you to make use of these features to set default branch names on new projects.
+
+    For more information, check out our [blog post](https://about.gitlab.com/blog/2021/03/10/new-git-default-branch-name/).

data/removals/14_0/create-code-review-draft-wip.yml

+- name: "WIP merge requests renamed 'draft merge requests'"
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  reporter: phikai
+  body: |
+    The WIP (work in progress) status for merge requests signaled to reviewers that the merge request in question wasn't ready to merge. We've renamed the WIP feature to **Draft**, a more inclusive and self-explanatory term. **Draft** clearly communicates the merge request in question isn't ready for review, and makes no assumptions about the progress being made toward it. **Draft** also reduces the cognitive load for new users, non-English speakers, and anyone unfamiliar with the WIP acronym.

data/removals/14_0/create-code-review-w-parameter-removal.yml

+- name: "Remove `?w=1` URL parameter to ignore whitespace changes"
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  reporter: phikai
+  body: |
+    To create a consistent experience for users based on their preferences, support for toggling whitespace changes via URL parameter has been removed in GitLab 14.0.

data/removals/14_0/deprecate_ci_project_config_path_variable.yml

+- name: "`CI_PROJECT_CONFIG_PATH` removed in GitLab 14.0"
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  reporter: stkerr
+  body: |
+    GitLab 14.0 removes the `CI_PROJECT_CONFIG_PATH` pre-defined project variable in favor of `CI_CONFIG_PATH`, which is functionally the same. If you are using `CI_PROJECT_CONFIG_PATH` in your pipeline configurations, update them to use `CI_CONFIG_PATH` instead.

data/removals/14_0/deprecation_bump_terraform_template_version.yml

+- name: "New Terraform template version"
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0" # example
+  issue_url: ""
+  reporter: nagyv-gitlab
+  body: |
+    As we continuously [develop GitLab's Terraform integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/325312), to minimize customer disruption, we maintain two GitLab CI/CD templates for Terraform:
+
+    - The ["latest version" template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.latest.gitlab-ci.yml), which is updated frequently between minor releases of GitLab (such as 13.10, 13.11, etc).
+    - The ["major version" template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml), which is updated only at major releases (such as 13.0, 14.0, etc).
+
+    At every major release of GitLab, the "latest version" template becomes the "major version" template, inheriting the "latest template" setup.
+    As we have added many new features to the Terraform integration, the new setup for the "major version" template can be considered a breaking change.
+
+    The latest template supports the [Terraform Merge Request widget](https://docs.gitlab.com/ee/user/infrastructure/mr_integration.html) and
+    doesn't need additional setup to work with the [GitLab managed Terraform state](https://docs.gitlab.com/ee/user/infrastructure/terraform_state.html).
+
+    To check the new changes, see the [new "major version" template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml).

data/removals/14_0/deprecation_manage_access_14_0.yml

+- name: Limit projects returned in `GET /groups/:id/`
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  reporter: ogolowisnki
+  issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/257829
+  body: |
+    To improve performance, we are limiting the number of projects returned from the `GET /groups/:id/` API call to 100. A complete list of projects can still be retrieved with the `GET /groups/:id/projects` API call.
+
+- name: "GitLab OAuth implicit grant deprecation"
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  reporter: ogolowinski
+  issue_url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/288516'
+  body: |
+    GitLab is deprecating the [OAuth 2 implicit grant flow](https://docs.gitlab.com/ee/api/oauth2.html#implicit-grant-flow) as it has been removed for [OAuth 2.1](https://oauth.net/2.1/).
+
+    Beginning in 14.0, new applications can't be created with the OAuth 2 implicit grant flow. Existing OAuth implicit grant flows are no longer supported in 14.4. Migrate your existing applications to other supported [OAuth2 flows](https://docs.gitlab.com/ee/api/oauth2.html#supported-oauth2-flows) before release 14.4.

data/removals/14_0/deprecation_update_cicd_templates_to_stop_using_hardcode_master.yml

+- name: "Update CI/CD templates to stop using hardcoded `master`"
+  reporter: dhershkovitch
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  body: |
+    Our CI/CD templates have been updated to no longer use hard-coded references to a `master` branch. In 14.0, they all use a variable that points to your project's configured default branch instead. If your CI/CD pipeline relies on our built-in templates, verify that this change works with your current configuration. For example, if you have a `master` branch and a different default branch, the updates to the templates may cause changes to your pipeline behavior. For more information, [read the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324131).

data/removals/14_0/deuley_servicetemplates_removal.yml

+- name: "Service Templates removed"
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  reporter: deuley
+  body: |
+    Service Templates are [removed in GitLab 14.0](https://gitlab.com/groups/gitlab-org/-/epics/5672). They were used to apply identical settings to a large number of projects, but they only did so at the time of project creation.
+
+    While they solved part of the problem, _updating_ those values later proved to be a major pain point. [Project Integration Management](https://docs.gitlab.com/ee/user/admin_area/settings/project_integration_management.html) solves this problem by enabling you to create settings at the Group or Instance level, and projects within that namespace inheriting those settings.

data/removals/14_0/release_announce_deprecation_of_release_notes_api.yml

+- name: "Removal of release description in the Tags API"
+  reporter: kbychu
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  issue_url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/300887'
+  body: |
+    GitLab 14.0 removes support for the release description in the Tags API. You can no longer add a release description when [creating a new tag](https://docs.gitlab.com/ee/api/tags.html#create-a-new-tag). You also can no longer [create](https://docs.gitlab.com/ee/api/tags.html#create-a-new-release) or [update](https://docs.gitlab.com/ee/api/tags.html#update-a-release) a release through the Tags API. Please migrate to use the [Releases API](https://docs.gitlab.com/ee/api/releases/#create-a-release) instead.

data/removals/14_0/release_deprecation_auto-deploy-image.yml

+- name: "Update Auto Deploy template version"
+  reporter: kbychu
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  issue_url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/300862'
+  body: |
+    In GitLab 14.0, we will update the [Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/stages.html#auto-deploy) CI template to the latest version. This includes new features, bug fixes, and performance improvements with a dependency on the v2 [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image). Auto Deploy CI tempalte v1 will is deprecated going forward.
+
+    Since the v1 and v2 versions are not backward-compatible, your project might encounter an unexpected failure if you already have a deployed application. Follow the [upgrade guide](https://docs.gitlab.com/ee/topics/autodevops/upgrading_auto_deploy_dependencies.html#upgrade-guide) to upgrade your environments. You can also start using the latest template today by following the [early adoption guide](https://docs.gitlab.com/ee/topics/autodevops/upgrading_auto_deploy_dependencies.html#early-adopters).

data/removals/14_0/release_domainsource_configuration_for_gitlab_pages_deprecation.yml

+- name: "Remove disk source configuration for GitLab Pages"
+  reporter: kbychu
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  issue_url: 'https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993'
+  body: |
+    GitLab Pages [API-based configuration](https://docs.gitlab.com/ee/administration/pages/#gitlab-api-based-configuration) has been available since GitLab 13.0. It replaces the unsupported `disk` source configuration removed in GitLab 14.0, which can no longer be chosen. You should stop using `disk` source configuration, and move to `gitlab` for an API-based configuration. To migrate away from the 'disk' source configuration, set `gitlab_pages['domain_config_source'] = "gitlab"` in your `/etc/gitlab/gitlab.rb` file. We recommend you migrate before updating to GitLab 14.0, to identify and troubleshoot any potential problems before upgrading.

data/removals/14_0/release_legacy_feature_flags_deprecation.yml

+- name: "Legacy feature flags removed"
+  reporter: kbychu
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  issue_url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/254324'
+  body: |
+    Legacy feature flags became read-only in GitLab 13.4. GitLab 14.0 removes support for legacy feature flags, so you must migrate them to the [new version](https://docs.gitlab.com/ee/operations/feature_flags.html). You can do this by first taking a note (screenshot) of the legacy flag, then deleting the flag through the API or UI (you don't need to alter the code), and finally create a new Feature Flag with the same name as the legacy flag you deleted. Also, make sure the strategies and environments match the deleted flag. We created a [video tutorial](https://www.youtube.com/watch?v=CAJY2IGep7Y) to help with this migration.

data/removals/14_0/release_remove_redundant_keyvalue_pair_from_the_payload_of_dora.yml

+- name: "Remove redundant timestamp field from DORA metrics API payload"
+  reporter: kbychu
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  issue_url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/325931'
+  body: |
+    The [deployment frequency project-level API](https://docs.gitlab.com/ee/api/dora4_project_analytics.html#list-project-deployment-frequencies) endpoint has been deprecated in favor of the [DORA 4 API](https://docs.gitlab.com/ee/api/dora/metrics.html), which consolidates all the metrics under one API with the specific metric as a required field. As a result, the timestamp field, which doesn't allow adding future extensions and causes performance issues, will be removed. With the old API, an example response would be `{ "2021-03-01": 3, "date": "2021-03-01", "value": 3 }`. The first key/value (`"2021-03-01": 3`) will be removed and replaced by the last two (`"date": "2021-03-01", "value": 3`).

data/removals/14_0/removal-geo-fdw-settings.yml

+- name: "Geo Foreign Data Wrapper settings removed"
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  reporter: fzimmer
+  body: |
+    As [announced in GitLab 13.3](https://about.gitlab.com/releases/2020/08/22/gitlab-13-3-released/#geo-foreign-data-wrapper-settings-deprecated), the following configuration settings in `/etc/gitlab/gitlab.rb` have been removed in 14.0:
+
+    - `geo_secondary['db_fdw']`
+    - `geo_postgresql['fdw_external_user']`
+    - `geo_postgresql['fdw_external_password']`
+    - `gitlab-_rails['geo_migrated_local_files_clean_up_worker_cron']`

data/removals/14_0/removal-graphql-fields.yml

+- name: "Deprecated GraphQL fields have been removed"
+  removal_date: "2021-06-22"
+  removal_milestone: "14.0"
+  reporter: gweaver
+  body: |
+    In accordance with our [GraphQL deprecation and removal process](https://docs.gitlab.com/ee/api/graphql/#deprecation-process), the following fields that were deprecated prior to 13.7 are [fully removed in 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/267966):
+
+    - `Mutations::Todos::MarkAllDone`, `Mutations::Todos::RestoreMany` - `:updated_ids`
+    - `Mutations::DastScannerProfiles::Create`, `Types::DastScannerProfileType` - `:global_id`
+    - `Types::SnippetType` - `:blob`
+    - `EE::Types::GroupType`, `EE::Types::QueryType` - `:vulnerabilities_count_by_day_and_severity`
+    - `DeprecatedMutations (concern**)` - `AddAwardEmoji`, `RemoveAwardEmoji`, `ToggleAwardEmoji`
+    - `EE::Types::DeprecatedMutations (concern***)` - `Mutations::Pipelines::RunDastScan`, `Mutations::Vulnerabilities::Dismiss`, `Mutations::Vulnerabilities::RevertToDetected`