index.md

stage: AI-powered
group: AI Framework
info: Any user with at least the Maintainer role can merge updates to this content. For details, see https://docs.gitlab.com/ee/development/development_processes.html#development-guidelines-review.

AI features based on 3rd-party integrations

Introduced in GitLab 15.11.

Features

• Async execution of the long running API requests
  • GraphQL Action starts the request
  • Background workers execute
  • GraphQL subscriptions deliver results back in real time
• Abstraction for
  • Google Vertex AI
  • Anthropic
• Rate Limiting
• Circuit Breaker
• Multi-Level feature flags
• License checks on group level
• Snowplow execution tracking
• Tracking of Token Spent on Prometheus
• Configuration for Moderation check of inputs
• Automatic Markdown Rendering of responses
• Centralised Group Level settings for experiment and 3rd party
• Experimental API endpoints for exploration of AI APIs by GitLab team members without the need for credentials
  • Google Vertex AI
  • Anthropic

Feature flags

Apply the following feature flags to any AI feature work:

• A general flag (ai_duo_chat_switch) that applies to all GitLab Duo Chat features.
• A general flag (ai_global_switch) that applies to all other AI features.
• A flag specific to that feature. The feature flag name must be different than the licensed feature name.

See the feature flag tracker epic for the list of all feature flags and how to use them.

Implement a new AI action

To implement a new AI action, connect to the preferred AI provider. You can connect to this API using either the:

• Experimental REST API.
• Abstraction layer.

All AI features are experimental.

Test self-managed AI features locally

See below

Test SaaS-only AI features locally

One-line setup

# Replace the <test-group-name> by the group name you want to enable GitLab Duo features. If the group doesn't exist, it creates a new one.
RAILS_ENV=development bundle exec rake gitlab:duo:setup['<test-group-name>']

Manual way

1. Enable the required general feature flags:

   Feature.enable(:ai_duo_chat_switch, type: :ops)
   Feature.enable(:ai_global_switch, type: :ops)

2. Ensure you have followed the process to obtain an EE license for your local instance and you applied this license.

3. Simulate the GDK to simulate SaaS and ensure the group you want to test has an Ultimate license

4. Enable Experiment & Beta features

   1. Go to the group with the Ultimate license
   2. Group Settings > General -> Permissions and group features
   3. Enable Experiment & Beta features

5. Enable the specific feature flag for the feature you want to test

6. You can use Rake task rake gitlab:duo:enable_feature_flags to enable all feature flags that are assigned to group AI Framework

7. Set the required access token. To receive an access token:

   1. For Vertex, follow the instructions below.
   2. For Anthropic, create an access request

Configure GCP Vertex access

In order to obtain a GCP service key for local development, follow the steps below:

• Create a sandbox GCP project by visiting this page and following the instructions, or by requesting access to our existing group GCP project by using this template.
• If you are using an individual GCP project, you may also need to enable the Vertex AI API:
  1. Visit welcome page, choose your project (e.g. jdoe-5d23dpe).
  2. Go to APIs & Services > Enabled APIs & services.
  3. Select + Enable APIs and Services.
  4. Search for Vertex AI API.
  5. Select Vertex AI API, then select Enable.
• Install the gcloud CLI
• Authenticate locally with GCP using the gcloud auth application-default login command.
• Open the Rails console. Update the settings to:

# PROJECT_ID = "your-gcp-project-name"

Gitlab::CurrentSettings.update(vertex_ai_project: PROJECT_ID)

Configure Anthropic access

Gitlab::CurrentSettings.update!(anthropic_api_key: <insert API key>)

Embeddings database

Embeddings are generated through the VertexAI text embeddings API. The sections below explain how to populate embeddings in the DB or extract embeddings to be used in specs.

Set up

1. Enable pgvector in GDK

2. Enable the embedding database in GDK

     gdk config set gitlab.rails.databases.embedding.enabled true

3. Run gdk reconfigure

4. Run database migrations to create the embedding database

     RAILS_ENV=development bin/rails db:migrate

Populate

Seed your development database with the embeddings for GitLab Documentation using this Rake task:

RAILS_ENV=development bundle exec rake gitlab:llm:embeddings:vertex:seed

This Rake Task populates the embeddings database with a vectorized representation of all GitLab Documentation. The file the Rake Task uses as a source is a snapshot of GitLab Documentation at some point in the past and is not updated regularly. As a result, it is helpful to know that this seed task creates embeddings based on GitLab Documentation that is out of date. Slightly outdated documentation embeddings are sufficient for the development environment, which is the use-case for the seed task.

When writing or updating tests related to embeddings, you may want to update the embeddings fixture file:

RAILS_ENV=development bundle exec rake gitlab:llm:embeddings:vertex:extract_embeddings

Use embeddings in specs

The seed Rake Task populates the development database with embeddings for all GitLab Documentation. The extract_embeddings Rake Task populates a fixture file with a subset of embeddings.

The set of questions listed in the Rake Task itself determines which embeddings are pulled into the fixture file. For example, one of the questions is "How can I reset my password?" The extract_embeddings Task pulls the most relevant embeddings for this question from the development database (which has data from the seed Rake Task) and saves those embeddings in ee/spec/fixtures/vertex_embeddings. This fixture is used in tests related to embeddings.

If you would like to change any of the questions supported in embeddings specs, update and re-run the extract_embeddings Rake Task.

In the specs where you need to use the embeddings, use the RSpec :ai_embedding_fixtures metadata.

context 'when asking about how to use GitLab', :ai_embedding_fixtures do
  # ...examples
end

Tips for local development

1. When responses are taking too long to appear in the user interface, consider restarting Sidekiq by running gdk restart rails-background-jobs. If that doesn't work, try gdk kill and then gdk start.
2. Alternatively, bypass Sidekiq entirely and run the chat service synchronously. This can help with debugging errors as GraphQL errors are now available in the network inspector instead of the Sidekiq logs. To do that temporary alter Llm::CompletionWorker.perform_async statements with Llm::CompletionWorker.perform_inline

Working with GitLab Duo Chat

View guidelines for working with GitLab Duo Chat.

Test AI features with AI Gateway locally

• Introduced in GitLab 16.8.

In order to develop an AI feature that is compatible with both SaaS and Self-managed GitLab instances, the feature must request to the AI Gateway instead of directly requesting to the 3rd party model providers. Therefore, a different setup is required from the SaaS-only AI features.

Setup

1. Setup CustomersDot:

   1. Install CustomersDot: internal video tutorial
      • This video loosely follows official installation steps
      • It also offers guidance on how to create a self-managed subscription. You will receive a cloud activation code in return.
        • Bookmark this link for creating Ultimate Self-Managed Subscription locally.
        • A list of other subscription plan ids are available here.

2. Setup GitLab Development Kit (GDK): internal video tutorial

   1. Install it as a separate GDK instance.

   2. Run gdk config set license.customer_portal_url 'http://localhost:5000'

   3. Set up gdk.test hostname.

   4. Follow Instruct your local CustomersDot instance to use the GitLab application

   5. Activate GitLab Enterprise license

      • To test Self Managed instances, follow Cloud Activation steps using the cloud activation code you received earlier.
      • To test SaaS, follow Activate GitLab Enterprise license with your license file.

   6. Export these environment variables in the same terminal session with gdk start:

      export AI_GATEWAY_URL=http://0.0.0.0:5052 # URL to the local AI Gateway instance
      export LLM_DEBUG=1                        # Enable debug logging

      Alternatively, you can create an env.runit file in the root of your GDK with the above snippet.

   7. Enable all AI feature flags:

   rake gitlab:duo:enable_feature_flags

3. Set up AI Gateway: internal video tutorial

   1. Install it.

   2. Ensure that the following environment variables are set in the .env file:

      AIGW_AUTH__BYPASS_EXTERNAL=true
      ANTHROPIC_API_KEY="[REDACTED]"        # IMPORTANT: Ensure you use Corp account. See https://gitlab.com/gitlab-org/gitlab/-/issues/435911#note_1701762954.

   3. (Optional) Configure OIDC if needed.

   4. Run poetry run ai_gateway.

   5. Visit OpenAPI playground (http://0.0.0.0:5052/docs), try an endpoint (e.g. /v1/chat/agent) and make sure you get a successful response. If something went wrong, check modelgateway_debug.log if it contains error information.

You are set, and should be able to verify AI feature by calling the following in GitLab-Rails console:

Gitlab::Llm::AiGateway::Client.new(User.first).stream(prompt: "\n\nHuman: Hi, how are you?\n\nAssistant:")

Verify the setup with GraphQL

1. Visit GraphQL explorer.

2. Execute the aiAction mutation. Here is an example:

   mutation {
     aiAction(
       input: {
         chat: {
           resourceId: "gid://gitlab/User/1",
           content: "Hello"
         }
       }
     ){
       requestId
       errors
     }
   }

3. (GitLab Duo Chat only) Execute the following query to fetch the response:

   query {
     aiMessages {
       nodes {
         requestId
         content
         role
         timestamp
         chunkId
         errors
       }
     }
   }

   If you can't fetch the response, check graphql_json.log, sidekiq_json.log, llm.log or modelgateway_debug.log if it contains error information.

Set OIDC provider in AI Gateway

1. Configure AI Gateway:
   1. Set the OIDC provider URLs. Make sure to adapt to the domain you use. Note that you can choose to only set either GitLab SaaS instance or CDot as a provider.
   2. Restart AI Gateway.
2. If GitLab instance is set as a provider, you need to configure GDK to run in SaaS mode:

   1. Export the following environment variables:

      export GITLAB_SIMULATE_SAAS=1 # Simulate a SaaS instance. See https://docs.gitlab.com/ee/development/ee_features.html#simulate-a-saas-instance.

   2. Restart GDK.

Experimental REST API

Use the experimental REST API endpoints to quickly experiment and prototype AI features.

The endpoints are:

• https://gitlab.example.com/api/v4/ai/experimentation/anthropic/complete
• https://gitlab.example.com/api/v4/ai/experimentation/vertex/chat

These endpoints are only for prototyping, not for rolling features out to customers.

In your local development environment, you can experiment with these endpoints locally with the feature flag enabled:

Feature.enable(:ai_experimentation_api)

On production, the experimental endpoints are only available to GitLab team members. Use a GitLab API token to authenticate.

Abstraction layer

GraphQL API

To connect to the AI provider API using the Abstraction Layer, use an extendable GraphQL API called aiAction. The input accepts key/value pairs, where the key is the action that needs to be performed. We only allow one AI action per mutation request.

Example of a mutation:

mutation {
  aiAction(input: {summarizeComments: {resourceId: "gid://gitlab/Issue/52"}}) {
    clientMutationId
  }
}