diff --git a/doc/development/ai_features/index.md b/doc/development/ai_features/index.md index ede2015c46b06075ef0970444ab523b0d721c3a9..a3007b337d93f057cadc495ecc8ab190404fd385 100644 --- a/doc/development/ai_features/index.md +++ b/doc/development/ai_features/index.md @@ -729,6 +729,10 @@ Gitlab::Llm::Anthropic::Client.new(user, unit_primitive: 'your_feature') In addition to standard logging in the GitLab Rails Monolith instance, specialized logging is available for features based on large language models (LLMs). +### Logged events + +Currently logged events are documented [here](logged_events.md). + ### Implementation #### Logger Class @@ -745,7 +749,7 @@ A feature flag named `expanded_ai_logging` controls the logging of sensitive dat Use the `conditional_info` helper method for conditional logging based on the feature flag status: - If the feature flag is enabled for the current user, it logs the information on `info` level (logs are accessible in Kibana). -- If the feature flag is disabled for the current user, it logs the information on `debug` level (logs are not accessible in Kibana). +- If the feature flag is disabled for the current user, it logs the information on `info` level, but without optional parameters (logs are accessible in Kibana, but only obligatory fields). ### Best Practices @@ -759,14 +763,14 @@ When implementing logging for LLM features, consider the following: ### Example Usage ```ruby -# Logging non-sensitive information -Gitlab::Llm::Logger.build.info("LLM feature initialized") +# including concern that handles logging +include Gitlab::Llm::Concerns::Logger # Logging potentially sensitive information -Gitlab::Llm::Logger.build.conditional_info(user, message:"User prompt processed: #{sanitized_prompt}") +log_conditional_info(user, message:"User prompt processed", event_name: 'ai_event', ai_component: 'abstraction_layer', prompt: sanitized_prompt) # Logging application error information -Gitlab::Llm::Logger.build.error(user, message: "System application error: #{sanitized_error_message}") +log_error(user, message: "System application error", event_name: 'ai_event', ai_component: 'abstraction_layer', error_message: sanitized_error_message) ``` **Important**: Please familiarize yourself with our [Data Retention Policy](../../user/gitlab_duo/data_usage.md#data-retention) and remember diff --git a/doc/development/ai_features/logged_events.md b/doc/development/ai_features/logged_events.md new file mode 100644 index 0000000000000000000000000000000000000000..6633267e1742e831393e7d09f53f0601155efbae --- /dev/null +++ b/doc/development/ai_features/logged_events.md @@ -0,0 +1,720 @@ +--- +stage: AI-powered +group: AI Framework +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Logged events + +In addition to standard logging in the GitLab Rails Monolith instance, specialized logging is available for features based on large language models (LLMs). + +## Events logged + +<!-- markdownlint-disable --> +<!-- vale off --> + +### Returning from Service due to validation + + - Description: user not permitted to perform action + - Class: `Llm::BaseService` + - Ai_event_name: permission_denied + - Level: info + - Arguments: + - none + - Part of the system: abstraction_layer + - Expanded logging?: no + - Rails: yes + - Sidekiq: no + +### Enqueuing CompletionWorker + + - Description: scheduling completion worker in sidekiq + - Class: `Llm::BaseService` + - Ai_event_name: worker_enqueued + - Level: info + - Arguments: + - user_id: message.user.id + - resource_id: message.resource&.id + - resource_class: message.resource&.class&.name + - request_id: message.request_id + - action_name: message.ai_action + - options: job_options + - Part of the system: abstraction_layer + - Expanded logging?: yes + - Rails: yes + - Sidekiq: no + +### aborting: missing resource + + - Description: If there is no resource for slash command + - Class: `Llm::ChatService` + - Ai_event_name: missing_resource + - Level: info + - Arguments: + - none + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: yes + - Sidekiq: no + +### Performing CompletionService + + - Description: performing completion + - Class: `Llm::Internal::CompletionService` + - Ai_event_name: completion_service_performed + - Level: info + - Arguments: + - user_id: prompt_message.user.to_gid + - resource_id: prompt_message.resource&.to_gid + - action_name: prompt_message.ai_action + - request_id: prompt_message.request_id + - client_subscription_id: prompt_message.client_subscription_id + - completion_service_name: completion_class_name + - Part of the system: abstraction_layer + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Answer from LLM response + + - Description: Get answer from response + - Class: `Gitlab::Llm::Chain::Answer` + - Ai_event_name: answer_received + - Level: info + - Arguments: + - llm_answer_content: content + - Part of the system: duo_chat + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Final answer + + - Description: Get final answer from response + - Class: `Gitlab::Llm::Chain::Answer` + - Ai_event_name: final_answer_received + - Level: info + - Arguments: + - llm_answer_content: content + - Part of the system: duo_chat + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Default final answer + + - Description: Default final answer: I'm sorry, I couldn't respond in time. Please try a more specific request or enter /clear to start a new chat. + - Class: `Gitlab::Llm::Chain::Answer` + - Ai_event_name: default_final_answer_received + - Level: info + - Arguments: + - error_code: "A6000" + - Part of the system: duo_chat + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Error message/ "Error" + + - Description: when answering with an error + - Class: `Gitlab::Llm::Chain::Answer` + - Ai_event_name: error_returned + - Level: error + - Arguments: + - error: content + - error_code: error_code + - source: source + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Received response from AI Gateway + + - Description: when response from AIGW is returned + - Class: `Gitlab::Llm::AiGateway::Client` + - Ai_event_name: response_received + - Level: info + - Arguments: + - response_from_llm: response_body + - Part of the system: abstraction_layer + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Received error from AI gateway + + - Description: when error is returned from AIGW for streaming command + - Class: `Gitlab::Llm::AiGateway::Client` + - Ai_event_name: error_response_received + - Level: error + - Arguments: + - response_from_llm: parsed_response.dig('detail' + - 0 + - 'msg') + - Part of the system: abstraction_layer + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Performing request to AI Gateway + + - Description: before performing request to the AI GW + - Class: `Gitlab::Llm::AiGateway::Client` + - Ai_event_name: performing_request + - Level: info + - Arguments: + - url: url + - body: body + - timeout: timeout + - stream: stream + - Part of the system: abstraction_layer + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Creating user access token + + - Description: creating short-lived token in AIGW + - Class: `Gitlab::Llm::AiGateway::CodeSuggestionsClient` + - Ai_event_name: user_token_created + - Level: info + - Arguments: + - none + - Part of the system: code suggestions + - Expanded logging?: no + - Rails: yes + - Sidekiq: no + +### Received response from Anthropic + + - Description: Received response + - Class: `Gitlab::Llm::Anthropic::Client` + - Ai_event_name: response_received + - Level: info + - Arguments: + - ai_request_type: request_type + - unit_primitive: unit_primitive + - Part of the system: abstraction_layer + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Response content + + - Description: Content of response + - Class: `Gitlab::Llm::Anthropic::Client` + - Ai_event_name: response_received + - Level: info + - Arguments: + - ai_request_type: request_type + - unit_primitive: unit_primitive + - response_from_llm: response_body + - Part of the system: abstraction_layer + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Performing request to Anthropic + + - Description: performing completion request + - Class: `Gitlab::Llm::Anthropic::Client` + - Ai_event_name: performing_request + - Level: info + - Arguments: + - options: options + - ai_request_type: request_type + - unit_primitive: unit_primitive + - Part of the system: abstraction_layer + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Searching docs from AI Gateway + + - Description: performing search docs request + - Class: `Gitlab::Llm::AiGateway::DocsClient` + - Ai_event_name: performing_request + - Level: info + - Arguments: + - options: options + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Searched docs content from AI Gateway + + - Description: response from AIGW with docs + - Class: `Gitlab::Llm::AiGateway::DocsClient` + - Ai_event_name: response_received + - Level: info + - Arguments: + - response_from_llm: response + - Part of the system: duo_chat + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Json parsing error during Question Categorization + + - Description: logged when json is not parsable + - Class: `Gitlab::Llm::Anthropic::Completions::CategorizeQuestions` + - Ai_event_name: error + - Level: error + - Arguments: + - none + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Response did not contain defined categories + + - Description: logged when response is not containing one of the defined categories + - Class: `Gitlab::Llm::Anthropic::Completions::CategorizeQuestions` + - Ai_event_name: error + - Level: error + - Arguments: + - none + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Picked tool + + - Description: information about tool picked by chat + - Class: `Gitlab::Llm::Chain::Agents::ZeroShot::Executor` + - Ai_event_name: picked_tool + - Level: info + - Arguments: + - duo_chat_tool: tool_class.to_s + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Made request to AI Client + + - Description: making request for chat + - Class: `Gitlab::Llm::Chain::Requests::AiGateway` + - Ai_event_name: response_received + - Level: info + - Arguments: + - prompt: prompt[:prompt] + - response_from_llm: response + - unit_primitive: unit_primitive + - Part of the system: duo_chat + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Streaming error + + - Description: Error returned when streaming + - Class: `Gitlab::Llm::Chain::Requests::Anthropic` + - Ai_event_name: error_response_received + - Level: error + - Arguments: + - error: data&.dig("error") + - Part of the system: abstraction_layer + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Got Final Result for documentation question content + + - Description: got result for documentation question - content + - Class: `Gitlab::Llm::Chain::Tools::EmbeddingsCompletion` + - Ai_event_name: response_received + - Level: info + - Arguments: + - prompt: final_prompt[:prompt] + - response_from_llm: final_prompt_result + - Part of the system: duo_chat + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Streaming error + + - Description: when error is returned from AIGW for streaming command in docs question + - Class: `Gitlab::Llm::Chain::Tools::EmbeddingsCompletion` + - Ai_event_name: error_response_received + - Level: error + - Arguments: + - error: error.message + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Answer already received from tool + + - Description: when tool was already picked up (content: You already have the answer from #{self.class::NAME} tool, read carefully.) + - Class: `Gitlab::Llm::Chain::Tools::Tool` + - Ai_event_name: incorrect_response_received + - Level: info + - Arguments: + - error_message: content + - Part of the system: duo_chat + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Tool cycling detected + + - Description: When tool is picked up again + - Class: `Gitlab::Llm::Chain::Tools::Tool` + - Ai_event_name: incorrect_response_received + - Level: info + - Arguments: + - picked_tool: cls.class.to_s + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Calling TanukiBot + + - Description: performing documentation request + - Class: `Gitlab::Llm::Chain::Tools::GitlabDocumentation::Executor` + - Ai_event_name: documentation_question_initial_request + - Level: info + - Arguments: + - none + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Error finding #{resource_name} + + - Description: when resource (issue/epic/mr) is not found + - Class: `Gitlab::Llm::Chain::Tools::Identifier` + - Ai_event_name: incorrect_response_received + - Level: error + - Arguments: + - error_message: authorizer.message + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Answer received from LLM + + - Description: response from identifier + - Class: `Gitlab::Llm::Chain::Tools::Identifier` + - Ai_event_name: response_received + - Level: info + - Arguments: + - response_from_llm: content + - Part of the system: duo_chat + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Json parsing error + + - Description: when json is malformed (Observation: JSON has an invalid format. Please retry) + - Class: `Gitlab::Llm::Chain::Tools::Identifier` + - Ai_event_name: error + - Level: error + - Arguments: + - none + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Resource already identified + + - Description: already identified resource (You already have identified the #{resource_name} #{resource.to_global_id}, read carefully.) + - Class: `Gitlab::Llm::Chain::Tools::Identifier` + - Ai_event_name: incorrect_response_received + - Level: info + - Arguments: + - error_message: content + - Part of the system: duo_chat + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Supported Issuable Typees Ability Allowed + + - Description: logging the ability (policy.can?) for the issue/epic + - Class: `Gitlab::Llm::Chain::Tools::SummarizeComments::Executor` + - Ai_event_name: permission + - Level: info + - Arguments: + - allowed: ability + - Part of the system: feature + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Supported Issuable Typees Ability Allowed + + - Description: logging the ability (policy.can?) for the issue/epic + - Class: `Gitlab::Llm::Chain::Tools::SummarizeComments::ExecutorOld` + - Ai_event_name: permission + - Level: info + - Arguments: + - allowed: ability + - Part of the system: feature + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Answer content for summarize_comments + + - Description: Answer for summarize comments feature + - Class: `Gitlab::Llm::Chain::Tools::SummarizeComments::ExecutorOld` + - Ai_event_name: response_received + - Level: info + - Arguments: + - response_from_llm: content + - Part of the system: feature + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Content of the prompt from chat request + + - Description: chat-related request + - Class: `Gitlab::Llm::Chain::Concerns::AiDependent` + - Ai_event_name: prompt_content + - Level: info + - Arguments: + - prompt: prompt_text + - Part of the system: duo_chat + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### "Too many requests, will retry in #{delay} seconds" + + - Description: When entered in exponential backoff loop + - Class: `Gitlab::Llm::Chain::Concerns::ExponentialBackoff` + - Ai_event_name: retrying_request + - Level: info + - Arguments: + - none + - Part of the system: abstraction_layer + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Resource not found + + - Description: Resource not found/not authorized + - Class: `Gitlab::Llm::Utils::Authorizer` + - Ai_event_name: permission_denied + - Level: info + - Arguments: + - error_code: "M3003" + - Part of the system: abstraction_layer + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### No access to Duo Chat + + - Description: No access to duo chat + - Class: `Gitlab::Llm::Utils::Authorizer` + - Ai_event_name: permission_denied + - Level: info + - Arguments: + - error_code: "M3004" + - Part of the system: abstraction_layer + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### AI is disabled + + - Description: AI is not enabled for container + - Class: `Gitlab::Llm::Utils::Authorizer` + - Ai_event_name: permission_denied + - Level: info + - Arguments: + - error_code: "M3002" + - Part of the system: abstraction_layer + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Performing request to Vertex + + - Description: performing request + - Class: `Gitlab::Llm::VertexAi::Client` + - Ai_event_name: performing_request + - Level: info + - Arguments: + - unit_primitive: unit_primitive + - options: config + - Part of the system: abstraction_layer + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Response content + + - Description: response from aigw - vertex -content + - Class: `Gitlab::Llm::VertexAi::Client` + - Ai_event_name: response_received + - Level: info + - Arguments: + - unit_primitive: unit_primitive + - response_from_llm: response.to_json + - Part of the system: abstraction_layer + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Received response from Vertex + + - Description: response from aigw - vertex + - Class: `Gitlab::Llm::VertexAi::Client` + - Ai_event_name: response_received + - Level: info + - Arguments: + - unit_primitive: unit_primitive + - Part of the system: abstraction_layer + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Empty response from Vertex + + - Description: empty response from aigw - vertex + - Class: `Gitlab::Llm::VertexAi::Client` + - Ai_event_name: empty_response_received + - Level: error + - Arguments: + - unit_primitive: unit_primitive + - Part of the system: abstraction_layer + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Surface an unknown event as a final answer to the user + + - Description: unknown event + - Class: `Gitlab::Llm::Chain::Agents::SingleActionExecutor` + - Ai_event_name: unknown_event + - Level: warn + - Arguments: + - none + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Failed to find a tool in GitLab Rails + + - Description: failed to find a tool + - Class: `Gitlab::Llm::Chain::Agents::SingleActionExecutor` + - Ai_event_name: tool_not_find + - Level: error + - Arguments: + - tool_name: tool_name + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Received an event from v2/chat/agent + + - Description: Received event + - Class: `Gitlab::Duo::Chat::StepExecutor` + - Ai_event_name: event_received + - Level: info + - Arguments: + - event: event + - Part of the system: duo_chat + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Failed to update observation + + - Description: Failed to update observation + - Class: `Gitlab::Duo::Chat::StepExecutor` + - Ai_event_name: agent_steps_empty + - Level: error + - Arguments: + - none + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Request to v2/chat/agent + + - Description: request + - Class: `Gitlab::Duo::Chat::StepExecutor` + - Ai_event_name: performing_request + - Level: info + - Arguments: + - params: params + - Part of the system: duo_chat + - Expanded logging?: yes + - Rails: no + - Sidekiq: yes + +### Finished streaming from v2/chat/agent + + - Description: finished streaming + - Class: `Gitlab::Duo::Chat::StepExecutor` + - Ai_event_name: streaming_finished + - Level: info + - Arguments: + - none + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Received error from Duo Chat Agent + + - Description: Error returned when streaming + - Class: `Gitlab::Duo::Chat::StepExecutor` + - Ai_event_name: error_returned + - Level: error + - Arguments: + - status: response.code + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Failed to parse a chunk from Duo Chat Agent + + - Description: failed to parse a chunk + - Class: `Gitlab::Duo::Chat::AgentEventParser` + - Ai_event_name: parsing_error + - Level: warn + - Arguments: + - event_json_size: event_json.length + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +### Failed to find the event class in GitLab-Rails + + - Description: no event class + - Class: `Gitlab::Duo::Chat::AgentEventParser` + - Ai_event_name: parsing_error + - Level: error + - Arguments: + - event_type: event['type'] + - Part of the system: duo_chat + - Expanded logging?: no + - Rails: no + - Sidekiq: yes + +<!-- markdownlint-enable --> +<!-- vale on -->