From d66f1ccd4015dfeaa4ed6fe398feae085f7f60f3 Mon Sep 17 00:00:00 2001 From: Hao Kung <HaoK@users.noreply.github.com> Date: Fri, 12 Jun 2020 17:10:19 -0700 Subject: [PATCH] Update helix doc with current info (#22855) --- docs/Helix.md | 61 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 45 insertions(+), 16 deletions(-) diff --git a/docs/Helix.md b/docs/Helix.md index d1134500f23..e8c733d5a88 100644 --- a/docs/Helix.md +++ b/docs/Helix.md @@ -16,27 +16,58 @@ To run Helix tests for one particular test project: This will restore, and then publish all the test project including some bootstrapping scripts that will install the correct dotnet runtime/sdk before running the test assembly on the helix machine(s), and upload the job to helix. +## Overview of the helix usage in our pipelines + +- Required queues: Windows10, OSX, Ubuntu1604 +- Full queue matrix: Windows[7, 81, 10], Ubuntu[1604, 1804, 2004], Centos7, Debian[8,9], Redhat7, Fedora28, Arm64 (Win10, Debian9) + +aspnetcore-ci runs non quarantined tests against the required helix queues as a required PR check and all builds on all branches. + +aspnetcore-quarantined-tests runs only quarantined tests against the required queues only on master every 4 hours. + +aspnetcore-helix-matrix runs non quarantined tests against all queues twice a day only on public master. + ## How do I look at the results of a helix run on Azure Pipelines? -There's a link embedded in the build.cmd log of the helix target on Azure Pipelines, near the bottom right that will look something like this: +The easiest way to look at a test failure is via the tests tab in azdo which now should show a summary of the errors and have attachments to the relevant console logs. + +You can also drill down into the helix web apis if you take the HelixJobId from the Debug tab of a failing test, and the HelixWorkItemName and go to: `https://helix.dot.net/api/2019-06-17/jobs/<jobId>/workitems/<workitemname>` which will show you more urls you can drill into for more info. + +There's also a link embedded in the build.cmd log of the Tests: Helix x64 job on Azure Pipelines, near the bottom right that will look something like this: ``` text -2019-02-07T21:55:48.1516089Z Results will be available from https://mc.dot.net/#/user/aspnetcore/pr~2Faspnet~2Faspnetcore/ci/20190207.34 -2019-02-07T21:56:43.2209607Z Job 0dedeef6-210e-4815-89f9-fd07513179fe is completed with 108 finished work items. -2019-02-07T21:56:43.5091018Z Job 4c45a464-9464-4321-906c-2503320066b0 is completed with 108 finished work items. -2019-02-07T21:56:43.6863473Z Job 91a826de-fd51-42c2-9e7e-84bbe18b16cf is completed with 108 finished work items. -2019-02-07T21:56:43.8591328Z Job b3595ab8-049d-4775-9cea-d1140a0cb446 is completed with 108 finished work items. -2019-02-07T21:56:44.0384313Z Job 2f174f2b-f6b1-4683-8303-3f120865c341 is completed with 108 finished work items. -2019-02-07T21:56:44.2069520Z Job b9387311-e670-4e18-9c84-479b7bfe67d1 is completed with 111 finished work items. -2019-02-07T21:56:44.3946686Z Job 43582e31-5648-47be-ac42-8a5e4129f15f is completed with 108 finished work items. -2019-02-07T21:56:44.5568847Z Job 1e6b0051-21a4-4b75-93f3-f739bb71d5dc is completed with 108 finished work items. -2019-02-07T22:01:26.0028154Z Job d597c581-f81b-446c-8daf-7c6511b526f7 is completed with 108 finished work items. -2019-02-07T22:06:33.6898567Z Job 82f27d4c-9099-4f0e-b383-870c24d8dc2c is completed with 108 finished work items. +Uploading payloads for Job on Ubuntu.1604.Amd64.Open... + Finished uploading payloads for Job on Ubuntu.1604.Amd64.Open... + Sending Job to Ubuntu.1604.Amd64.Open... + Sent Helix Job a5cbf405-1363-452f-af4b-de5b2a61c8cf + Uploading payloads for Job on Windows.10.Amd64.Open... + Finished uploading payloads for Job on Windows.10.Amd64.Open... + Sending Job to Windows.10.Amd64.Open... + Sent Helix Job cbec3697-c298-412a-953a-e375e49d1fe0 + Uploading payloads for Job on OSX.1014.Amd64.Open... + Finished uploading payloads for Job on OSX.1014.Amd64.Open... + Sending Job to OSX.1014.Amd64.Open... + Sent Helix Job a54359cf-f74d-4d02-9faf-07e0a8380995 + Waiting for completion of job cbec3697-c298-412a-953a-e375e49d1fe0 + Waiting for completion of job a54359cf-f74d-4d02-9faf-07e0a8380995 + Waiting for completion of job a5cbf405-1363-452f-af4b-de5b2a61c8cf + Job a54359cf-f74d-4d02-9faf-07e0a8380995 is completed with 136 finished work items. + Job cbec3697-c298-412a-953a-e375e49d1fe0 is completed with 156 finished work items. + Job a5cbf405-1363-452f-af4b-de5b2a61c8cf is completed with 136 finished work items. + Stopping Azure Pipelines Test Run Ubuntu.1604.Amd64.Open + Stopping Azure Pipelines Test Run Windows.10.Amd64.Open + Stopping Azure Pipelines Test Run OSX.1014.Amd64.Open +F:\workspace\_work\1\s\.packages\microsoft.dotnet.helix.sdk\5.0.0-beta.20280.1\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : Work item a5cbf405-1363-452f-af4b-de5b2a61c8cf/Microsoft.AspNetCore.Authentication.Test--net5.0 in job a5cbf405-1363-452f-af4b-de5b2a61c8cf has failed. [F:\workspace\_work\1\s\eng\helix\helix.proj] +F:\workspace\_work\1\s\.packages\microsoft.dotnet.helix.sdk\5.0.0-beta.20280.1\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : Failure log: https://helix.dot.net/api/2019-06-17/jobs/a5cbf405-1363-452f-af4b-de5b2a61c8cf/workitems/Microsoft.AspNetCore.Authentication.Test--net5.0/console [F:\workspace\_work\1\s\eng\helix\helix.proj] +##[error].packages\microsoft.dotnet.helix.sdk\5.0.0-beta.20280.1\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item a5cbf405-1363-452f-af4b-de5b2a61c8cf/Microsoft.AspNetCore.Authentication.Test--net5.0 in job a5cbf405-1363-452f-af4b-de5b2a61c8cf has failed. +Failure log: https://helix.dot.net/api/2019-06-17/jobs/a5cbf405-1363-452f-af4b-de5b2a61c8cf/workitems/Microsoft.AspNetCore.Authentication.Test--net5.0/console ``` -The link will take you to an overview of all the tests with clickable links to the logs and each run broken down by queue. +The https://helix.dot.net/ home page displays information about the available public queues (nothing about the related BYOC pools and queues or the internal Helix queues) -All of the helix runs for aspnetcore can be found here <https://mc.dot.net/#/user/aspnetcore/builds>. +Some superficial information about both BYOC and Helix agents is available at https://github.com/dotnet/arcade/blob/8ca46105193bd25c95af49bc6cd3604aaefec980/Documentation/AzureDevOps/AzureDevOpsOnboarding.md#agent-queues + +More detailed and always up-to-date information about all of the agents is available at https://helix.dot.net/api/2018-03-14/info/queues ## What do I do if a test fails? @@ -48,8 +79,6 @@ cd <the publish directory> dotnet vstest My.Tests.dll ``` -If that doesn't help, you can try the Get Repro environment link from mission control and try to debug that way. - ## Differences from running tests locally Most tests that don't just work on helix automatically are ones that depend on the source code being accessible. The helix payloads only contain whatever is in the publish directories, so any thing else that test depends on will need to be included to the payload. -- GitLab