@@ -16,27 +16,58 @@ To run Helix tests for one particular test project:
...
@@ -16,27 +16,58 @@ To run Helix tests for one particular test project:
This will restore, and then publish all the test project including some bootstrapping scripts that will install the correct dotnet runtime/sdk before running the test assembly on the helix machine(s), and upload the job to helix.
This will restore, and then publish all the test project including some bootstrapping scripts that will install the correct dotnet runtime/sdk before running the test assembly on the helix machine(s), and upload the job to helix.
aspnetcore-ci runs non quarantined tests against the required helix queues as a required PR check and all builds on all branches.
aspnetcore-quarantined-tests runs only quarantined tests against the required queues only on master every 4 hours.
aspnetcore-helix-matrix runs non quarantined tests against all queues twice a day only on public master.
## How do I look at the results of a helix run on Azure Pipelines?
## How do I look at the results of a helix run on Azure Pipelines?
There's a link embedded in the build.cmd log of the helix target on Azure Pipelines, near the bottom right that will look something like this:
The easiest way to look at a test failure is via the tests tab in azdo which now should show a summary of the errors and have attachments to the relevant console logs.
You can also drill down into the helix web apis if you take the HelixJobId from the Debug tab of a failing test, and the HelixWorkItemName and go to: `https://helix.dot.net/api/2019-06-17/jobs/<jobId>/workitems/<workitemname>` which will show you more urls you can drill into for more info.
There's also a link embedded in the build.cmd log of the Tests: Helix x64 job on Azure Pipelines, near the bottom right that will look something like this:
``` text
``` text
2019-02-07T21:55:48.1516089Z Results will be available from https://mc.dot.net/#/user/aspnetcore/pr~2Faspnet~2Faspnetcore/ci/20190207.34
Uploading payloads for Job on Ubuntu.1604.Amd64.Open...
2019-02-07T21:56:43.2209607Z Job 0dedeef6-210e-4815-89f9-fd07513179fe is completed with 108 finished work items.
Finished uploading payloads for Job on Ubuntu.1604.Amd64.Open...
2019-02-07T21:56:43.5091018Z Job 4c45a464-9464-4321-906c-2503320066b0 is completed with 108 finished work items.
Sending Job to Ubuntu.1604.Amd64.Open...
2019-02-07T21:56:43.6863473Z Job 91a826de-fd51-42c2-9e7e-84bbe18b16cf is completed with 108 finished work items.
Sent Helix Job a5cbf405-1363-452f-af4b-de5b2a61c8cf
2019-02-07T21:56:43.8591328Z Job b3595ab8-049d-4775-9cea-d1140a0cb446 is completed with 108 finished work items.
Uploading payloads for Job on Windows.10.Amd64.Open...
2019-02-07T21:56:44.0384313Z Job 2f174f2b-f6b1-4683-8303-3f120865c341 is completed with 108 finished work items.
Finished uploading payloads for Job on Windows.10.Amd64.Open...
2019-02-07T21:56:44.2069520Z Job b9387311-e670-4e18-9c84-479b7bfe67d1 is completed with 111 finished work items.
Sending Job to Windows.10.Amd64.Open...
2019-02-07T21:56:44.3946686Z Job 43582e31-5648-47be-ac42-8a5e4129f15f is completed with 108 finished work items.
Sent Helix Job cbec3697-c298-412a-953a-e375e49d1fe0
2019-02-07T21:56:44.5568847Z Job 1e6b0051-21a4-4b75-93f3-f739bb71d5dc is completed with 108 finished work items.
Uploading payloads for Job on OSX.1014.Amd64.Open...
2019-02-07T22:01:26.0028154Z Job d597c581-f81b-446c-8daf-7c6511b526f7 is completed with 108 finished work items.
Finished uploading payloads for Job on OSX.1014.Amd64.Open...
2019-02-07T22:06:33.6898567Z Job 82f27d4c-9099-4f0e-b383-870c24d8dc2c is completed with 108 finished work items.
Sending Job to OSX.1014.Amd64.Open...
Sent Helix Job a54359cf-f74d-4d02-9faf-07e0a8380995
Waiting for completion of job cbec3697-c298-412a-953a-e375e49d1fe0
Waiting for completion of job a54359cf-f74d-4d02-9faf-07e0a8380995
Waiting for completion of job a5cbf405-1363-452f-af4b-de5b2a61c8cf
Job a54359cf-f74d-4d02-9faf-07e0a8380995 is completed with 136 finished work items.
Job cbec3697-c298-412a-953a-e375e49d1fe0 is completed with 156 finished work items.
Job a5cbf405-1363-452f-af4b-de5b2a61c8cf is completed with 136 finished work items.
Stopping Azure Pipelines Test Run Ubuntu.1604.Amd64.Open
Stopping Azure Pipelines Test Run Windows.10.Amd64.Open
Stopping Azure Pipelines Test Run OSX.1014.Amd64.Open
F:\workspace\_work\1\s\.packages\microsoft.dotnet.helix.sdk\5.0.0-beta.20280.1\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : Work item a5cbf405-1363-452f-af4b-de5b2a61c8cf/Microsoft.AspNetCore.Authentication.Test--net5.0 in job a5cbf405-1363-452f-af4b-de5b2a61c8cf has failed. [F:\workspace\_work\1\s\eng\helix\helix.proj]
##[error].packages\microsoft.dotnet.helix.sdk\5.0.0-beta.20280.1\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(76,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item a5cbf405-1363-452f-af4b-de5b2a61c8cf/Microsoft.AspNetCore.Authentication.Test--net5.0 in job a5cbf405-1363-452f-af4b-de5b2a61c8cf has failed.
The link will take you to an overview of all the tests with clickable links to the logs and each run broken down by queue.
The https://helix.dot.net/ home page displays information about the available public queues (nothing about the related BYOC pools and queues or the internal Helix queues)
All of the helix runs for aspnetcore can be found here <https://mc.dot.net/#/user/aspnetcore/builds>.
Some superficial information about both BYOC and Helix agents is available at https://github.com/dotnet/arcade/blob/8ca46105193bd25c95af49bc6cd3604aaefec980/Documentation/AzureDevOps/AzureDevOpsOnboarding.md#agent-queues
More detailed and always up-to-date information about all of the agents is available at https://helix.dot.net/api/2018-03-14/info/queues
## What do I do if a test fails?
## What do I do if a test fails?
...
@@ -48,8 +79,6 @@ cd <the publish directory>
...
@@ -48,8 +79,6 @@ cd <the publish directory>
dotnetvstestMy.Tests.dll
dotnetvstestMy.Tests.dll
```
```
If that doesn't help, you can try the Get Repro environment link from mission control and try to debug that way.
## Differences from running tests locally
## Differences from running tests locally
Most tests that don't just work on helix automatically are ones that depend on the source code being accessible. The helix payloads only contain whatever is in the publish directories, so any thing else that test depends on will need to be included to the payload.
Most tests that don't just work on helix automatically are ones that depend on the source code being accessible. The helix payloads only contain whatever is in the publish directories, so any thing else that test depends on will need to be included to the payload.
