diff --git a/docs/Helix.md b/docs/Helix.md index 660b0b52f09e95437e062a703bf70a6aaac54929..d1134500f2320df0d8828e5ed13be47f7c4a5395 100644 --- a/docs/Helix.md +++ b/docs/Helix.md @@ -1,5 +1,4 @@ -Helix testing in ASP.NET Core -============================== +# Helix testing in ASP.NET Core Helix is the distributed test platform that we use to run tests. We build a helix payload that contains the publish directory of every test project that we want to test send a job with with this payload to a set of queues for the various combinations of OS that we want to test @@ -11,16 +10,17 @@ For more info about helix see: [SDK](https://github.com/dotnet/arcade/blob/maste To run Helix tests for one particular test project: -``` +``` powershell .\eng\scripts\RunHelix.ps1 -Project path\mytestproject.csproj ``` This will restore, and then publish all the test project including some bootstrapping scripts that will install the correct dotnet runtime/sdk before running the test assembly on the helix machine(s), and upload the job to helix. - ## How do I look at the results of a helix run on Azure Pipelines? + There's a link embedded in the build.cmd log of the helix target on Azure Pipelines, near the bottom right that will look something like this: -``` + +``` text 2019-02-07T21:55:48.1516089Z Results will be available from https://mc.dot.net/#/user/aspnetcore/pr~2Faspnet~2Faspnetcore/ci/20190207.34 2019-02-07T21:56:43.2209607Z Job 0dedeef6-210e-4815-89f9-fd07513179fe is completed with 108 finished work items. 2019-02-07T21:56:43.5091018Z Job 4c45a464-9464-4321-906c-2503320066b0 is completed with 108 finished work items. @@ -36,11 +36,13 @@ There's a link embedded in the build.cmd log of the helix target on Azure Pipeli The link will take you to an overview of all the tests with clickable links to the logs and each run broken down by queue. -All of the helix runs for aspnetcore can be found here https://mc.dot.net/#/user/aspnetcore/builds +All of the helix runs for aspnetcore can be found here <https://mc.dot.net/#/user/aspnetcore/builds>. ## What do I do if a test fails? + You can simulate how most tests run locally: -``` + +``` powershell dotnet publish cd <the publish directory> dotnet vstest My.Tests.dll @@ -49,11 +51,12 @@ dotnet vstest My.Tests.dll If that doesn't help, you can try the Get Repro environment link from mission control and try to debug that way. ## Differences from running tests locally + Most tests that don't just work on helix automatically are ones that depend on the source code being accessible. The helix payloads only contain whatever is in the publish directories, so any thing else that test depends on will need to be included to the payload. This can be accomplished by using the `HelixContent` property like so. -``` +``` msbuild <ItemGroup> <HelixContent Include="$(RepoRoot)src\KeepMe.js"/> <HelixContent Include="$(RepoRoot)src\Project\**"/> @@ -62,7 +65,7 @@ This can be accomplished by using the `HelixContent` property like so. By default, these files will be included in the root directory. To include these files in a different directory, you can use either the `Link` or `LinkBase` attributes to set the included path. -``` +``` msbuild <ItemGroup> <HelixContent Include="$(RepoRoot)src\KeepMe.js" Link="$(MSBuildThisFileDirectory)\myassets\KeepMe.js"/> <HelixContent Include="$(RepoRoot)src\Project\**" LinkBase="$(MSBuildThisFileDirectory)\myassets"/> @@ -70,8 +73,10 @@ By default, these files will be included in the root directory. To include these ``` ## How to skip tests on helix + There are two main ways to opt out of helix + - Skipping the entire test project via `<BuildHelixPayload>false</BuildHelixPayload>` in csproj (the default value for this is IsTestProject). -- Skipping an individual test via `[SkipOnHelix("url to github issue")]` which might require including a compile reference to: `<Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />` +- Skipping an individual test via `[SkipOnHelix("url to github issue")]`. Make sure to file an issue for any skipped tests and include that in a comment next to either of these diff --git a/eng/helix/content/RunTests/TestRunner.cs b/eng/helix/content/RunTests/TestRunner.cs index 3643df3aacad174dd07882013f6ea65bef892622..da4b4c228f0c8ed0eefc9dad0dc9faf8e02db1b1 100644 --- a/eng/helix/content/RunTests/TestRunner.cs +++ b/eng/helix/content/RunTests/TestRunner.cs @@ -272,7 +272,7 @@ namespace RunTests // Filter syntax: https://github.com/Microsoft/vstest-docs/blob/master/docs/filter.md var result = await ProcessUtil.RunAsync($"{Options.DotnetRoot}/dotnet", - commonTestArgs + " --TestCaseFilter:\"Quarantined!=true\"", + commonTestArgs + " --TestCaseFilter:\"Quarantined!=true|Quarantined=false\"", environmentVariables: EnvironmentVariables, outputDataReceived: Console.WriteLine, errorDataReceived: Console.Error.WriteLine, diff --git a/eng/targets/CSharp.Common.props b/eng/targets/CSharp.Common.props index 412aa80c54ede3771ce3f932039876b2c7248da9..c63d4ac7c7661b534196012a71fca3ec0c77e278 100644 --- a/eng/targets/CSharp.Common.props +++ b/eng/targets/CSharp.Common.props @@ -42,6 +42,7 @@ <Reference Include="Microsoft.AspNetCore.Testing" /> <Reference Include="Moq" /> <Reference Include="NETStandard.Library" /> + <Compile Include="$(SharedSourceRoot)test\SuccessfulTests.cs" LinkBase="SharedTests" /> </ItemGroup> <Import Project="$(RepoRoot)src\Testing\src\build\Microsoft.AspNetCore.Testing.props" Condition=" '$(IsTestProject)' == 'true' " /> diff --git a/src/Analyzers/Analyzers/test/Microsoft.AspNetCore.Analyzers.Test.csproj b/src/Analyzers/Analyzers/test/Microsoft.AspNetCore.Analyzers.Test.csproj index 3d791a7367b9abbd0deb6e0faa25abb20d8e1dcd..8f85e4be8c9f06f72f0dc10f228abeae08e73c16 100644 --- a/src/Analyzers/Analyzers/test/Microsoft.AspNetCore.Analyzers.Test.csproj +++ b/src/Analyzers/Analyzers/test/Microsoft.AspNetCore.Analyzers.Test.csproj @@ -7,7 +7,6 @@ </PropertyGroup> <ItemGroup> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Content Include="TestFiles\**\*.*" CopyToOutputDirectory="PreserveNewest" /> </ItemGroup> diff --git a/src/Components/Analyzers/test/Microsoft.AspNetCore.Components.Analyzers.Tests.csproj b/src/Components/Analyzers/test/Microsoft.AspNetCore.Components.Analyzers.Tests.csproj index 4d1ad9ba522a6b3c7c651e7ee398e6dd5c5ad934..935e05301e92659cc9e4eb4788e4fba10ca99c93 100644 --- a/src/Components/Analyzers/test/Microsoft.AspNetCore.Components.Analyzers.Tests.csproj +++ b/src/Components/Analyzers/test/Microsoft.AspNetCore.Components.Analyzers.Tests.csproj @@ -16,7 +16,6 @@ </ItemGroup> <ItemGroup> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Content Include="TestFiles\**\*.*" CopyToPublishDirectory="PreserveNewest" /> </ItemGroup> </Project> diff --git a/src/Components/Authorization/test/Microsoft.AspNetCore.Components.Authorization.Tests.csproj b/src/Components/Authorization/test/Microsoft.AspNetCore.Components.Authorization.Tests.csproj index 5f7ae8f772c215b471fe4cc7dfc02dc5199b4c5e..ed1092d444ffb65793c1893a6b016f7fdbbd1c69 100644 --- a/src/Components/Authorization/test/Microsoft.AspNetCore.Components.Authorization.Tests.csproj +++ b/src/Components/Authorization/test/Microsoft.AspNetCore.Components.Authorization.Tests.csproj @@ -12,7 +12,6 @@ <ItemGroup> <Compile Include="$(ComponentsSharedSourceRoot)test\**\*.cs" LinkBase="Helpers" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> </ItemGroup> </Project> diff --git a/src/Components/Components/test/Microsoft.AspNetCore.Components.Tests.csproj b/src/Components/Components/test/Microsoft.AspNetCore.Components.Tests.csproj index fd2a5406c165b36f0c7845e8fb8ee71b516470f0..8f414c52f80c6881c9e63db7febafa7703eb2786 100644 --- a/src/Components/Components/test/Microsoft.AspNetCore.Components.Tests.csproj +++ b/src/Components/Components/test/Microsoft.AspNetCore.Components.Tests.csproj @@ -12,7 +12,6 @@ <ItemGroup> <Compile Include="$(ComponentsSharedSourceRoot)test\**\*.cs" LinkBase="Helpers" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> </ItemGroup> </Project> diff --git a/src/Components/Forms/test/Microsoft.AspNetCore.Components.Forms.Tests.csproj b/src/Components/Forms/test/Microsoft.AspNetCore.Components.Forms.Tests.csproj index 6c611379a5ca0e74965d36bd031e5999db6a7cb2..c6a022be09237ff183c3b835c96a02d278f34685 100644 --- a/src/Components/Forms/test/Microsoft.AspNetCore.Components.Forms.Tests.csproj +++ b/src/Components/Forms/test/Microsoft.AspNetCore.Components.Forms.Tests.csproj @@ -11,7 +11,6 @@ <ItemGroup> <Compile Include="$(ComponentsSharedSourceRoot)test\**\*.cs" LinkBase="Helpers" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> </ItemGroup> </Project> diff --git a/src/Components/Web/test/Microsoft.AspNetCore.Components.Web.Tests.csproj b/src/Components/Web/test/Microsoft.AspNetCore.Components.Web.Tests.csproj index 5c8a5a0d1998c91f585a4e3d9512faa394b6ccd5..b7697ce8adb827c271b566c9fca708642119cc92 100644 --- a/src/Components/Web/test/Microsoft.AspNetCore.Components.Web.Tests.csproj +++ b/src/Components/Web/test/Microsoft.AspNetCore.Components.Web.Tests.csproj @@ -13,7 +13,6 @@ <ItemGroup> <Compile Include="$(ComponentsSharedSourceRoot)test\**\*.cs" LinkBase="Helpers" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> </ItemGroup> </Project> diff --git a/src/Components/WebAssembly/Build/test/Microsoft.AspNetCore.Components.WebAssembly.Build.Tests.csproj b/src/Components/WebAssembly/Build/test/Microsoft.AspNetCore.Components.WebAssembly.Build.Tests.csproj index 3f622c9f94c6423c0fdd5e25fac472277abc80e3..fd25c908f9cb2e506785bfd0434bd227e95c4a08 100644 --- a/src/Components/WebAssembly/Build/test/Microsoft.AspNetCore.Components.WebAssembly.Build.Tests.csproj +++ b/src/Components/WebAssembly/Build/test/Microsoft.AspNetCore.Components.WebAssembly.Build.Tests.csproj @@ -34,7 +34,6 @@ <ItemGroup> <ProjectReference Include="..\..\testassets\StandaloneApp\StandaloneApp.csproj" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Compile Include="$(SharedSourceRoot)CommandLineUtils\**\*.cs" /> </ItemGroup> diff --git a/src/Components/WebAssembly/Build/test/RuntimeDependenciesResolverTest.cs b/src/Components/WebAssembly/Build/test/RuntimeDependenciesResolverTest.cs index 54df8ed79bea9f2557ebc49e14e95e2cffae349e..a4c802e3215612f254b57cf26909002a1dd4e61c 100644 --- a/src/Components/WebAssembly/Build/test/RuntimeDependenciesResolverTest.cs +++ b/src/Components/WebAssembly/Build/test/RuntimeDependenciesResolverTest.cs @@ -7,7 +7,7 @@ using System.IO; using System.Linq; using System.Reflection; using System.Text; -using Microsoft.AspNetCore.Testing.xunit; +using Microsoft.AspNetCore.Testing; using Xunit; namespace Microsoft.AspNetCore.Components.WebAssembly.Build diff --git a/src/DataProtection/Extensions/test/Microsoft.AspNetCore.DataProtection.Extensions.Tests.csproj b/src/DataProtection/Extensions/test/Microsoft.AspNetCore.DataProtection.Extensions.Tests.csproj index f184844f0cb4cf1d0221a57c3ba935fff1aa215c..6d8f3d313a240f5362a77043d26a9cf24bb781b8 100644 --- a/src/DataProtection/Extensions/test/Microsoft.AspNetCore.DataProtection.Extensions.Tests.csproj +++ b/src/DataProtection/Extensions/test/Microsoft.AspNetCore.DataProtection.Extensions.Tests.csproj @@ -7,7 +7,6 @@ <ItemGroup> <Compile Include="..\..\shared\test\*.cs" /> <Content Include="TestFiles\**\*" CopyToOutputDirectory="PreserveNewest" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> </ItemGroup> <ItemGroup> diff --git a/src/Hosting/Hosting/test/Microsoft.AspNetCore.Hosting.Tests.csproj b/src/Hosting/Hosting/test/Microsoft.AspNetCore.Hosting.Tests.csproj index 0f7f87f097b1a1ce468832c2dd4d84451370542f..cf25a1547daa801a9b0b5c51541bf955b52e49d4 100644 --- a/src/Hosting/Hosting/test/Microsoft.AspNetCore.Hosting.Tests.csproj +++ b/src/Hosting/Hosting/test/Microsoft.AspNetCore.Hosting.Tests.csproj @@ -5,7 +5,6 @@ </PropertyGroup> <ItemGroup> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Compile Include="$(SharedSourceRoot)EventSource.Testing\TestEventListener.cs" /> <Compile Include="$(SharedSourceRoot)EventSource.Testing\TestCounterListener.cs" /> <Content Include="testroot\**\*" CopyToOutputDirectory="PreserveNewest" CopyToPublishDirectory="PreserveNewest" /> diff --git a/src/Http/Http/test/BindingAddressTests.cs b/src/Http/Http/test/BindingAddressTests.cs index 858aafeefb1e7da9b693788d07163e7df2c1004b..b130bcd2df402af9dde96a7cecf2d392a99c87dc 100644 --- a/src/Http/Http/test/BindingAddressTests.cs +++ b/src/Http/Http/test/BindingAddressTests.cs @@ -2,11 +2,9 @@ // Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information. using System; -using System.Collections.Generic; -using System.Text; using Microsoft.AspNetCore.Testing; -using Microsoft.AspNetCore.Testing.xunit; using Xunit; + namespace Microsoft.AspNetCore.Http.Tests { public class BindingAddressTests diff --git a/src/Identity/ApiAuthorization.IdentityServer/test/Microsoft.AspNetCore.ApiAuthorization.IdentityServer.Tests.csproj b/src/Identity/ApiAuthorization.IdentityServer/test/Microsoft.AspNetCore.ApiAuthorization.IdentityServer.Tests.csproj index e50086da71970ccef917ce91b46635a112e3d800..8b291f49797b03f0c06a073616984f4b449be7f9 100644 --- a/src/Identity/ApiAuthorization.IdentityServer/test/Microsoft.AspNetCore.ApiAuthorization.IdentityServer.Tests.csproj +++ b/src/Identity/ApiAuthorization.IdentityServer/test/Microsoft.AspNetCore.ApiAuthorization.IdentityServer.Tests.csproj @@ -12,7 +12,6 @@ <FrameworkReference Remove="Microsoft.AspNetCore.App" /> <Reference Include="Microsoft.AspNetCore.ApiAuthorization.IdentityServer" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> </ItemGroup> <ItemGroup> diff --git a/src/Identity/test/Identity.Test/Microsoft.AspNetCore.Identity.Test.csproj b/src/Identity/test/Identity.Test/Microsoft.AspNetCore.Identity.Test.csproj index ee81519dde1757848e3be55508ac58da58ed4919..33785fb7ce8d3accceae339383ca21e2f03eb98e 100644 --- a/src/Identity/test/Identity.Test/Microsoft.AspNetCore.Identity.Test.csproj +++ b/src/Identity/test/Identity.Test/Microsoft.AspNetCore.Identity.Test.csproj @@ -6,7 +6,6 @@ <ItemGroup> <Compile Include="$(IdentityTestSharedSourceRoot)**\*.cs" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> </ItemGroup> <ItemGroup> diff --git a/src/Middleware/Diagnostics.EntityFrameworkCore/test/FunctionalTests/Diagnostics.EFCore.FunctionalTests.csproj b/src/Middleware/Diagnostics.EntityFrameworkCore/test/FunctionalTests/Diagnostics.EFCore.FunctionalTests.csproj index d784bb29f6aba86dfe6840583e245f14955b6ae5..d8c8f7598977b7e788cffd5f083cf6e67b545bb8 100644 --- a/src/Middleware/Diagnostics.EntityFrameworkCore/test/FunctionalTests/Diagnostics.EFCore.FunctionalTests.csproj +++ b/src/Middleware/Diagnostics.EntityFrameworkCore/test/FunctionalTests/Diagnostics.EFCore.FunctionalTests.csproj @@ -15,8 +15,6 @@ <!-- Avoid CS1705 errors due to mix of assemblies brought in transitively. --> <Reference Include="Microsoft.Extensions.DependencyInjection.Abstractions" /> <Reference Include="Microsoft.Extensions.Logging.Abstractions" /> - - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> </ItemGroup> </Project> diff --git a/src/Middleware/NodeServices/test/Microsoft.AspNetCore.NodeServices.Tests.csproj b/src/Middleware/NodeServices/test/Microsoft.AspNetCore.NodeServices.Tests.csproj index f8e0099b04acdfd1b630affc1bf8fdebf0d564f3..2d1a5aab26b4b4934700b814fac3cdd4ab839c7a 100644 --- a/src/Middleware/NodeServices/test/Microsoft.AspNetCore.NodeServices.Tests.csproj +++ b/src/Middleware/NodeServices/test/Microsoft.AspNetCore.NodeServices.Tests.csproj @@ -8,7 +8,6 @@ <ItemGroup> <Reference Include="Microsoft.AspNetCore.NodeServices" /> <Reference Include="Microsoft.AspNetCore.TestHost" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Content Include="js\**\*" /> </ItemGroup> diff --git a/src/Mvc/Mvc.Analyzers/test/Mvc.Analyzers.Test.csproj b/src/Mvc/Mvc.Analyzers/test/Mvc.Analyzers.Test.csproj index bcbc066c953330f9a29fb1e4c7b7c6b67bd09349..0d2fecc2724bf02bdeaa8b5257775bfd8e2675bd 100644 --- a/src/Mvc/Mvc.Analyzers/test/Mvc.Analyzers.Test.csproj +++ b/src/Mvc/Mvc.Analyzers/test/Mvc.Analyzers.Test.csproj @@ -8,7 +8,6 @@ <ItemGroup> <None Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Compile Remove="TestFiles\TagHelpersInCodeBlocksAnalyzerTest\*.*" /> <Content Include="TestFiles\**\*.*" CopyToPublishDirectory="PreserveNewest" /> </ItemGroup> diff --git a/src/Mvc/Mvc.Api.Analyzers/test/Mvc.Api.Analyzers.Test.csproj b/src/Mvc/Mvc.Api.Analyzers/test/Mvc.Api.Analyzers.Test.csproj index a37838a6bfcace58a2bc5ff4327d2f86d0cfa13f..3b4f947e80c9f6dd58e9add99be1da0f5ae170f9 100644 --- a/src/Mvc/Mvc.Api.Analyzers/test/Mvc.Api.Analyzers.Test.csproj +++ b/src/Mvc/Mvc.Api.Analyzers/test/Mvc.Api.Analyzers.Test.csproj @@ -8,7 +8,6 @@ <ItemGroup> <Compile Include="..\..\Mvc.Analyzers\test\Infrastructure\MvcDiagnosticAnalyzerRunner.cs" Link="Infrastructure\MvcDiagnosticAnalyzerRunner.cs" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Content Include="TestFiles\**\*.*" CopyToPublishDirectory="PreserveNewest" /> </ItemGroup> diff --git a/src/ProjectTemplates/test/ProjectTemplates.Tests.csproj b/src/ProjectTemplates/test/ProjectTemplates.Tests.csproj index 569804c90a601c339333d72bf64f89384229e052..63fcb43fe719e67c26e57f823af38cc2b4c60525 100644 --- a/src/ProjectTemplates/test/ProjectTemplates.Tests.csproj +++ b/src/ProjectTemplates/test/ProjectTemplates.Tests.csproj @@ -27,7 +27,6 @@ <ItemGroup> <EmbeddedResource Include="template-baselines.json" /> <Compile Include="$(SharedSourceRoot)Process\*.cs" LinkBase="shared\Process" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Compile Include="..\Shared\**" LinkBase="Helpers" /> </ItemGroup> diff --git a/src/Security/Authentication/Negotiate/test/Negotiate.FunctionalTest/Microsoft.AspNetCore.Authentication.Negotiate.FunctionalTest.csproj b/src/Security/Authentication/Negotiate/test/Negotiate.FunctionalTest/Microsoft.AspNetCore.Authentication.Negotiate.FunctionalTest.csproj index d0c7e5a9a2f85ac4d10298402855cb0e457bb0cf..085e3ddfd831ed4afa4179a08a7edbde530baeee 100644 --- a/src/Security/Authentication/Negotiate/test/Negotiate.FunctionalTest/Microsoft.AspNetCore.Authentication.Negotiate.FunctionalTest.csproj +++ b/src/Security/Authentication/Negotiate/test/Negotiate.FunctionalTest/Microsoft.AspNetCore.Authentication.Negotiate.FunctionalTest.csproj @@ -5,10 +5,6 @@ <IsWindowsOnlyTest>true</IsWindowsOnlyTest> </PropertyGroup> - <ItemGroup> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> - </ItemGroup> - <ItemGroup> <Reference Include="Microsoft.AspNetCore.Authentication.Negotiate" /> <Reference Include="Microsoft.AspNetCore.Routing" /> diff --git a/src/Security/Authentication/Negotiate/test/Negotiate.Test/Microsoft.AspNetCore.Authentication.Negotiate.Test.csproj b/src/Security/Authentication/Negotiate/test/Negotiate.Test/Microsoft.AspNetCore.Authentication.Negotiate.Test.csproj index 3f483fc9e1d231f1212165f6ac0fa4df13580485..40c623bd117ab3c9c6689486cc296aa263904e16 100644 --- a/src/Security/Authentication/Negotiate/test/Negotiate.Test/Microsoft.AspNetCore.Authentication.Negotiate.Test.csproj +++ b/src/Security/Authentication/Negotiate/test/Negotiate.Test/Microsoft.AspNetCore.Authentication.Negotiate.Test.csproj @@ -4,10 +4,6 @@ <TargetFrameworks>$(DefaultNetCoreTargetFramework)</TargetFrameworks> </PropertyGroup> - <ItemGroup> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> - </ItemGroup> - <ItemGroup> <Reference Include="Microsoft.AspNetCore.Authentication.Negotiate" /> <Reference Include="Microsoft.AspNetCore.Diagnostics" /> diff --git a/src/Security/Authentication/test/Microsoft.AspNetCore.Authentication.Test.csproj b/src/Security/Authentication/test/Microsoft.AspNetCore.Authentication.Test.csproj index 815d30e0277043467372fbb65e7a2f7739a13d2b..738515b336a03c020c4546e61df2ee065048f71d 100644 --- a/src/Security/Authentication/test/Microsoft.AspNetCore.Authentication.Test.csproj +++ b/src/Security/Authentication/test/Microsoft.AspNetCore.Authentication.Test.csproj @@ -33,10 +33,6 @@ <EmbeddedResource Include="OpenIdConnect\wellknownkeys.json" /> </ItemGroup> - <ItemGroup> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> - </ItemGroup> - <ItemGroup> <Reference Include="Microsoft.AspNetCore.Authentication.Certificate" /> <Reference Include="Microsoft.AspNetCore.Authentication.Cookies" /> diff --git a/src/Servers/IIS/IIS/test/FunctionalTest.props b/src/Servers/IIS/IIS/test/FunctionalTest.props index 8582115239262f00d38ffad11b33b263d20ea549..e701d2d02173106b5fa559ea15c67830c736de65 100644 --- a/src/Servers/IIS/IIS/test/FunctionalTest.props +++ b/src/Servers/IIS/IIS/test/FunctionalTest.props @@ -5,8 +5,6 @@ </PropertyGroup> <ItemGroup> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> - <Content Include="..\Common.FunctionalTests\AppHostConfig\*.config" CopyToOutputDirectory="PreserveNewest" /> </ItemGroup> diff --git a/src/Servers/Kestrel/Kestrel/test/Microsoft.AspNetCore.Server.Kestrel.Tests.csproj b/src/Servers/Kestrel/Kestrel/test/Microsoft.AspNetCore.Server.Kestrel.Tests.csproj index abf2fb6b3a1a37e2b1da921d9471710d04a92121..571f2cda30284155b73d051ebc5c950eabe3c01e 100644 --- a/src/Servers/Kestrel/Kestrel/test/Microsoft.AspNetCore.Server.Kestrel.Tests.csproj +++ b/src/Servers/Kestrel/Kestrel/test/Microsoft.AspNetCore.Server.Kestrel.Tests.csproj @@ -7,7 +7,6 @@ <ItemGroup> <Compile Include="$(SharedSourceRoot)NullScope.cs" /> <Compile Include="$(KestrelSharedSourceRoot)test\*.cs" LinkBase="shared" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Content Include="$(KestrelSharedSourceRoot)test\TestCertificates\*.pfx" LinkBase="shared\TestCertificates" CopyToOutputDirectory="PreserveNewest" /> <Content Include="$(KestrelRoot)Core\src\Internal\Http\HttpHeaders.Generated.cs" LinkBase="shared\GeneratedContent" CopyToOutputDirectory="PreserveNewest" /> <Content Include="$(KestrelRoot)Core\src\Internal\Http\HttpProtocol.Generated.cs" LinkBase="shared\GeneratedContent" CopyToOutputDirectory="PreserveNewest" /> diff --git a/src/Servers/Kestrel/test/InMemory.FunctionalTests/InMemory.FunctionalTests.csproj b/src/Servers/Kestrel/test/InMemory.FunctionalTests/InMemory.FunctionalTests.csproj index cad32b08bd5f18ee84d1f98c2bce067e03bbb525..952a0a9f50c98b6e2d807ff95e8ec2b7b774388b 100644 --- a/src/Servers/Kestrel/test/InMemory.FunctionalTests/InMemory.FunctionalTests.csproj +++ b/src/Servers/Kestrel/test/InMemory.FunctionalTests/InMemory.FunctionalTests.csproj @@ -10,7 +10,6 @@ <ItemGroup> <Compile Include="$(SharedSourceRoot)NullScope.cs" /> <Compile Include="$(KestrelSharedSourceRoot)test\*.cs" LinkBase="shared" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Content Include="$(KestrelSharedSourceRoot)test\TestCertificates\*.pfx" LinkBase="shared\TestCertificates" CopyToOutputDirectory="PreserveNewest" /> <Compile Include="$(RepoRoot)src\Shared\Buffers.MemoryPool\*.cs" LinkBase="MemoryPool" /> <Compile Include="$(KestrelSharedSourceRoot)\CorrelationIdGenerator.cs" Link="Internal\CorrelationIdGenerator.cs" /> @@ -27,4 +26,4 @@ <Reference Include="Newtonsoft.Json" /> </ItemGroup> -</Project> \ No newline at end of file +</Project> diff --git a/src/Servers/Kestrel/test/Interop.FunctionalTests/Interop.FunctionalTests.csproj b/src/Servers/Kestrel/test/Interop.FunctionalTests/Interop.FunctionalTests.csproj index 9fff0431d95b7b8afd60a8b68fe467f55ad2dad6..555d7445a5aa627ecc9d9bb03b5de06b7d9bb7a7 100644 --- a/src/Servers/Kestrel/test/Interop.FunctionalTests/Interop.FunctionalTests.csproj +++ b/src/Servers/Kestrel/test/Interop.FunctionalTests/Interop.FunctionalTests.csproj @@ -17,7 +17,6 @@ <Compile Include="$(KestrelSharedSourceRoot)test\TestConstants.cs" LinkBase="shared" /> <Compile Include="$(KestrelSharedSourceRoot)test\TestResources.cs" LinkBase="shared" /> <Content Include="$(KestrelSharedSourceRoot)test\TestCertificates\*.pfx" LinkBase="shared\TestCertificates" CopyToOutputDirectory="PreserveNewest" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> </ItemGroup> <ItemGroup> diff --git a/src/Servers/Kestrel/test/Libuv.BindTests/Libuv.BindTests.csproj b/src/Servers/Kestrel/test/Libuv.BindTests/Libuv.BindTests.csproj index 79752c12b2810b5eed0844d5c140a178c86a637c..29239ef61c2ad0a5ea3a4e221292e0c942a8db2e 100644 --- a/src/Servers/Kestrel/test/Libuv.BindTests/Libuv.BindTests.csproj +++ b/src/Servers/Kestrel/test/Libuv.BindTests/Libuv.BindTests.csproj @@ -5,7 +5,7 @@ <ServerGarbageCollection>true</ServerGarbageCollection> <TestGroupName>Libuv.BindTests</TestGroupName> <!-- https://github.com/dotnet/aspnetcore/issues/22114 --> - <SkipHelixQueues>Windows.10.Arm64;Windows.10.Arm64.Open</SkipHelixQueues> + <SkipHelixQueues>Windows.10.Arm64;Windows.10.Arm64.Open</SkipHelixQueues> </PropertyGroup> <ItemGroup> @@ -14,7 +14,6 @@ </ItemGroup> <ItemGroup> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Compile Include="..\BindTests\**\*.cs" /> <Compile Include="..\Libuv.FunctionalTests\TransportSelector.cs" /> <Compile Include="$(SharedSourceRoot)NullScope.cs" /> diff --git a/src/Servers/Kestrel/test/Libuv.FunctionalTests/Libuv.FunctionalTests.csproj b/src/Servers/Kestrel/test/Libuv.FunctionalTests/Libuv.FunctionalTests.csproj index 65f62bd6412c121b6dd03dfc9449145ab4600d99..3358fcfdb99e1a8bde0e2147f2cdbdbc7c73eacc 100644 --- a/src/Servers/Kestrel/test/Libuv.FunctionalTests/Libuv.FunctionalTests.csproj +++ b/src/Servers/Kestrel/test/Libuv.FunctionalTests/Libuv.FunctionalTests.csproj @@ -14,7 +14,6 @@ <ItemGroup> <Compile Include="..\FunctionalTests\**\*.cs" /> <Compile Include="$(SharedSourceRoot)NullScope.cs" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Compile Include="$(KestrelSharedSourceRoot)test\*.cs" LinkBase="shared" /> <Compile Include="$(KestrelSharedSourceRoot)test\TransportTestHelpers\*.cs" LinkBase="shared\TransportTestHelpers" /> <Content Include="$(KestrelSharedSourceRoot)test\TestCertificates\*.pfx" LinkBase="shared\TestCertificates" CopyToOutputDirectory="PreserveNewest" /> diff --git a/src/Servers/Kestrel/test/Sockets.BindTests/Sockets.BindTests.csproj b/src/Servers/Kestrel/test/Sockets.BindTests/Sockets.BindTests.csproj index 3ad3eb297afdfeffaa73497b0f93e334f9ac929b..0f7576c452d98d70c2d53a4a203010682b6fc558 100644 --- a/src/Servers/Kestrel/test/Sockets.BindTests/Sockets.BindTests.csproj +++ b/src/Servers/Kestrel/test/Sockets.BindTests/Sockets.BindTests.csproj @@ -7,7 +7,6 @@ </PropertyGroup> <ItemGroup> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Compile Include="..\BindTests\**\*.cs" /> <Compile Include="..\Sockets.FunctionalTests\TransportSelector.cs" /> <Compile Include="$(SharedSourceRoot)NullScope.cs" /> diff --git a/src/Servers/Kestrel/test/Sockets.FunctionalTests/Sockets.FunctionalTests.csproj b/src/Servers/Kestrel/test/Sockets.FunctionalTests/Sockets.FunctionalTests.csproj index 4bc4f8fbdf9b41ee6a371b28b751dd04e4b64c1e..325b301e0bbfc2fc6337bae9dc37e64810abbfe3 100644 --- a/src/Servers/Kestrel/test/Sockets.FunctionalTests/Sockets.FunctionalTests.csproj +++ b/src/Servers/Kestrel/test/Sockets.FunctionalTests/Sockets.FunctionalTests.csproj @@ -10,7 +10,6 @@ <ItemGroup> <Compile Include="..\FunctionalTests\**\*.cs" /> <Compile Include="$(SharedSourceRoot)NullScope.cs" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Compile Include="$(KestrelSharedSourceRoot)test\*.cs" LinkBase="shared" /> <Compile Include="$(KestrelSharedSourceRoot)test\TransportTestHelpers\*.cs" LinkBase="shared\TransportTestHelpers" /> <Content Include="$(KestrelSharedSourceRoot)test\TestCertificates\*.pfx" LinkBase="shared\TestCertificates" CopyToOutputDirectory="PreserveNewest" /> diff --git a/src/Shared/test/SkipOnHelixAttribute.cs b/src/Shared/test/SkipOnHelixAttribute.cs deleted file mode 100644 index e1a74467bc70245ec9cca90eafea82d48cb5a528..0000000000000000000000000000000000000000 --- a/src/Shared/test/SkipOnHelixAttribute.cs +++ /dev/null @@ -1,50 +0,0 @@ -// Copyright (c) .NET Foundation. All rights reserved. -// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information. - -using System; -using System.Linq; - -namespace Microsoft.AspNetCore.Testing.xunit -{ - /// <summary> - /// Skip test if running on helix (or a particular helix queue). - /// </summary> - [AttributeUsage(AttributeTargets.Method | AttributeTargets.Class, AllowMultiple = false)] - public class SkipOnHelixAttribute : Attribute, ITestCondition - { - public SkipOnHelixAttribute(string issueUrl) - { - if (string.IsNullOrEmpty(issueUrl)) - { - throw new ArgumentException(); - } - IssueUrl = issueUrl; - } - - public string IssueUrl { get; } - - public bool IsMet - { - get - { - var skip = OnHelix() && (Queues == null || Queues.ToLowerInvariant().Split(";").Contains(GetTargetHelixQueue().ToLowerInvariant())); - return !skip; - } - } - - // Queues that should be skipped on, i.e. "Windows.10.Amd64.ClientRS4.VS2017.Open;OSX.1012.Amd64.Open" - public string Queues { get; set; } - - public string SkipReason - { - get - { - return $"This test is skipped on helix"; - } - } - - public static bool OnHelix() => !string.IsNullOrEmpty(GetTargetHelixQueue()); - - public static string GetTargetHelixQueue() => Environment.GetEnvironmentVariable("helix"); - } -} diff --git a/src/Shared/test/SuccessfulTests.cs b/src/Shared/test/SuccessfulTests.cs new file mode 100644 index 0000000000000000000000000000000000000000..9843582b171c8409556c07930877586fdc6be302 --- /dev/null +++ b/src/Shared/test/SuccessfulTests.cs @@ -0,0 +1,39 @@ +// Copyright (c) .NET Foundation. All rights reserved. +// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information. + +using Microsoft.AspNetCore.Testing; +using Xunit; + +namespace AlwaysTestTests +{ + /// <summary> + /// Tests for every test assembly to ensure quarantined and unquarantined runs report at least one test execution. + /// </summary> + public class SuccessfulTests + { + /// <summary> + /// Test that executes in quarantined runs and always succeeds. + /// </summary> + [Fact] + [QuarantinedTest] + public void GuaranteedQuarantinedTest() + { + } + + /// <summary> + /// Test that executes in unquarantined runs and always succeeds. + /// </summary> + /// <remarks> + /// <see cref="TraitAttribute"/> applied to ensure test runs even if assembly is quarantined overall. + /// <c>dotnet test --filter</c>, <c>dotnet vstest --testcasefilter</c>, and + /// <c>vstest.console.exe --testcasefilter</c> handle the "no Quarantined=true trait OR Quarantined=false" case + /// e.g. <c>"Quarantined!=true|Quarantined=false</c>. But, <c>xunit.console.exe</c> doesn't have a syntax to + /// make it work (yet?). + /// </remarks> + [Fact] + [Trait("Quarantined", "false")] + public void GuaranteedUnquarantinedTest() + { + } + } +} diff --git a/src/Tools/FirstRunCertGenerator/test/Microsoft.AspNetCore.DeveloperCertificates.XPlat.Tests.csproj b/src/Tools/FirstRunCertGenerator/test/Microsoft.AspNetCore.DeveloperCertificates.XPlat.Tests.csproj index dd470dbf374ab8a9e7cf7dff16bc3ad2b1c1e7e1..c276d7d02086ddf250ef0c9d777a41ca59ec018b 100644 --- a/src/Tools/FirstRunCertGenerator/test/Microsoft.AspNetCore.DeveloperCertificates.XPlat.Tests.csproj +++ b/src/Tools/FirstRunCertGenerator/test/Microsoft.AspNetCore.DeveloperCertificates.XPlat.Tests.csproj @@ -6,7 +6,6 @@ <ItemGroup> <Reference Include="Microsoft.AspNetCore.DeveloperCertificates.XPlat" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> </ItemGroup> </Project> diff --git a/src/Tools/Microsoft.dotnet-openapi/test/dotnet-microsoft.openapi.Tests.csproj b/src/Tools/Microsoft.dotnet-openapi/test/dotnet-microsoft.openapi.Tests.csproj index 288c651ed309bb481533a89642a1e6c5d7f7f27c..1d216328e8ccbb62eafbadf4d1ea472e0321b005 100644 --- a/src/Tools/Microsoft.dotnet-openapi/test/dotnet-microsoft.openapi.Tests.csproj +++ b/src/Tools/Microsoft.dotnet-openapi/test/dotnet-microsoft.openapi.Tests.csproj @@ -12,7 +12,6 @@ <ItemGroup> <Compile Include="$(ToolSharedSourceRoot)TestHelpers\**\*.cs" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Content Include="TestContent\*" LinkBase="TestContent\" CopyToOutputDirectory="PreserveNewest" /> <Compile Include="$(SharedSourceRoot)Process\ProcessExtensions.cs" /> </ItemGroup> diff --git a/src/Tools/dotnet-watch/test/dotnet-watch.Tests.csproj b/src/Tools/dotnet-watch/test/dotnet-watch.Tests.csproj index 219f55d6f42f4de92fb1c49b5688f1f79d98db6c..71ebe061da7640db7e85f6a4d83defb672d80bb7 100644 --- a/src/Tools/dotnet-watch/test/dotnet-watch.Tests.csproj +++ b/src/Tools/dotnet-watch/test/dotnet-watch.Tests.csproj @@ -9,7 +9,6 @@ <ItemGroup> <Compile Include="$(ToolSharedSourceRoot)TestHelpers\**\*.cs" /> - <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" /> <Content Include="TestProjects\**\*" CopyToOutputDirectory="PreserveNewest" /> </ItemGroup>