Remove some low hanging allocations (#35397)

* [x] In the ordinary case, diagnostics aren't produced by builders. The current pattern results in allocating a HashSet
and an empty array for this case. Removing some of these allocations
* [x] ViewComponentTagHelperPass does not need to operate on .razor files, but does some unnecessary work. We can avoid it