Audit E2E Suite for DaemonSet Couplings

[jmdeal]

The addition of several DaemonSets to our E2E clusters by #5982 caused failures across the E2E suite. These failures can be categorized into two buckets:

1. Tests which have a deep coupling between the size of nodes and what pods will be able to schedule on those nodes (example).
2. Tests which rely on the number of DaemonSets or the DaemonSet overhead (example). These tests were broke by the addition of GPU specific DaemonSets (see #715).

Currently, whenever a new DaemonSet is added to our E2E clusters there is a risk that tests will begin to fail for one of these reasons. We should audit our E2E suite and, when possible, eliminate these couplings. For tests where these couplings are required, we should evaluate if they need to be E2E tests or can be sufficiently covered by functional tests.