Update all reconciles to fail if the health checker fails to start

[BlaineEXE]

We should investigate taking the same approach from #8708 for all health checkers:

• ceph status
  • special considerations:
    • We should not create a condition where Rook will not reconcile OSDs if the Ceph status is not HEALTH_OK because reconciling the OSDs may fix the health status in some cases
• osd status
• pool mirror
• fs mirror

The CephObjectStore reconciler will fail the reconcile if the health checker fails to start. This is imperative for the object store because we use an HTTP API for the health check which can fail.

For other statuses, if the status check can fail, we should update the reconcile to ensure we only report a successful reconcile once we know we can start getting status reports.

See original discussion: #8708 (comment)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.