Fix static generation and crawler requests

[shuding]

Right now the SSG condition is determined only based on supportsDynamicHTML. However that supportsDynamicHTML flag can be affected by bots too.

Here we exclude the isBot condition from the SSG flag, and set proper status and meta tags for static HTML requests (and/or 404 cases).

Bug

• Related issues linked using fixes #number
• Integration tests added
• Errors have a helpful link attached, see contributing.md

Feature

• Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
• Related issues linked using fixes #number
• Integration tests added
• Documentation added
• Telemetry added. In case of a feature if it's used or not.
• Errors have a helpful link attached, see contributing.md

Documentation / Examples

• Make sure the linting passes by running pnpm lint
• The "examples guidelines" are followed from our contributing doc

[ijjk]

Failing test suites

Commit: 6067bd6

pnpm testheadless test/development/acceptance-app/ReactRefreshLogBox.test.ts

• ReactRefreshLogBox app > logbox: can recover from a syntax error without losing state

Expand output

● ReactRefreshLogBox app › logbox: can recover from a syntax error without losing state

TIMED OUT: /Count: 1/

Count: 0

  519 |
  520 |   if (hardError) {
> 521 |     throw new Error('TIMED OUT: ' + regex + '\n\n' + content)
      |           ^
  522 |   }
  523 |   return false
  524 | }

  at check (lib/next-test-utils.js:521:11)
  at Object.<anonymous> (development/acceptance-app/ReactRefreshLogBox.test.ts:107:5)

Read more about building and testing Next.js in contributing.md.