Error when server restart within middleware mode

[rainbowatcher]

Describe the bug

I'm using the middleware mode of Vite, and upon updating vite.config.js, the development server commenced restarting. After it launch, the browser console began throwing errors,

while in Vite's logs there was also a message of the port being occupied.

When I switched to vite@4.1.4, this issue was not present.

the port error is raise on packages/vite/src/node/server/index.ts:840
when the server.close() called.
related PR: #12379

Reproduction

https://github.com/rainbowatcher/vite-middleware-mode-error-eproduction.git

Steps to reproduce

1. Clone the reproduction repo via git clone https://github.com/rainbowatcher/vite-middleware-mode-error-eproduction.git
2. Install dependencies via pnpm i
3. Start the server via node src/server/server.js
4. Open the server page in browser
5. Change the vite.config.js and save, then wait the dev server restart.

System Info

System:
  OS: macOS 13.3
  CPU: (8) x64 Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
  Memory: 276.07 MB / 16.00 GB
  Shell: 5.9 - /bin/zsh
Binaries:
  Node: 18.15.0 - ~/Library/Caches/fnm_multishells/81355_1680497113342/bin/node
  Yarn: 1.22.19 - ~/Library/Caches/fnm_multishells/81355_1680497113342/bin/yarn
  npm: 9.5.0 - ~/Library/Caches/fnm_multishells/81355_1680497113342/bin/npm
Browsers:
  Safari: 16.4
npmPackages:
  @vitejs/plugin-vue: ^4.1.0 => 4.1.0 
  vite: ^4.2.1 => 4.2.1

Used Package Manager

pnpm

Logs

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.