Cannot return a 404 from getInitialProps anymore by throwing an error with code='ENOENT'

[fabb]

Bug report

Describe the bug

Until next.js 9.2.1 I could return a 404 for a page when throwing an error with code = 'ENOENT' in getInitialProps. That no longer works in 9.3.2, now a 500 is returned. This is bad for SEO, we need to be able to throw 404s.

To Reproduce

1. In getInitialProps, throw an error like this one:

export class Custom404Error extends Error {
    code: string

    constructor(message?: string) {
        super(message)
        this.code = 'ENOENT' // triggers a 404 in next.js
    }
}

2. Open that page in the browser

Expected behavior

Opening the page in the browser should return status code 404.

System information

• OS: macOS Catalina
• Browser (if applies) [e.g. chrome, safari]
• Version of Next.js: 9.3.2

Additional context

If there is another recommended way to return 404s from getInitialProps, I'd be happy about pointers.
Could the regression (or change of behavior, since ENOENT was kind of an internal behavior) be caused by #10572?

[fabb]

It seems to be related to this code having been removed in 9.3:
https://github.com/zeit/next.js/blob/v9.2.1/packages/next/next-server/server/next-server.ts#L1044

@devknoll It seems that you removed this in your PR #10476, was this by accident, or is there now another recommended way to return statusCode 404 from getInitialProps?

[fabb]

I reverted the problematic change from #10476 in a new PR: #11480.

[timneutkens]

Note that throwing a ENOENT error was never supported. You should render the 404 page as a state of your UI with the correct robots tag. eg like: #10960 (comment)

[fabb]

@timneutkens This would cause the page to be rendered with a 200, I would need to research if 200 + noindex would be equally good for SEO as 404. I doubt it.

[timneutkens]

This would cause the page to be rendered with a 200, I would need to research if 200 + noindex would be equally good for SEO as 404. I doubt it.

It's the same thing, so yes.

[andreas1327250]

You could still accept the PR, set the "feature" to deprecated and remove it with the next major version, so that we are able use 9.3 in the meanwhile.

[timneutkens]

I'm not saying the PR won't be merged, I'm saying you should never rely on throwing errors around. Especially when it's not documented that it's the correct way to do what you want to achieve.

[fabb]

According to our SEO consultant, 200 + noindex will be seen as "soft 404" by Google which is worse than a real 404 page. Also we couldn't see these 404s anymore in our logs since they have a different status code.

[AVGP]

@fabb A soft 404 isn't "worse", but it could leak into search result pages...which is prevented by setting the page to "noindex". The alternative way to deal with this would be to redirect to a page that is configured to return a 404 status code. Our guide has a section on soft 404

[timneutkens]

Just FYI, asked Martin to weight in on this one, he's a developer advocate at Google.

[fabb]

Thanks for the extra effort, much appreciated.

While I‘m not happy that 404s are no longer possible, and some effort is required to change the implementation and monitoring, I understand that ENOENT is an internal implementation detail which cannot be relied on.

[fabb]

@timneutkens So is it final that #11480 won't get merged, or is there still a chance to defer the "breaking" ENOENT change to next.js v10 as @andreas1327250 suggested? We're right at releasing our first SSG page, so we need to decide whether we need to refactor our dynamic pages that could be 404s now or can defer it to later.

[timneutkens]

So is it final that #11480 won't get merged

I haven't said that, was just pointing out that you're relying on undocumented implementation details of Next.js.

We still need to review the PR.

[estevecastells]

From an SEO perspective, this can be a huge issue. I hope you reconsider having the ability to show proper 404's as it could lead to massive problems in all websites based on Next.js. Having no way of seeing properly where we are throwing 404 is a complete lack of visibility.

I don't think anyone can believe what Martin is saying here, sorry. Please, be transparent with the community with real impact on topics, being polite doesn't help.

[timneutkens]

I hope you reconsider having the ability to show proper 404's

This is already possible, you can add res.status = 404 in getServerSideProps.

I've mentioned multiple times in this thread that the way that you're triggering the 404 page through ENOENT is wrong and that if you want to use a render-based approach you have to do it in the way explained, as React runs in 1 pass and you can't just pass along res.

You have ways to see where 404 is shown even with the render based approach, you can easily log out the routes that are rendered in that way.

TLDR:
The code being used (throwing ENOENT) is wrong and shouldn't be written in that way as it relies on internals, there's many alternatives.

[fabb]

Ok, so could I also use res.status = 404 in getInitialProps? (+ render the visual 404 component as part of the page with some helper props on the page).

I‘m totally for using public api where possible, I just thought there was no other way when I implemented it using ENOENT (which was recommended in some other issue by a user).

[AVGP]

@estevecastells You are aware that what I'm saying is based on what actually happens in Googlebot, right? It's not "politeness" that we have this in our official guidance. Having a method to give the proper HTTP status is great but in situations where it isn't feasible we deal with that. Don't create FUD.

As @timneutkens said: There are ways to generate proper 404s. This issue is about a specific situation where that isn't supported apparently and for that, we have other ways to deal with it in Googlebot.

[fabb]

Ok, setting res.status = 404 in getInitialProps works fine, thank you!

(I only have a small issue that I need to hide an ad position when the 404 is shown, but the ad position is defined in the global layout based on router path. The layout is only overridable statically for each page, so I'm not yet sure how to solve this.)

[annewanghy]

setting res.status = 404 in getInitialProps works fine

hi @fabb, How does it work in your site? I tried set res.status = 404 in home.js getInitialProps, and then return , but it doesn't redirect to _error.js, neither to 404.js.

Oh, it shouldn't render error page anymore, cause it doesn't an error page, need to use status === 404? <ErrorPage />: <Home>, right?

so could I also use res.status = 404 in getInitialProps? (+ render the visual 404 component as part of the page with some helper props on the page).

Find it, should use UI to render a 404 component

[balazsorban44]

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.