Add ability to warn about pages existing in the docs directory but not included in the "nav" configuration

[tjanez]

Use case

A team uses a CI system and would like to ensure documentation quality never regresses.
To this end, the CI job has a step that runs mkdocs build --strict which reports a failure if building the documentation results in any warnings or errors.

However, it can still happen that someone will write new documentation but forget to include it in the nav configuration. And MkDocs will detect this and report it. Here is an example output:

INFO    -  The following pages exist in the docs directory, but are not included in the "nav" configuration:
  - README.md
  - foo.md

Feature request

Would it be possible to configure this INFO message to be a WARNING?

As a bonus, would it be possible to configure which files in the docs directory should be ignored by MkDocs completely?

[waylan]

So, this used to be a warning, but was changed to an info message because we got complaints. I initially made it a warning for this very use case. Unfortunately, for users who want to have pages not included in the nav, this would mean they can't use --strict.

We could create a new setting, but we generally try to reduce the number of settings, not add more. From one viewpoint, the setting already exists in --strict, but as it turns out that is not granular enough to meet everyone's needs.

For reference, #1604 seems related in part.

As a workaround, is there any reason why you can't let MkDocs auto-generate the nav? That would ensure all pages are already included in the nav. There are other ways to define page titles. Or do you need to define a custom sort order (which can only be done in the nav config setting)?

[facelessuser]

Could you do something like an even stronger strict mode if you have multiple strict flags? Much like how you can sometimes raise verbosity in some applications with multiple verbose flags -vv? I think mkdocs uses click, but I haven't checked if it supports such functionality, but I would hope it would as you can do this with argparse.

[waylan]

I considered that briefly, although I haven't given it much thought. And yes click supports it, IIRC. Perhaps something like, --strict: fail on warning and --strict --strict: fail on info? But there are some info messages that should never cause a failure. How do we differentiate between them? Or should be set a policy that those messages should be debug messages?

And how do we handle the matching config setting? Using strict: 1 for --strict and strict: 2 for --strict --strict? Or do we match the log levels with a string: strict: warn and strict: info?

[facelessuser]

I personally don't think it needs to be tied to all INFO or anything like that as some INFO is just INFO, but yeah, it could elevate some INFO to WARN level. For this reason, I don't think strict: warn and strict: info makes sense, as a user may expect that all INFO should be elevated to WARN, which doesn't make sense.

If you wanted some distinction, than you could tag some info with DEBUG instead of INFO if later it could be elevated to warn. I'm not sure how many levels of strict will be needed in the future (hopefully only two), but if one day you need a third level, you probably wouldn't want to tie strict to anything like strict: debug as you may only want to elevate certain items in a certain level. I'd probably just stick with something like strict: 1 or strict: 2 for --strict and --strict --strict respectively. That's just me though. If you wanted to communicate debug statements as part of some "strictness" level, you could include that in the debug output I guess.

[waylan]

It occurs to me that you are assuming that we do the following at each location where a warning is raised:

if config['strict']:
    raise error(msg)
else:
    log.warning(msg)

But that is not what we do. We simply raise a warning. We then use a logging.filter to count all warnings and then, as the final step of build, we raise an error if count > 0. Using the same method, escalating any info messages, would escalate all info messages.

However, I suppose we could do something like the following at select locations in the code:

if config['strict'] >= 2:
    log.warning(msg)
else:
    log.info(msg)

I just think that would be weird. Why are only some info messages escalated? Yet, there are some info messages that don't make sense as debug messages, but should never raise an error. Personally, I'm uncomfortable with the inconsistency.

[facelessuser]

If inconsistency is a problem, you would have to have more granularity in how you log messages or group things in a way that makes more sense. I agree that raising all info doesn't make sense. I would argue that info should always just be info and not get raised at all. Which means you may need to place messages that can be raised in a category that makes sense.

Of course this is only a suggestion, and if it doesn't make sense due to how this project is architected, then that is okay too. I was mainly just trying to suggest a way to avoid a new config variable by extended the existing functionality in a way that might make sense for the project, but if it doesn't, then maybe some other option is more suitable.

[waylan]

Well, we are just using the logging lib's logging levels, which are just integers. The lib fully supports defining and using your own levels. We could define a level between warning (30) and info (20) with a value of 25. Not sure what to call it though.

[facelessuser]

warn-level-2 🤷‍♂️ . Not sure what would be best. I'm not sure if I have a suggestion that doesn't sound a little awkward, but maybe a little awkward is okay?

[tjanez]

@waylan, @facelessuser, thanks for your quick response and such fruitful discussion!

As a workaround, is there any reason why you can't let MkDocs auto-generate the nav? That would ensure all pages are already included in the nav. There are other ways to define page titles. Or do you need to define a custom sort order (which can only be done in the nav config setting)?

The problem is we have Markdown files pretty much unordered in a directory, i.e. we don't prefix them with numbers or similar, and use the nav option to bring them in order.

Regarding the proposed solutions to tackle this use case, I have a few thoughts.

Are there other info messages that one might consider a warning?
If not, would it not be simpler to just create a config option for this use case?

If yes, then perhaps the introduction of another logging level in between info and warning seems a sane approach. I would call it something like warning-soft since warn-level-2 seems to imply this is something with a higher level than warning 🙂.

Then strict could have two options, warning or warning-soft depending on whether one wants to treat warning messages as errors or warning-soft messages as errors.

Note that changing the strict setting to have this semantics implies that the user has some knowledge of the ordering/severity of logging messages. Is that ok?
In any case, we could probably solve it elegantly by explaining it in the documentation:
for example, Setting string to warning-soft would treat all logging messages of severity warning-soft or above (i.e. warning, error) as error and return a non-zero exit code.

[waylan]

warn-level-2 seems to imply this is something with a higher level than warning

I agree. warn-soft is an interesting suggestion. Personally, I was thinking something like semi-warning, which is awful IMO. I'm certainly open to other suggestions in this regard.

[trel]

Generally, introducing an esoteric logging level seems... off.

That said... consider 'notice' to be between 'info' and 'warning'?

https://en.wikipedia.org/wiki/Syslog#Severity_level

[froschdesign]

We could define a level between warning (30) and info (20) with a value of 25.

A log level between "warning" and "info" is "notice". See RFC 3164 - The BSD syslog Protocol.

---

@trel was a bit faster.

[waylan]

Excellent! notice it is--if we go that route.

[ceteri]

What is the approach for a plugin that generates a Markdown file, which is also referenced in the "nav" section of mkdocs.yml ? Because I keep seeing this warning:

The following pages exist in the docs directory, but are not included in the "nav" configuration:

even though the events are ostensibly following correct means of adding the associated File

[waylan]

@Cetra, please start a new discussion and ask your question there. While related to this issue, discussing your question will add a lot of noise to the discussion of the issue.

Also, please provide some information about how you are generating Markdown files. Which events are you using in your plugin?

[annabaker]

Any update on this? Similarly, would love this feature in place for CI purposes.

[karl-johan-grahn]

I was also using strict under the assumption that it would error out in this situation, and would prefer if you reverted it to warning instead of info.

[pawamoy]

The redirect plugin creates pages that are not included in the nav. Setting the level back to warning would prevent every user of the redirect plugin to use strict 😕 That's probably not the only plugin doing that, and there are probably other reasons why a page could exist while not being referenced in the nav. So this feature definitely needs a way to skip checking specific files.

[oprypin]

@pawamoy This particular aspect I think is actually not a concern at all. It's only about one-off setups that may exist. I.e. "I just want MkDocs to copy this file, why do I have to put it in nav"

The redirect plugin creates pages that are not included in the nav.

It doesn't actually create pages, only files in the last stage of the build, so there's no warnings associated with it.
There's also a way for plugins to register a particular page to be excluded from warnings
https://github.com/oprypin/mkdocs-literate-nav/blob/0825f8a683a27c13e738b6d7dd2d7b6c962d2a48/mkdocs_literate_nav/plugin.py#L90-L93

[pawamoy]

Ah, true, my bad, I had the log because I was using a custom script and not mkdocs-redirects! Sorry! And thanks for the link, will check this out

[ssbarnea]

I would be quite curious about how to trick mkdocs to not complain about "The following pages exist in the docs directory" for stuff that I need it there but I do not want to include it inside the navigation (they are included from other files).

[oprypin]

• @ssbarnea Disable navigation warning #1888 (comment)

[oprypin]

I would like to make many of the messages have configurable logging levels among warn, info, ignore (DEBUG).

Here's a survey of relevant logging messages and situations.

File

• File exists in docs, missing from nav.
  Defaults to info, in the future should default to warn.
  Can also be granularly configured through not_in_nav.
  

  mkdocs/mkdocs/structure/nav.py

  Lines 151 to 153 in 7b1bc92

  	log.info(
  	'The following pages exist in the docs directory, but are not '
  	'included in the "nav" configuration:\n - ' + '\n - '.join(missing_from_config)

• (serve) File built but will be excluded.
  Maybe can be left non-configurable as info.
  Caused by granular configuration of exclude_docs.
  

  mkdocs/mkdocs/commands/build.py

  Lines 318 to 321 in 7b1bc92

  	log.info(
  	"The following pages are being built only for the preview "
  	"but will be excluded from `mkdocs build` per `exclude_docs`:\n - "
  	+ "\n - ".join(excluded)

Nav

• Relative link to nonexistent doc.
  Defaults to warn.
  

  mkdocs/mkdocs/structure/nav.py

  Lines 167 to 170 in 7b1bc92

  	log.warning(
  	f"A relative path to '{link.url}' is included in the 'nav' "
  	"configuration, which is not found in the documentation files."
  	)

  • (serve) Relative link to excluded doc.
    Logging level is based on the above but capped to info
    

    mkdocs/mkdocs/structure/nav.py

    Lines 193 to 195 in 7b1bc92

    	log.info(
    	f"A relative path to '{file.src_path}' is included in the 'nav' "
    	"configuration, but this file is excluded from the built site."

• / absolute link.
  Defaults to ignore. In the future maybe should receive an option beyond logging - turn it into site-relative and apply the same checking as for relative links above.
  

  mkdocs/mkdocs/structure/nav.py

  Lines 162 to 164 in 7b1bc92

  	log.debug(
  	f"An absolute path to '{link.url}' is included in the 'nav' "
  	"configuration, which presumably points to an external resource."

• http:// absolute link.
  Just ignore. Not sure if this should be made configurable. Maybe check that it doesn't start with site_url, but how to express this?
  

  mkdocs/mkdocs/structure/nav.py

  Line 160 in 7b1bc92

  log.debug(f"An external link to '{link.url}' is included in the 'nav' configuration.")

Markdown

• Relative link to nonexistent doc
  Defaults to warn.
  

  mkdocs/mkdocs/structure/pages.py

  Lines 332 to 334 in 7b1bc92

  	log.warning(
  	f"Documentation file '{self.file.src_uri}' contains a link to "
  	f"'{target_uri}' which is not found in the documentation files."

  • (serve) Relative link to excluded doc.
    Logging level is based on the above but capped to info.
    

    mkdocs/mkdocs/structure/pages.py

    Lines 338 to 340 in 7b1bc92

    	log.info(
    	f"Documentation file '{self.file.src_uri}' contains a link to "
    	f"'{target_uri}' which is excluded from the built site."

• Absolute links (both kinds).
  Currently a message of any kind is missing, should get a configuration that parallels the nav configuration described above and should default to ignore.
  

  mkdocs/mkdocs/structure/pages.py

  Line 320 in 7b1bc92

  # Ignore URLs unless they are a relative link to a source file.

[majkinetor]

@oprypin , can this be overrided on page level ?

I want to exclude specific pages from warnings, but I don't want to turn off warning in general, as it is useful to me. Nevertheless, for the pages I acknowledged that are correct, I don't want warning anymore.

[oprypin]

@majkinetor Yes that is available! https://github.com/mkdocs/mkdocs/releases/tag/1.5.0 added exactly such a feature not_in_nav