Add and require version_added lexer attribute

[jeanas]

Next step for #2585 :)

• MXML: Remove bogus mimetypes line

• Savi: Fix indentation

• Move versionadded data to a lexer attribute

  That way, we can set it to "" for old lexers, and check that it's
  present on new lexers. (In the future, we might also use it for better
  presentation in the documentation.)

• Make documentation use the new version_added lexer attribute

• Require the version_added lexer attribute

• Harmonize version_added format

[jeanas]

Today I learnt that GitHub doesn't test the PR branch but the result of merging that branch with master... I'll fix the CI failures.

[jeanas]

Done.

For the record, here's the script I've used for the third commit:

import importlib
import inspect
from pathlib import Path
import re

from pygments.lexers import LEXERS

for class_name, (module_name, _, _, _, _) in LEXERS.items():
    mod = importlib.import_module(module_name)
    mod = importlib.reload(mod)
    cls = getattr(mod, class_name)

    file = inspect.getsourcefile(cls)
    lines, first_line_index = inspect.getsourcelines(cls)
    first_line_index -= 1 # make it zero-based
    end_line_index = first_line_index + len(lines)

    docstring_finished = False
    for version_added_index, version_added_line in enumerate(lines):
        if m := re.match(r"    .. versionadded:: (.*)", version_added_line):
            version_added = m.group(1)
            cursor = version_added_index
            del lines[version_added_index]
            cursor -= 1
            if not lines[version_added_index-1].strip():
                del lines[version_added_index-1]
                cursor -= 1
            break
    else:
        version_added = ''
        assert lines[1].startswith('    """'), f"{file}:{first_line_index}"
        if lines[1].endswith('"""\n'):
            docstring_finished = True
        cursor = 2

    if not docstring_finished:
        for i, line in enumerate(lines[cursor:], cursor):
            if line == '    """\n':
                cursor = i+1
                break
        else:
            assert False, f"{file}:{first_line_index}"

    unlocked = False
    for insertion_index, line in enumerate(lines[cursor:], cursor):
        if re.match("    (name|url|aliases|filenames|mimetypes) ?=", line):
            unlocked = True
        elif unlocked and not line.startswith("     ") and not re.match(r"    (\[|\])", line):
            break
    else:
        insertion_index += 1

    new_version_added_line = f"    version_added = {version_added!r}\n"
    lines.insert(insertion_index, new_version_added_line)

    all_lines = Path(file).read_text(encoding="utf-8").splitlines(keepends=True)
    all_lines[first_line_index:end_line_index] = lines
    Path(file).write_text(''.join(all_lines), encoding="utf-8")

[birkenfeld]

Very nice!

[jeanas]

Should we merge this one? @Anteru What do you think about it?

[jeanas]

(Updated again for new lexers in master)

[Anteru]

Looks good to me, thanks! I assume finding the ones which are set to "" at the moment is a question of Git history analysis? After all, they must have been added at some point (not that I think it's worth fixing for now.)

[jeanas]

Yes, that's it. It's not always easy to find the version for old lexers; some of them have been moved around, and there are also some quirks in the Git history like commit db1e5ef (I think Pygments was originally using Mercurial and migrated to Git at some point?).