[BUG] Text.append_tokens does not strip control codes

[hamdanal]

• I've checked docs and closed issues for possible solutions.
• I can't find my issue in the FAQ.

Describe the bug

A string containing control codepoints passed to Text.append_tokens sometimes crashes with IndexError when printed

Related issues: #2284 and #2963

Minimal code example that demonstrates the issue:

>>> from rich import get_console
>>> from rich.text import Text
>>> t = Text().append_tokens([("long text that will be wrapped with a control code \r\n.", "red")])
>>> get_console().print(t, width=40)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/.venv/lib/python3.10/site-packages/rich/console.py", line 1699, in print
    extend(render(renderable, render_options))
  File "/tmp/.venv/lib/python3.10/site-packages/rich/console.py", line 1331, in render
    for render_output in iter_render:
  File "/tmp/.venv/lib/python3.10/site-packages/rich/text.py", line 654, in __rich_console__
    lines = self.wrap(
  File "/tmp/.venv/lib/python3.10/site-packages/rich/text.py", line 1184, in wrap
    new_lines = line.divide(offsets)
  File "/tmp/.venv/lib/python3.10/site-packages/rich/text.py", line 1113, in divide
    line_start, line_end = line_ranges[end_line_no]
IndexError: list index out of range

Platform

Click to expand

$ python -m rich.diagnose
╭───────────────────────── <class 'rich.console.Console'> ─────────────────────────╮
│ A high level console interface.                                                  │
│                                                                                  │
│ ╭──────────────────────────────────────────────────────────────────────────────╮ │
│ │ <console width=102 ColorSystem.EIGHT_BIT>                                    │ │
│ ╰──────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                  │
│     color_system = '256'                                                         │
│         encoding = 'utf-8'                                                       │
│             file = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'> │
│           height = 51                                                            │
│    is_alt_screen = False                                                         │
│ is_dumb_terminal = False                                                         │
│   is_interactive = True                                                          │
│       is_jupyter = False                                                         │
│      is_terminal = True                                                          │
│   legacy_windows = False                                                         │
│         no_color = False                                                         │
│          options = ConsoleOptions(                                               │
│                        size=ConsoleDimensions(width=102, height=51),             │
│                        legacy_windows=False,                                     │
│                        min_width=1,                                              │
│                        max_width=102,                                            │
│                        is_terminal=True,                                         │
│                        encoding='utf-8',                                         │
│                        max_height=51,                                            │
│                        justify=None,                                             │
│                        overflow=None,                                            │
│                        no_wrap=False,                                            │
│                        highlight=None,                                           │
│                        markup=None,                                              │
│                        height=None                                               │
│                    )                                                             │
│            quiet = False                                                         │
│           record = False                                                         │
│         safe_box = True                                                          │
│             size = ConsoleDimensions(width=102, height=51)                       │
│        soft_wrap = False                                                         │
│           stderr = False                                                         │
│            style = None                                                          │
│         tab_size = 8                                                             │
│            width = 102                                                           │
╰──────────────────────────────────────────────────────────────────────────────────╯
╭─── <class 'rich._windows.WindowsConsoleFeatures'> ────╮
│ Windows features available.                           │
│                                                       │
│ ╭───────────────────────────────────────────────────╮ │
│ │ WindowsConsoleFeatures(vt=False, truecolor=False) │ │
│ ╰───────────────────────────────────────────────────╯ │
│                                                       │
│ truecolor = False                                     │
│        vt = False                                     │
╰───────────────────────────────────────────────────────╯
╭────── Environment Variables ───────╮
│ {                                  │
│     'TERM': 'xterm-256color',      │
│     'COLORTERM': None,             │
│     'CLICOLOR': None,              │
│     'NO_COLOR': None,              │
│     'TERM_PROGRAM': None,          │
│     'COLUMNS': None,               │
│     'LINES': None,                 │
│     'JUPYTER_COLUMNS': None,       │
│     'JUPYTER_LINES': None,         │
│     'JPY_PARENT_PID': None,        │
│     'VSCODE_VERBOSE_LOGGING': None │
│ }                                  │
╰────────────────────────────────────╯
platform="Linux"

$ pip freeze | grep rich
rich==13.4.2

[github-actions]

Thank you for your issue. Give us a little time to review it.

PS. You might want to check the FAQ if you haven't done so already.

This is an automated reply, generated by FAQtory