Reports that contain unicode will cause Bandit to crash

[Brcrwilliams]

Describe the bug
If bandit output contains unicode characters, and the user attempts to create a report file, then Bandit will crash.

To Reproduce
Steps to reproduce the behavior:

1. Create the following python file, test.py:

secret = u'Don\'t👏hard👏code👏secrets'

2. Run a Bandit scan on the file:

bandit -f txt -o report.txt test.py 

3. Observe the resulting crash and traceback:

[main]  INFO    profile include tests: None
[main]  INFO    profile exclude tests: None
[main]  INFO    cli include tests: None
[main]  INFO    cli exclude tests: None
[main]  INFO    running on Python 3.6.5
[node_visitor]  INFO    Unable to find qualified name for module: test.py
Traceback (most recent call last):
  File "c:\users\<username>\appdata\local\programs\python\python36\lib\site-packages\bandit\core\manager.py", line 157, in output_results
    conf_level=conf_level, lines=lines)
  File "c:\users\<username>\appdata\local\programs\python\python36\lib\site-packages\bandit\formatters\text.py", line 161, in report
    wrapped_file.write(utils.convert_file_contents(result))
  File "c:\users\<username>\appdata\local\programs\python\python36\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f44f' in position 135: character maps to <undefined>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\<username>\appdata\local\programs\python\python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\<username>\appdata\local\programs\python\python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\w107566\AppData\Local\Programs\Python\Python36\Scripts\bandit.exe\__main__.py", line 9, in <module>
  File "c:\users\<username>\appdata\local\programs\python\python36\lib\site-packages\bandit\cli\main.py", line 345, in main
    args.output_format)
  File "c:\users\<username>\appdata\local\programs\python\python36\lib\site-packages\bandit\core\manager.py", line 161, in output_results
    "%s" % (output_format, str(e)))
RuntimeError: Unable to output report using 'txt' formatter: 'charmap' codec can't encode character '\U0001f44f' in position 135: character maps to <undefined>

Expected behavior
I expect Bandit to successfully create the report, either by handing unicode encodings, or by removing the problematic unicode characters.

Bandit version

bandit 1.4.0

[codingo]

I visited this repository to report the same issue, can back up the original finding.

[stannum-l]

Wonder if it is bandit 1.4.0, I tried this and don't seem to see the issue. http://paste.openstack.org/show/727859/

[ericwb]

I wasn't able to recreate it either. But I haven't tried the combo of Python 3.6 with Bandit 1.4.0 yet. At the very least, even if fixed in master, we could add a new unit test for this.

[Brcrwilliams]

I grabbed my MacBook and tried it on there and I got the same result as @stannum-l. I think it's a Windows issue.

[stannum-l]

Ah, that makes sense. Don't think windows console support unicode; perhaps that's why there is an error.

[Brcrwilliams]

Ok so I whipped out the debugger and found out that for some reason, on Windows, the file opened by argparse.FileType is created with cp1252 encoding by default. Manually specifying the encoding as utf-8 fixes this.

Opened a PR at #364

[ericwb]

Have you tried setting the encoding in the Python file? Python will default to ASCII as standard encoding if no other encoding hints are given.

If using Unicode characters, according to PEP 263 you should declare the encoding within your Python file.

https://www.python.org/dev/peps/pep-0263/

#!/usr/bin/python
# -*- coding: utf-8 -*-

[Dantos7]

Issue is still present for the pre-commit hook at version 1.7.5