Code to preserve original linefeeds (issue #121) #131
Conversation
|
Nice, I like this idea |
| # detect linefeeds | ||
| lineends = {'\n':0, '\r\n':0, '\r':0} | ||
| lines = [] | ||
| for line in open(filename, 'rb'): |
There was a problem hiding this comment.
To make this work the same way on Python 2 and 3, use:
io.open(filename, newline='')There was a problem hiding this comment.
But file needs to be opened in binary more or else the information about lineends will be lost.
There was a problem hiding this comment.
No, the newline='' means it will do no conversion of line endings. Binary mode has a much bigger effect on Python 3, because it means you're dealing with bytes rather than strs.
There was a problem hiding this comment.
But binary mode guarantees that Python 3 won't bail out with UnicodeDecodeError. How to address that? I can't know the file encoding before opening it as Python 3 requires.
There was a problem hiding this comment.
Fair point. There are ways around that, like using errors='ignore' to skip characters that can't be decoded, but it may be easier to use binary mode here. In that case, though, you'll need to update the checks below to use bytes, e.g. if line.endswith(b'\r\n').
|
Well, Python 3 is a pain. But it should work for now on Python 2. I may be able to finish this later. |
| newline = [x for x in lineends if lineends[x] != 0][0] | ||
| if os.linesep != newline: | ||
| with open(filename, 'wb') as f: | ||
| for line in lines: |
There was a problem hiding this comment.
You're using the lines that you read before the file was rewritten, so this will undo the changes modernize made. You need to base it on new_text, or re-read the file after it is changed.
There was a problem hiding this comment.
Right! Fixed in 7787615
|
I am not sure I can work on a test. Not today. Basically, the test need to create two files - one with LF linefeeds and another with CRLF. And after fix is applied, check that linefeeds didn't change. So either produce 4 files (input and output) or copy/paste the function being tested to the test case (which is no good). The files are also can not be committed to Git, because it messes with linefeeds. |
| CRLF = '\r\n' | ||
| CR = '\r' | ||
| else: | ||
| LF = bytes('\n', encoding='ascii') |
There was a problem hiding this comment.
Just use the b prefix, like b'\n'. It's valid syntax on Python 2.6 and above, which is what we support. And then you don't need an if/else, because on Python 2, b'\n' == '\n'.
|
This isn't correct as-is, but I like the basic approach. I will work on it tomorrow. |
|
@techtonik I think this feature would be best in https://github.com/jreese/fissix |
|
@graingert I see this project became derived from |
|
@techtonik not much has changed, but you can see what enhancements are applied here: Everything goes via black + a git merge from CPython @jreese maybe a changelog will help? |
This is pseudo code - it may work or may not. I haven't got chance to test it yet