TypeError: 'buffer' does not have the buffer interface

[mfenner1]

Hi folks,

I don't have a better way to communicate with the project, so the following may be user error on my part and not a bug. But, here's what's going on:

(Morning of July 18) I grabbed the development branches of repo-handler and cvsanaly:

git clone -b develop git://github.com/SoftwareIntrospectionLab/repositoryhandler.git
git clone -b develop git://github.com/SoftwareIntrospectionLab/cvsanaly.git

I installed them (python setup.py install for both)

I moved into a downloaded project directory (which I successfully processed with the msr2011 tagged repohandler/cvsanaly):

cd voldemort

I fired up cvsanaly (results follow my question):

mfenner [663] % cvsanaly2 --db-driver sqlite -d ../tmpdb/vold3.db

Any thoughts on the following error? Database problem? Threading/execution problem? Other? I'll try to dig into.

Best,
Mark

results:

% cvsanaly2 --db-driver sqlite -d ../tmpdb/vold3.db
Parsing log for /voldemort (git)
Warning: Detected empty branch 'master', it'll be ignored
Warning: Detected empty branch 'li-r1106', it'll be ignored
Warning: Detected empty branch 'li-r1104', it'll be ignored
Warning: Detected empty branch 'li-r1028', it'll be ignored
Warning: Detected empty branch 'release-081', it'll be ignored
Warning: Detected empty branch 'release-0802', it'll be ignored
Warning: Detected empty branch 'release-0801', it'll be ignored
Warning: Detected empty branch 'release-080', it'll be ignored
Warning: Detected empty branch 'li-r1008', it'll be ignored
Warning: Detected empty branch 'release-070', it'll be ignored
Warning: Detected empty branch 'li-r1006', it'll be ignored
Warning: Detected empty branch 'release-060', it'll be ignored
Warning: Detected empty branch 'socketpool', it'll be ignored
Warning: Detected empty branch 'li-rel-926', it'll be ignored
Database looks empty, removing cache file /home/mfenner/.cvsanaly2/cache/git:__github.com_voldemort_voldemort.git
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 530, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 483, in run
self.__target(_self.__args, *_self.__kwargs)
File "/usr/lib64/python2.7/site-packages/cvsanaly-2.4-py2.7.egg/pycvsanaly2/DBProxyContentHandler.py", line 56, in __reader
templog.foreach(commit_cb, self.order)
File "/usr/lib64/python2.7/site-packages/cvsanaly-2.4-py2.7.egg/pycvsanaly2/DBTempLog.py", line 176, in foreach
io = BytesIO(obj)
TypeError: 'buffer' does not have the buffer interface

Executing extensions

[mfenner1]

So, two other pieces of info:

(1)
Oddly, given the error message:
TypeError: 'buffer' does not have the buffer interface

If we jump into pdb and look at obj (the obj from line 176 [ io = BytesIO(obj) ] we see that:
(Pdb) obj
<read-write buffer ptr 0x2588450, size 26667 at 0x2588410>

I don't have experience with Python's buffer objects, but I do know that there is are "new" and "old" style buffers. Could this be a mis-match between them?

(2) Reverting DBTempLog.py back to cStringIO (instead of BytesIO) got the database to build properly.

I tried out a few things to see what types fit where:
(Pdb) BytesIO(obj)
*** TypeError: 'buffer' does not have the buffer interface
(Pdb) BytesIO(buffer(obj))
*** TypeError: 'buffer' does not have the buffer interface
(Pdb) buffer(obj)
<read-only buffer for 0x7f8b48027af0, size -1, offset 0 at 0x7f8b480a0ab0>
(Pdb) memoryview(obj)
*** TypeError: cannot make memory view because object does not have the buffer interface

For convenience, here's the block of code around what I'm looking at (from DBTempLog.py starting at line 172 in dev):

    icursor = ICursor(cnn.cursor(), self.INTERVAL_SIZE)
    icursor.execute(statement(query, self.db.place_holder))
    rs = icursor.fetchmany()
    while rs:
        for t in rs:
            obj = t[0]
            import pdb; pdb.set_trace()
            #io = BytesIO(obj)
            io = StringIO(obj)
            commit = load(io)
            io.close()
            cb(commit)

        rs = icursor.fetchmany()

    icursor.close()
    cnn.close()

[cflewis]

Works for me using Python 2.6 and the latest Voldemort. Maybe it's a 2.7 problem?

dhcp-172-19-19-15:cvsanaly-test cflewis$ git clone -b develop git://github.com/SoftwareIntrospectionLab/repositoryhandler.git

dhcp-172-19-19-15:cvsanaly-test cflewis$ git clone -b develop git://github.com/SoftwareIntrospectionLab/cvsanaly.git

dhcp-172-19-19-15:cvsanaly-test cflewis$ cd repositoryhandler/

dhcp-172-19-19-15:repositoryhandler cflewis$ sudo python setup.py install

dhcp-172-19-19-15:repositoryhandler cflewis$ cd ../cvsanaly/
dhcp-172-19-19-15:cvsanaly cflewis$ sudo python setup.py install

dhcp-172-19-19-15:~ cflewis$ git clone git://github.com/voldemort/voldemort.git ~/

dhcp-172-19-19-15:cvsanaly cflewis$ cvsanaly2 --db-driver sqlite ~/voldemort

[mfenner1]

Yeah, so it appears that Python 2.7 merged the "new buffer interface" model (for compatibility with Python 3.0). I don't know much about the internals of it, but this thread on Python-Dev seems related:

http://mail.python.org/pipermail/python-dev/2010-October/104917.html

I'm not sure, but maybe if we wrap stuff with a str(obj) it will be sufficient. But, it seems like that would defeat the point of having a memory region accessed without a distinct, separate copy being created.