SystemError: error return without exception set

[ctheune]

@gforcada and I are running into semi-rare weird error messages that seem to indicate that something in the BTrees can't handle issues when an object can't be loaded properly. This seems to be somewhat of a race condition around the C-level implementation of getitem, as many times the exception will occur around something like those lines of code:

  File "/srv/s-derfreitag/deployment/work/plone/eggs/zope.intid-3.7.2-py2.7.egg/zope/intid/__init__.py", line 87, in getId
    return self.ids[key]
SystemError: error return without exception set

  Module Products.PluginIndexes.common.UnIndex, line 458, in _apply_index
SystemError: error return without exception set

    return self._tree[id].__of__(self)
SystemError: error return without exception set

Whenever this appears, either the Zope server is being shut down at the moment and ZEO has already disconnected, or the ZEO server has had a disconnect and immediately before those errors we see the ZEO client complaining:

  File "/srv/s-derfreitag/deployment/work/plone/eggs/ZODB3-3.10.7-py2.7-linux-x86_64.egg/ZODB/Connection.py", line 901, in _setstate
    p, serial = self._storage.load(obj._p_oid, '')
AttributeError: 'NoneType' object has no attribute 'load'

  File "/srv/s-derfreitag/deployment/work/plone/eggs/ZODB3-3.10.7-py2.7-linux-x86_64.egg/ZEO/ClientStorage.py", line 88, in __getattr__
    raise ClientDisconnected()

One thing I'm wondering about is why there would be anything still processing data or requests if Zope already disconnected the database. My memory tells me that Zope would allow threads to finish their requests and then shut down.

So, from reading and googling the error message it seems somewhere the Python C API is signalled an error but no exception is set. I looked around a bit and a wild guess would be that this could be an issue from cPersistence in the PER_USE macro (it seems to signal an error condition but doesn't set an exception apparently) or by the _bucket_get function in BucketTemplate.c:86 where we return NULL indicating an error and seem to implictly rely on someone else already having set an exception. I could be digging at the completely wrong end, though ... :)

[gforcada]

Just one extra information regarding the AttributeError: 'NoneType' object has no attribute 'load' we had those since a looooong time ago, maybe since the very beginning... at least much much before we had the SystemError.

[NicolasGoeddel]

We've got exactly that problem on an old Plone project we are currently migrating. It happens when trying to rename specific folders or when trying to delete them.

2020-07-02 17:49:41 ERROR five.intid error return without exception set
Traceback (most recent call last):
  File "/home/user/buildout-cache/eggs/five.intid-1.1.2-py2.7.egg/five/intid/intid.py", line 162, in moveIntIdSubscriber
    uid = utility.getId(ob)
  File "/home/user/buildout-cache/eggs/five.intid-1.1.2-py2.7.egg/five/intid/intid.py", line 44, in getId
    return z3IntIds.getId(self, ob)
  File "/home/user/buildout-cache/eggs/zope.intid-3.7.2-py2.7.egg/zope/intid/__init__.py", line 87, in getId
    return self.ids[key]
SystemError: error return without exception set

Has anyone an idea how to fix that or getting these objects deleted ignoring that error? We just wanna get rid of them. It does also not work in the Zope management interface. Or is there a tool which just scans the whole site and can detect these kind of errors?

[d-maurer]

NicolasGoeddel wrote at 2020-7-2 08:55 -0700:

We've got exactly that problem on an old Plone project we are currently migrating. It happens when trying to rename specific folders or when trying to delete them. ``` 2020-07-02 17:49:41 ERROR five.intid error return without exception set Traceback (most recent call last): File "/home/user/buildout-cache/eggs/five.intid-1.1.2-py2.7.egg/five/intid/intid.py", line 162, in moveIntIdSubscriber uid = utility.getId(ob) File "/home/user/buildout-cache/eggs/five.intid-1.1.2-py2.7.egg/five/intid/intid.py", line 44, in getId return z3IntIds.getId(self, ob) File "/home/user/buildout-cache/eggs/zope.intid-3.7.2-py2.7.egg/zope/intid/__init__.py", line 87, in getId return self.ids[key] SystemError: error return without exception set ``` Has anyone an idea how to fix that or getting these objects deleted ignoring that error? We just wanna get rid of them. It does also not work in the Zope management interface. Or is there a tool which just scans the whole site and can detect these kind of errors?

Maybe, the `BTrees` became physically damanged. This should not happen but computers and hardware are only almost (but not fully) reliable. In my life, I have met one such damaged `BTree`. If you are lucky, then the sequence of leaves are still accessible (--> `leaves = list(tree.items())`. In this case, you can clear the tree and then recreate it from its leaves (--> `tree.clear(); for i in leaves: tree[i[0]] = i[1]`).

[NicolasGoeddel]

Hi Dieter,
on the frontend side you can navigate flawlessly. Only deleting and renaming just does not work and creates these errors.

I am not sure at which point I can access the BTrees to recreate and save them persistently. Usually I would inject import pdb; pdb.set_trace() right before the exception and look around the variables. I first will it test that way. If this does not work, I will come back here.

[d-maurer]

NicolasGoeddel wrote at 2020-7-3 01:16 -0700:

I am not sure at which point I can access the BTrees to recreate and save them persistently. Usually I would inject `import pdb; pdb.set_trace()` right before the exception and look around the variables. I first will it test that way. If this does not work, I will come back here.

For fixups of this kind, I prefer the "interactive Zope session" (i.e. `bin/{instance|client1} debug`) There, you can explore the ZODB content and (if necessary) change it (and after a change test it). Use "transaction.commit()" to make changes persistent.

[NicolasGoeddel]

Okay, I got rid of the folders making these problems. I used the debug mode but instead of trying to recreate BTrees I simply used plone.api.content.delete() with check_integrity=False and a transaction.commit() afterwards. There was still an exception after the delete but now it's gone and now a reindex runs without an issue.

So for my case it seems solved now.

[jamadden]

I've been going over these code paths (both the __getitem__ path, and the paths that involve unghosting objects, both BTrees/buckets and keys). It's not clear to me, by inspection, where we could be returning NULL without an exception being set. All the paths I've traced through that return NULL appear to only be doing so when an exception is set.

There is one exception for an "impossible" case: a tuple with a negative size. If we encounter such a tuple in __setstate__, we immediately return an error without setting an exception. Tuples with a negative size aren't supposed to be possible, and unless there's memory corruption at the C level, the API doesn't allow you to create them (even with a malicious or malformed pickle). Still, it would be an easy, no-runtime-cost thing to do to make that case set an exception.

If we do that, and someone can still reproduce this, then we can add some ASSERT(result != NULL || PyErr_Occurred()) to some key points to narrow down exactly where things are going wrong.