Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bookie translate to readonly mode due to NullPointerException while flushing mem table. #4270

Open
zyj5340 opened this issue Apr 9, 2024 · 0 comments
Labels

Comments

@zyj5340
Copy link

zyj5340 commented Apr 9, 2024

BUG REPORT
When flushing mem table, the EntryMemTable.flush() method throws an NullPointerException and it cause the Bookie translate into readonly mode, and never recover to R/W mode.
Describe the bug
While processing FileInfo.readAbsolute() method , it passed the first check (fc == null) but got NullPointerException while doing fc.read(bb.start), we think the fc may be closed during two synchronized lock, and that cause the problem.

ERROR LOG:

2024-04-04 04:26:10,349Z [SortedLedgerStorage-0] ERROR org.apache.bookkeeper.bookie.SortedLedgerStorage - Exception thrown while flushing skip list cache.
java.lang.NullPointerException: Cannot invoke "java.nio.channels.FileChannel.read(java.nio.ByteBuffer, long)" because "this.fc" is null
        at org.apache.bookkeeper.bookie.FileInfo.readAbsolute(FileInfo.java:426) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3]
        at org.apache.bookkeeper.bookie.FileInfo.read(FileInfo.java:396) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3]
        at org.apache.bookkeeper.bookie.LedgerEntryPage.readPage(LedgerEntryPage.java:196) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3]
        at org.apache.bookkeeper.bookie.IndexPersistenceMgr.updatePage(IndexPersistenceMgr.java:650) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3]
        at org.apache.bookkeeper.bookie.IndexInMemPageMgr.grabLedgerEntryPage(IndexInMemPageMgr.java:449) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3]
        at org.apache.bookkeeper.bookie.IndexInMemPageMgr.getLedgerEntryPage(IndexInMemPageMgr.java:414) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3]
        at org.apache.bookkeeper.bookie.IndexInMemPageMgr.putEntryOffset(IndexInMemPageMgr.java:573) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3]
        at org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:107) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3]
        at org.apache.bookkeeper.bookie.InterleavedLedgerStorage.processEntry(InterleavedLedgerStorage.java:539) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3]
        at org.apache.bookkeeper.bookie.SortedLedgerStorage.process(SortedLedgerStorage.java:288) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3]
        at org.apache.bookkeeper.bookie.EntryMemTable.flushSnapshot(EntryMemTable.java:255) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3]
        at org.apache.bookkeeper.bookie.EntryMemTable.flush(EntryMemTable.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3]
       timed out waiting for input: auto-logoutgerStorage.java:317) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3]
Connection closed.il.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]

To Reproduce
It difficult to reproduce, happen occasionally.

Additional context
We want to add a second check if the fc is null in second synchronized, and if the fc equals null ,we clear the ByteBuffer and return 0. However, we don't know whether this change will cause another problem or is there a better way to fix this bug.
org.apache.bookkeeper.bookie.FileInfo#readAbsolute

/**
     * Read data from position <i>start</i> to fill the byte buffer <i>bb</i>.
     * If <i>bestEffort </i> is provided, it would return when it reaches EOF.
     * Otherwise, it would throw {@link org.apache.bookkeeper.bookie.ShortReadException}
     * if it reaches EOF.
     *
     * @param bb
     *          byte buffer of data
     * @param start
     *          start position to read data
     * @param bestEffort
     *          flag indicates if it is a best-effort read
     * @return number of bytes read
     * @throws IOException
     */
    private int readAbsolute(ByteBuffer bb, long start, boolean bestEffort)
            throws IOException {
        checkOpen(false);
        synchronized (this) {
            if (fc == null) {
                return 0;
            }
        }
        int total = 0;
        int rc = 0;
        while (bb.remaining() > 0) {
            synchronized (this) {
                // fix code start
                if (null == fc){
                    bb.clear();
                    return 0;
                }
                // fix code end
                rc = fc.read(bb, start);
            }
            if (rc <= 0) {
                if (bestEffort) {
                    return total;
                } else {
                    throw new ShortReadException("Short read at " + getLf().getPath() + "@" + start);
                }
            }
            total += rc;
            // should move read position
            start += rc;
        }
        return total;
    }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant