Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

carbonReader read incomplete data #4275

Open
Black-max12138 opened this issue May 26, 2022 · 0 comments
Open

carbonReader read incomplete data #4275

Black-max12138 opened this issue May 26, 2022 · 0 comments

Comments

@Black-max12138
Copy link

Look at this line of code. boolean hasNext = currentReader.nextKeyValue();
If hasNext returns false and currentReader is not the last one, it indicates that the iterator exits and subsequent data is not parsed. How to solve this problem?

/**

   * Return true if has next row
   */
  public boolean hasNext() throws IOException, InterruptedException {
    if (0 == readers.size() || currentReader == null) {
      return false;
    }
    validateReader();
    if (currentReader.nextKeyValue()) {
      return true;
    } else {
      if (index == readers.size() - 1) {
        // no more readers
        return false;
      } else {
        // current reader is closed
        currentReader.close();
        // no need to keep a reference to CarbonVectorizedRecordReader,
        // until all the readers are processed.
        // If readers count is very high,
        // we get OOM as GC not happened for any of the content in CarbonVectorizedRecordReader
        readers.set(index, null);
        index++;
        currentReader = readers.get(index);
        boolean hasNext = currentReader.nextKeyValue();
        if (hasNext) {
          return true;
        }
      }
    }
    return false;
  }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant