[HUDI-786] InlineFileSystem.read API should ensure content beyond inline length gets an EOF

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Component/s: Common Core
• Labels: bug-bash-0.6.0 pull-request-available
• Assignee: sivabalan narayanan
• Reporter: Vinoth Chandar
• Created: 11/Apr/20 23:39
• Updated: 30/May/20 12:28
• Resolved: 30/May/20 12:28

Description

While trying to investigate a flaky test, noticed that the readFully() just proceeds to read bytes from the outerStream without any bounds checking

@Override
  public void readFully(long position, byte[] buffer, int offset, int length) throws IOException {
    if ((length - offset) > this.length) {
      throw new IOException("Attempting to read past inline content");
    }
    outerStream.readFully(startOffset + position, buffer, offset, length);
  }

  @Override
  public void readFully(long position, byte[] buffer)
      throws IOException {
    readFully(position, buffer, 0, buffer.length);
  }

we need to throw an error for buffers that are trying to read past the inline content.. (potentially buggy) example shown above.

I have also ignored the TestInlineFileSystem#testFileSystemAPIs() ... we need to make a change to respect suffix length (we randomly generate) while attempting to read past the 1000 bytes of inline content..

 actualBytes = new byte[1000 + outerPathInfo.suffixLength];
    fsDataInputStream.readFully(0, actualBytes);
    verifyArrayEquality(outerPathInfo.expectedBytes, 0, 1000, actualBytes, 0, 1000);

Issue Links

• is depended upon by: HUDI-901 Bug Bash 0.6.0 Tracking Ticket
• links to: GitHub Pull Request #1510
• links to: GitHub Pull Request #1616
• links to: GitHub Pull Request #1616