[HDFS-15679] DFSOutputStream close should be a no-op when called multiple times

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Later
• Labels: pull-request-available
• Assignee: Ahmed Hussein
• Reporter: Ahmed Hussein
• Created: 10/Nov/20 18:06
• Updated: 18/Dec/20 18:56
• Resolved: 18/Dec/20 18:56

Description

While I was looking into the incorrect implementation of HDFS-15678, I found that once I implement the correct logic, the Junit test fails.
It turns out that there is inconsistency in DFSOutputStream.closeImpl() introduced by HDFS-13164.

The change in that line makes the close() throws exception multiple times which contradicts with HDFS-5335.

That change breaks the semantic of close(). For convenience, this is a test code that fails because of the change in HDFS-13164.

  public void testCloseTwice() throws IOException {
    DistributedFileSystem fs = cluster.getFileSystem();
    FSDataOutputStream os = fs.create(new Path("/test"));
    DFSOutputStream dos = (DFSOutputStream) Whitebox.getInternalState(os,
        "wrappedStream");
    DataStreamer streamer = (DataStreamer) Whitebox
        .getInternalState(dos, "streamer");
    @SuppressWarnings("unchecked")
    LastExceptionInStreamer ex = (LastExceptionInStreamer) Whitebox
        .getInternalState(streamer, "lastException");
    Throwable thrown = (Throwable) Whitebox.getInternalState(ex, "thrown");
    Assert.assertNull(thrown);
    // force stream to break. output stream needs to encounter a real
    // error to properly mark it closed with an exception
    cluster.shutdown(true, false);
    try {
      dos.close();
      Assert.fail("should have thrown");
    } catch (IOException e) {
      Assert.assertEquals(e.toString(), EOFException.class, e.getClass());
    }
    thrown = (Throwable) Whitebox.getInternalState(ex, "thrown");
    Assert.assertNull(thrown);
    dos.close();
    // even if the exception is set again, close should not throw it
    ex.set(new IOException("dummy"));
    dos.close();
  }

Issue Links

• blocks: HDFS-15678 TestDFSOutputStream#testCloseTwice implementation is broken
• is caused by: HDFS-13164 File not closed if streamer fail with DSQuotaExceededException
• supercedes: HDFS-15678 TestDFSOutputStream#testCloseTwice implementation is broken
• links to: GitHub Pull Request #2456