Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16998

WASB : NativeAzureFsOutputStream#close() throwing IllegalArgumentException

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.1, 3.4.0
    • Component/s: fs/azure
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      During HFile create, at the end when called close() on the OutputStream, there is some pending data to get flushed. When this flush happens, an Exception is thrown back from Storage. The Azure-storage SDK layer will throw back IOE. (Even if it is a StorageException thrown from the Storage, the SDK converts it to IOE.) But at HBase, we end up getting IllegalArgumentException which causes the RS to get aborted. If we get back IOE, the flush will get retried instead of aborting RS.
      The reason is this
      NativeAzureFsOutputStream uses Azure-storage SDK's BlobOutputStreamInternal. But the BlobOutputStreamInternal is wrapped within a SyncableDataOutputStream which is a FilterOutputStream. During the close op, NativeAzureFsOutputStream calls close on SyncableDataOutputStream and it uses below method from FilterOutputStream

      public void close() throws IOException {
        try (OutputStream ostream = out) {
                    flush();
        }
      }
      

      Here the flush call caused an IOE to be thrown to here. The finally will issue close call on ostream (Which is an instance of BlobOutputStreamInternal)
      When BlobOutputStreamInternal#close() is been called, if there was any exception already occured on that Stream, it will throw back the same Exception

      public synchronized void close() throws IOException {
        try {
                    // if the user has already closed the stream, this will throw a STREAM_CLOSED exception
                    // if an exception was thrown by any thread in the threadExecutor, realize it now
                    this.checkStreamState();
                    ...
      }
      private void checkStreamState() throws IOException {
        if (this.lastError != null) {
                    throw this.lastError;
        }
      }
      

      So here both try and finally block getting Exceptions and Java uses Throwable#addSuppressed()
      Within this method if both Exceptions are same objects, it throws back IllegalArgumentException

      public final synchronized void addSuppressed(Throwable exception) {
                    if (exception == this)
                                   throw new IllegalArgumentException(SELF_SUPPRESSION_MESSAGE, exception);
                    ....
      }
      

        Attachments

        1. HADOOP-16998.patch
          2 kB
          Anoop Sam John

          Issue Links

            Activity

              People

              • Assignee:
                anoop.hbase Anoop Sam John
                Reporter:
                anoop.hbase Anoop Sam John
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: