Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13403

AzureNativeFileSystem rename/delete performance improvements

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: fs/azure
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      WASB has added an optional capability to execute certain FileSystem operations in parallel on multiple threads for improved performance. Please refer to the Azure Blob Storage documentation page for more information on how to enable and control the feature.
      Show
      WASB has added an optional capability to execute certain FileSystem operations in parallel on multiple threads for improved performance. Please refer to the Azure Blob Storage documentation page for more information on how to enable and control the feature.
    • Flags:
      Patch

      Description

      WASB Performance Improvements

      Problem
      -----------
      Azure Native File system operations like rename/delete which has large number of directories and/or files in the source directory are experiencing performance issues. Here are possible reasons
      a) We first list all files under source directory hierarchically. This is a serial operation.
      b) After collecting the entire list of files under a folder, we delete or rename files one by one serially.
      c) There is no logging information available for these costly operations even in DEBUG mode leading to difficulty in understanding wasb performance issues.

      Proposal
      -------------
      Step 1: Rename and delete operations will generate a list all files under the source folder. We need to use azure flat listing option to get list with single request to azure store. We have introduced config fs.azure.flatlist.enable to enable this option. The default value is 'false' which means flat listing is disabled.

      Step 2: Create thread pool and threads dynamically based on user configuration. These thread pools will be deleted after operation is over. We are introducing introducing two new configs
      a) fs.azure.rename.threads : Config to set number of rename threads. Default value is 0 which means no threading.
      b) fs.azure.delete.threads: Config to set number of delete threads. Default value is 0 which means no threading.

      We have provided debug log information on number of threads not used for the operation which can be useful .

      Failure Scenarios:
      If we fail to create thread pool due to ANY reason (for example trying create with thread count with large value such as 1000000), we fall back to serialization operation.

      Step 3: Bob operations can be done in parallel using multiple threads executing following snippet
      while ((currentIndex = fileIndex.getAndIncrement()) < files.length)

      { FileMetadata file = files[currentIndex]; Rename/delete(file); }

      The above strategy depends on the fact that all files are stored in a final array and each thread has to determine synchronized next index to do the job. The advantage of this strategy is that even if user configures large number of unusable threads, we always ensure that work doesn’t get serialized due to lagging threads.

      We are logging following information which can be useful for tuning number of threads

      a) Number of unusable threads
      b) Time taken by each thread
      c) Number of files processed by each thread
      d) Total time taken for the operation

      Failure Scenarios:

      Failure to queue a thread execute request shouldn’t be an issue if we can ensure at least one thread has completed execution successfully. If we couldn't schedule one thread then we should take serialization path. Exceptions raised while executing threads are still considered regular exceptions and returned to client as operation failed. Exceptions raised while stopping threads and deleting thread pool shouldn't can be ignored if operation all files are done with out any issue.

      1. HADOOP-13403-001.patch
        40 kB
        Subramanyam Pattipaka
      2. HADOOP-13403-002.patch
        58 kB
        Subramanyam Pattipaka
      3. HADOOP-13403-003.patch
        66 kB
        Subramanyam Pattipaka
      4. HADOOP-13403-004.patch
        67 kB
        Subramanyam Pattipaka
      5. HADOOP-13403-005.patch
        67 kB
        Subramanyam Pattipaka
      6. HADOOP-13403-006.patch
        67 kB
        Subramanyam Pattipaka

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          Hello Subramanyam Pattipaka. This sounds interesting. I won't be certain until I see the code changes, but some of it (particularly the "flat listing" option) sounds similar to optimizations done for S3A listings in HADOOP-13208. Cc'ing Steve Loughran who authored that patch, just FYI.

          Show
          cnauroth Chris Nauroth added a comment - Hello Subramanyam Pattipaka . This sounds interesting. I won't be certain until I see the code changes, but some of it (particularly the "flat listing" option) sounds similar to optimizations done for S3A listings in HADOOP-13208 . Cc'ing Steve Loughran who authored that patch, just FYI.
          Hide
          pattipaka Subramanyam Pattipaka added a comment -

          Chris Nauroth, I have submitted initial patch for review. HADOOP-13208 seems to generic implementation. Still going through details. My changes are specific to Native Azure FileSystem. I am using the flat listing option provided by Azure Storage client which returns all files and directories.

          Show
          pattipaka Subramanyam Pattipaka added a comment - Chris Nauroth , I have submitted initial patch for review. HADOOP-13208 seems to generic implementation. Still going through details. My changes are specific to Native Azure FileSystem. I am using the flat listing option provided by Azure Storage client which returns all files and directories.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          HADOOP-13208 is for S3a; this does sound similar, at least for listing.

          What you are proposing here is async rename/delete, which is somewhat akin to HDFS-9924: the caller issues a command and doesn't wait for a response.

          I'm worried here at out the fact you propose radically changing the semantics of rename and delete, so that they are now async. A lot of code expects rename(), a the very least, to be synchronous. (They also assume its atomic O(1),....). If the application can explicitly enable this, or sets up its own async thread, we'll know that the app is compatible with async mv/rm. Otherwise, I'm doubtful.

          Listing wise, yes, I'm trying to speed that up. FWIW, speeding up globStatus() looks even more promising, though I have to tread carefully to avoid making performance worse for real-world directory structures

          Show
          stevel@apache.org Steve Loughran added a comment - HADOOP-13208 is for S3a; this does sound similar, at least for listing. What you are proposing here is async rename/delete, which is somewhat akin to HDFS-9924 : the caller issues a command and doesn't wait for a response. I'm worried here at out the fact you propose radically changing the semantics of rename and delete, so that they are now async. A lot of code expects rename(), a the very least, to be synchronous. (They also assume its atomic O(1),....). If the application can explicitly enable this, or sets up its own async thread, we'll know that the app is compatible with async mv/rm. Otherwise, I'm doubtful. Listing wise, yes, I'm trying to speed that up. FWIW, speeding up globStatus() looks even more promising, though I have to tread carefully to avoid making performance worse for real-world directory structures
          Hide
          pattipaka Subramanyam Pattipaka added a comment -

          Adding Kiran Kumar Kolli

          Steve Loughran, This change doesn't introduce async rename/delete. The current proposal is still a blocking operation at delete/rename File System call level. This performance issue is specific Azure Native File System. This change shouldn't affect any API contracts at File System level.

          If a folder has large number of files and directories then we are deleting one file or directory at a time with current code serially. Delete/rename on multiple individual files can be done in parallel and we are using multiple threads to do these operations on individual files or directories. I have already attached path which has code level details. Please let me know if you have any more concerns.

          Show
          pattipaka Subramanyam Pattipaka added a comment - Adding Kiran Kumar Kolli Steve Loughran , This change doesn't introduce async rename/delete. The current proposal is still a blocking operation at delete/rename File System call level. This performance issue is specific Azure Native File System. This change shouldn't affect any API contracts at File System level. If a folder has large number of files and directories then we are deleting one file or directory at a time with current code serially. Delete/rename on multiple individual files can be done in parallel and we are using multiple threads to do these operations on individual files or directories. I have already attached path which has code level details. Please let me know if you have any more concerns.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I see -no worries. I was thinking of the next step —to do it fully asynchronously. I do think I'd like that, but we will need to extend the API or add it to applications

          Show
          stevel@apache.org Steve Loughran added a comment - I see -no worries. I was thinking of the next step —to do it fully asynchronously. I do think I'd like that, but we will need to extend the API or add it to applications
          Hide
          pattipaka Subramanyam Pattipaka added a comment -

          Uploaded new patch. I have refactored code and added extra unit tests for failure scenarios.

          Show
          pattipaka Subramanyam Pattipaka added a comment - Uploaded new patch. I have refactored code and added extra unit tests for failure scenarios.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 33s trunk passed
          +1 compile 0m 17s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 19s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 25s trunk passed
          +1 javadoc 0m 13s trunk passed
          +1 mvninstall 0m 15s the patch passed
          +1 compile 0m 14s the patch passed
          +1 javac 0m 14s the patch passed
          -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 39 new + 43 unchanged - 1 fixed = 82 total (was 44)
          +1 mvnsite 0m 18s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          -1 whitespace 0m 0s The patch has 55 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 2s The patch 4 line(s) with tabs.
          -1 findbugs 0m 39s hadoop-tools/hadoop-azure generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
          +1 javadoc 0m 11s the patch passed
          +1 unit 1m 31s hadoop-azure in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          13m 27s



          Reason Tests
          FindBugs module:hadoop-tools/hadoop-azure
            Redundant nullcheck of ioThreadPool, which is known to be non-null in org.apache.hadoop.fs.azure.NativeAzureFileSystem.executeParallel(NativeAzureFileSystem$AzureFileSystemOperation, String, FileMetadata[], String, int, Configuration, String, NativeAzureFileSystem$AzureFileSystemThreadOperation) Redundant null check at NativeAzureFileSystem.java:is known to be non-null in org.apache.hadoop.fs.azure.NativeAzureFileSystem.executeParallel(NativeAzureFileSystem$AzureFileSystemOperation, String, FileMetadata[], String, int, Configuration, String, NativeAzureFileSystem$AzureFileSystemThreadOperation) Redundant null check at NativeAzureFileSystem.java:[line 910]
            Should org.apache.hadoop.fs.azure.NativeAzureFileSystem$AzureFileSystemThreadFactory be a static inner class? At NativeAzureFileSystem.java:inner class? At NativeAzureFileSystem.java:[lines 717-737]
            new org.apache.hadoop.fs.azure.NativeAzureFileSystem$AzureFileSystemThreadRunnable(NativeAzureFileSystem, NativeAzureFileSystem$AzureFileSystemOperation, String, FileMetadata[], NativeAzureFileSystem$AzureFileSystemThreadOperation) may expose internal representation by storing an externally mutable object into NativeAzureFileSystem$AzureFileSystemThreadRunnable.files At NativeAzureFileSystem.java:NativeAzureFileSystem$AzureFileSystemThreadOperation) may expose internal representation by storing an externally mutable object into NativeAzureFileSystem$AzureFileSystemThreadRunnable.files At NativeAzureFileSystem.java:[line 796]
            Should org.apache.hadoop.fs.azure.NativeAzureFileSystem$AzureFileSystemThreadRunnable be a static inner class? At NativeAzureFileSystem.java:inner class? At NativeAzureFileSystem.java:[lines 766-841]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820673/HADOOP-13403-002.patch
          JIRA Issue HADOOP-13403
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 52faf7262ecd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 26de4f0
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10108/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10108/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10108/artifact/patchprocess/whitespace-tabs.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10108/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-azure.html
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10108/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10108/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 33s trunk passed +1 compile 0m 17s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 19s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 15s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 39 new + 43 unchanged - 1 fixed = 82 total (was 44) +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 9s the patch passed -1 whitespace 0m 0s The patch has 55 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 2s The patch 4 line(s) with tabs. -1 findbugs 0m 39s hadoop-tools/hadoop-azure generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) +1 javadoc 0m 11s the patch passed +1 unit 1m 31s hadoop-azure in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 13m 27s Reason Tests FindBugs module:hadoop-tools/hadoop-azure   Redundant nullcheck of ioThreadPool, which is known to be non-null in org.apache.hadoop.fs.azure.NativeAzureFileSystem.executeParallel(NativeAzureFileSystem$AzureFileSystemOperation, String, FileMetadata[], String, int, Configuration, String, NativeAzureFileSystem$AzureFileSystemThreadOperation) Redundant null check at NativeAzureFileSystem.java:is known to be non-null in org.apache.hadoop.fs.azure.NativeAzureFileSystem.executeParallel(NativeAzureFileSystem$AzureFileSystemOperation, String, FileMetadata[], String, int, Configuration, String, NativeAzureFileSystem$AzureFileSystemThreadOperation) Redundant null check at NativeAzureFileSystem.java: [line 910]   Should org.apache.hadoop.fs.azure.NativeAzureFileSystem$AzureFileSystemThreadFactory be a static inner class? At NativeAzureFileSystem.java:inner class? At NativeAzureFileSystem.java: [lines 717-737]   new org.apache.hadoop.fs.azure.NativeAzureFileSystem$AzureFileSystemThreadRunnable(NativeAzureFileSystem, NativeAzureFileSystem$AzureFileSystemOperation, String, FileMetadata[], NativeAzureFileSystem$AzureFileSystemThreadOperation) may expose internal representation by storing an externally mutable object into NativeAzureFileSystem$AzureFileSystemThreadRunnable.files At NativeAzureFileSystem.java:NativeAzureFileSystem$AzureFileSystemThreadOperation) may expose internal representation by storing an externally mutable object into NativeAzureFileSystem$AzureFileSystemThreadRunnable.files At NativeAzureFileSystem.java: [line 796]   Should org.apache.hadoop.fs.azure.NativeAzureFileSystem$AzureFileSystemThreadRunnable be a static inner class? At NativeAzureFileSystem.java:inner class? At NativeAzureFileSystem.java: [lines 766-841] Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820673/HADOOP-13403-002.patch JIRA Issue HADOOP-13403 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 52faf7262ecd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 26de4f0 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10108/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10108/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10108/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10108/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-azure.html Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10108/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10108/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Hello Subramanyam Pattipaka. Thank you for the patch.

          I started code reviewing this while doing a full test run with the patch against my Azure Storage account. I saw these failures:

          Tests run: 67, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 591.899 sec <<< FAILURE! - in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
          testRenameSigleRenameException(org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads)  Time elapsed: 7.062 sec  <<< ERROR!
          java.lang.Exception: Unexpected exception, expected<java.io.IOException> but was<java.lang.AssertionError>
          	at org.junit.Assert.fail(Assert.java:86)
          	at org.junit.Assert.assertTrue(Assert.java:41)
          	at org.junit.Assert.assertTrue(Assert.java:52)
          	at org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads.testRenameSigleRenameException(TestFileSystemOperationsWithThreads.java:630)
          
          testDeleteSigleDeleteException(org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads)  Time elapsed: 4.289 sec  <<< ERROR!
          java.lang.Exception: Unexpected exception, expected<java.io.IOException> but was<java.lang.AssertionError>
          	at org.junit.Assert.fail(Assert.java:86)
          	at org.junit.Assert.assertTrue(Assert.java:41)
          	at org.junit.Assert.assertTrue(Assert.java:52)
          	at org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads.testDeleteSigleDeleteException(TestFileSystemOperationsWithThreads.java:479)
          
          testDeleteSigleDeleteFailure(org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads)  Time elapsed: 4.403 sec  <<< FAILURE!
          java.lang.AssertionError: null
          	at org.junit.Assert.fail(Assert.java:86)
          	at org.junit.Assert.assertTrue(Assert.java:41)
          	at org.junit.Assert.assertFalse(Assert.java:64)
          	at org.junit.Assert.assertFalse(Assert.java:74)
          	at org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads.testDeleteSigleDeleteFailure(TestFileSystemOperationsWithThreads.java:455)
          

          I'll halt my code review at this point so that you can investigate the test failures. Here is my feedback so far, based on what I've read of the patch.

                threadCount = conf.getInt(config, defaultThreadCount);
          

          I recommend restructuring this so that the thread count for delete and rename are read just once in NativeAzureFileSystem#initialize and stored to member variables. Accessing Configuration can be expensive, and contributors reading the code generally expect to find all config access in initialize, not throughout other methods.

                // Enable flat listing option only if depth is unboubded and config 
          

          Typo: "unbounded".

          Date start = new Date();
          // Do something.
          Date end = new Date();
          long duration = end.getTime() - start.getTime();
          

          For all occurrences of logic like the above, please switch to using org.apache.hadoop.util.Time#monotonicNow(), which is a wrapper over the JDK java.lang.System#nanoTime(). By switching to this, the logic won't be prone to issues caused by clock drift or resetting time on the host.

              private String threadIdPrefix = "AzureFileSytemThread";
          

          Typo: "AzureFileSystemThread".

            enum AzureFileSystemOperation{
              Unkown,
              Delete,
              Rename
            }
          

          Typo: "Unknown".

            class AzureFileSystemThreadFactory implements ThreadFactory {
          
            public class AzureFileSystemThreadRunnable implements Runnable {
          

          NativeAzureFileSystem is a pretty long class already. I propose refactoring AzureFileSystemThreadFactory and AzureFileSystemThreadRunnable to top-level classes with package-private visibility in separate files.

                  if ((ioThreadPool = getThreadPool(threadCount, threadNamePrefix)) != null) {
          

          I don't understand the null check. getThreadPool cannot return null, because it simply calls the ThreadPoolExecutor constructor and returns the new object. I see some tests mock it to return null, but since this condition cannot really happen, why do we need test coverage for it? The same feedback applies to the exception handling. I do not see any checked exceptions thrown from getThreadPool. The ThreadPoolExecutor constructor can throw unchecked exceptions if the arguments don't make sense (e.g. negative thread pool size), but we can validate the sanity of those configuration parameters during initialize after following the feedback above about configuration handling.

                    lastException = new IOException(operation + " failed as operation on subfolers and files failed.");
          

          Typo: "subfolders".

            @Test(expected=IOException.class)
          

          Please don't use this technique for writing tests that expect a certain exception. The problem with this is that it's very coarse-grained. It allows the test to pass if an IOException is thrown from any point in the method. There are multiple points in these test methods where the exception can be thrown. Even if it's thrown from the wrong place, JUnit will still think the test passed. Instead, I recommend using the JUnit ExpectedException rule, which gives you fine-grained control of declaring exactly where you expect the exception to be thrown. See TestS3AAWSCredentialsProvider for an example of an existing test suite that uses ExpectedException.

            public void testDeleteSigleDeleteFailure() throws Exception {
          
            public void testDeleteSigleDeleteException() throws Exception {
          
            public void testRenameSigleRenameException() throws Exception {
          

          Same typo on all 3: "Single".

          The FindBugs and Checkstyle warnings from the last pre-commit run look legitimate, so please fix those. You can check these reports locally by running mvn -o findbugs:findbugs and mvn -o checkstyle:checkstyle.

          I recommend updating the documentation at src/site/markdown/index.md to describe this new feature and its configuration properties.

          Show
          cnauroth Chris Nauroth added a comment - Hello Subramanyam Pattipaka . Thank you for the patch. I started code reviewing this while doing a full test run with the patch against my Azure Storage account. I saw these failures: Tests run: 67, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 591.899 sec <<< FAILURE! - in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads testRenameSigleRenameException(org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads) Time elapsed: 7.062 sec <<< ERROR! java.lang.Exception: Unexpected exception, expected<java.io.IOException> but was<java.lang.AssertionError> at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads.testRenameSigleRenameException(TestFileSystemOperationsWithThreads.java:630) testDeleteSigleDeleteException(org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads) Time elapsed: 4.289 sec <<< ERROR! java.lang.Exception: Unexpected exception, expected<java.io.IOException> but was<java.lang.AssertionError> at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads.testDeleteSigleDeleteException(TestFileSystemOperationsWithThreads.java:479) testDeleteSigleDeleteFailure(org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads) Time elapsed: 4.403 sec <<< FAILURE! java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertFalse(Assert.java:64) at org.junit.Assert.assertFalse(Assert.java:74) at org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads.testDeleteSigleDeleteFailure(TestFileSystemOperationsWithThreads.java:455) I'll halt my code review at this point so that you can investigate the test failures. Here is my feedback so far, based on what I've read of the patch. threadCount = conf.getInt(config, defaultThreadCount); I recommend restructuring this so that the thread count for delete and rename are read just once in NativeAzureFileSystem#initialize and stored to member variables. Accessing Configuration can be expensive, and contributors reading the code generally expect to find all config access in initialize , not throughout other methods. // Enable flat listing option only if depth is unboubded and config Typo: "unbounded". Date start = new Date(); // Do something. Date end = new Date(); long duration = end.getTime() - start.getTime(); For all occurrences of logic like the above, please switch to using org.apache.hadoop.util.Time#monotonicNow() , which is a wrapper over the JDK java.lang.System#nanoTime() . By switching to this, the logic won't be prone to issues caused by clock drift or resetting time on the host. private String threadIdPrefix = "AzureFileSytemThread" ; Typo: "AzureFileSystemThread". enum AzureFileSystemOperation{ Unkown, Delete, Rename } Typo: "Unknown". class AzureFileSystemThreadFactory implements ThreadFactory { public class AzureFileSystemThreadRunnable implements Runnable { NativeAzureFileSystem is a pretty long class already. I propose refactoring AzureFileSystemThreadFactory and AzureFileSystemThreadRunnable to top-level classes with package-private visibility in separate files. if ((ioThreadPool = getThreadPool(threadCount, threadNamePrefix)) != null ) { I don't understand the null check. getThreadPool cannot return null , because it simply calls the ThreadPoolExecutor constructor and returns the new object. I see some tests mock it to return null , but since this condition cannot really happen, why do we need test coverage for it? The same feedback applies to the exception handling. I do not see any checked exceptions thrown from getThreadPool . The ThreadPoolExecutor constructor can throw unchecked exceptions if the arguments don't make sense (e.g. negative thread pool size), but we can validate the sanity of those configuration parameters during initialize after following the feedback above about configuration handling. lastException = new IOException(operation + " failed as operation on subfolers and files failed." ); Typo: "subfolders". @Test(expected=IOException.class) Please don't use this technique for writing tests that expect a certain exception. The problem with this is that it's very coarse-grained. It allows the test to pass if an IOException is thrown from any point in the method. There are multiple points in these test methods where the exception can be thrown. Even if it's thrown from the wrong place, JUnit will still think the test passed. Instead, I recommend using the JUnit ExpectedException rule, which gives you fine-grained control of declaring exactly where you expect the exception to be thrown. See TestS3AAWSCredentialsProvider for an example of an existing test suite that uses ExpectedException . public void testDeleteSigleDeleteFailure() throws Exception { public void testDeleteSigleDeleteException() throws Exception { public void testRenameSigleRenameException() throws Exception { Same typo on all 3: "Single". The FindBugs and Checkstyle warnings from the last pre-commit run look legitimate, so please fix those. You can check these reports locally by running mvn -o findbugs:findbugs and mvn -o checkstyle:checkstyle . I recommend updating the documentation at src/site/markdown/index.md to describe this new feature and its configuration properties.
          Hide
          cnauroth Chris Nauroth added a comment -

          Subramanyam Pattipaka, in addition to the earlier code review feedback, I have a higher-level question about the implementation of the executeParallel method.

          Typical usage of a ThreadPoolExecutor is to configure it with a certain thread count and then submit multiple tasks, with each task performing a single isolated action. When tasks are submitted, they get added to an internal queue (the LinkedBlockingQueue in your code). Then, the number of threads all independently pull and execute tasks from the queue.

          The executeParallel method in your current patch does not follow this typical usage of ThreadPoolExecutor. Instead, it submits a number of tasks identical to the thread count, and the code in each task iterates through an array of work items (FileMetadata[]) tracked manually. The array is shared state across all of those tasks/threads, so safe access requires additional synchronization code.

          It appears the current executeParallel reimplements functionality already built into the ThreadPoolExecutor. Switching this code to more idiomatic usage of ThreadPoolExecutor would make it easier for maintainers to understand, and it would remove the need for extra cross-thread synchronization, which is always tricky to get right. For example, assuming 10 threads configured for fs.azure.delete.threads and a deletion of 100 blobs, I recommend using a ThreadPoolExecutor configured with 10 threads and submitting 100 task instances to it, each task deleting a single blob, with no shared state across the tasks/threads.

          I also see the current logic carefully tracks the last exception that happened on a thread. Maybe that's why you implemented executeParallel this way? However, there is also a more idiomatic way to look for exceptions: call ThreadPoolExecutor#submit for all tasks, track every returned Future in a list, and then iterate through that list calling Future#get on each one. If any task encountered an exception, it would propagate as an ExecutionException, and then you can unwrap it to get the underlying exception that was thrown on the thread.

          I also see logic for handling RejectedExecutionException. In practice, I expect this won't happen, because the LinkedBlockingQueue will cause the caller submitting the task to block instead of failing if it runs out of capacity. If you're concerned about this, then another way to approach this is to look into ThreadPoolExecutor.CallerRunsPolicy, which would fall back to running the task in the main submitting thread.

          If there is some other reason I'm missing that requires the current unusual implementation of executeParallel, please let me know. Thank you.

          Show
          cnauroth Chris Nauroth added a comment - Subramanyam Pattipaka , in addition to the earlier code review feedback, I have a higher-level question about the implementation of the executeParallel method. Typical usage of a ThreadPoolExecutor is to configure it with a certain thread count and then submit multiple tasks, with each task performing a single isolated action. When tasks are submitted, they get added to an internal queue (the LinkedBlockingQueue in your code). Then, the number of threads all independently pull and execute tasks from the queue. The executeParallel method in your current patch does not follow this typical usage of ThreadPoolExecutor . Instead, it submits a number of tasks identical to the thread count, and the code in each task iterates through an array of work items ( FileMetadata[] ) tracked manually. The array is shared state across all of those tasks/threads, so safe access requires additional synchronization code. It appears the current executeParallel reimplements functionality already built into the ThreadPoolExecutor . Switching this code to more idiomatic usage of ThreadPoolExecutor would make it easier for maintainers to understand, and it would remove the need for extra cross-thread synchronization, which is always tricky to get right. For example, assuming 10 threads configured for fs.azure.delete.threads and a deletion of 100 blobs, I recommend using a ThreadPoolExecutor configured with 10 threads and submitting 100 task instances to it, each task deleting a single blob, with no shared state across the tasks/threads. I also see the current logic carefully tracks the last exception that happened on a thread. Maybe that's why you implemented executeParallel this way? However, there is also a more idiomatic way to look for exceptions: call ThreadPoolExecutor#submit for all tasks, track every returned Future in a list, and then iterate through that list calling Future#get on each one. If any task encountered an exception, it would propagate as an ExecutionException , and then you can unwrap it to get the underlying exception that was thrown on the thread. I also see logic for handling RejectedExecutionException . In practice, I expect this won't happen, because the LinkedBlockingQueue will cause the caller submitting the task to block instead of failing if it runs out of capacity. If you're concerned about this, then another way to approach this is to look into ThreadPoolExecutor.CallerRunsPolicy , which would fall back to running the task in the main submitting thread. If there is some other reason I'm missing that requires the current unusual implementation of executeParallel , please let me know. Thank you.
          Hide
          pattipaka Subramanyam Pattipaka added a comment -

          Chris Nauroth, thanks for your comments. I have incorporated your comments. I have fixed issues showed by findbugs and also fixed new errors shown with checkstyle.

          Regarding you questions on thread pools, I have removed null check for thread pool. I have retained other checks. getThreadPool may be called with very large value. It could be a naive mistake from user application. Hive script may pass this user configuration dynamically and the current thread may fail to allocate memory for thread pool. We don't want to fail the operation in this case. Instead we want to do serial processing which could go through and operation can be success. I have added unit tests to verify the behavior in case of any exception raised while creating thread pool.

          Not sure why you are seeing this assertion while running tests. I have modified these test cases as per your recommendation. If you still see the issue then I can take a look. I have verified that test cases are passing with latest changes (patch 003).

          Show
          pattipaka Subramanyam Pattipaka added a comment - Chris Nauroth , thanks for your comments. I have incorporated your comments. I have fixed issues showed by findbugs and also fixed new errors shown with checkstyle. Regarding you questions on thread pools, I have removed null check for thread pool. I have retained other checks. getThreadPool may be called with very large value. It could be a naive mistake from user application. Hive script may pass this user configuration dynamically and the current thread may fail to allocate memory for thread pool. We don't want to fail the operation in this case. Instead we want to do serial processing which could go through and operation can be success. I have added unit tests to verify the behavior in case of any exception raised while creating thread pool. Not sure why you are seeing this assertion while running tests. I have modified these test cases as per your recommendation. If you still see the issue then I can take a look. I have verified that test cases are passing with latest changes (patch 003).
          Hide
          pattipaka Subramanyam Pattipaka added a comment -

          Chris Nauroth,

          Regarding your question on synchronization and idiomatic usage of ThreadPoolExecutor

          Yes. But, it is very simple one and worth using it due to its benefits as explained below.
          1. The input data set is final and doesn't change. To ensure this, I have changed input parameter to be final. Further, we don't have a code paths which can change this input list.
          2. As the input array is final, each thread starts from index 0 and walks through till the end of the array. The synchronization is done through generating "next index" ATOMICALLY. This efficient way due to following reasons
          a) As the input set doesn't change, its overhead to convert array to synchronized queue. If number of files are in millions then having double memory is overkill.
          b) Each thread has to spend extra CPU to dequeue from synchronized queue which is costly compared to simple atomic increment.
          3. With above benefits, this method still ensures that work doesn't get delayed even if some threads got stuck due to some reasons.

          Regarding your question on use of futures

          The reason for tracking lastException or operationStatus is to ensure
          a) Failure of operation on single file means overall operation is failed anyway. No point in processing further by anyt threads. bail them out right away.
          b) In case if threads are still yet to be submitted then bail out there is well. A probable case in case we use large number of threads like 128 and the very first file processing encountered issue.

          Is there any way to achieve this through futures?

          Regarding your question on RejectedExecutionException,

          After thinking deep, may be this exception will not be raised as threadPool.shutdown happens on the same thread inline after submit calls are done. All submit requests must be accepted. But, having this code will ensure that operation is completed cleanly even if this exception occurs due to any reason. I have unit tests to validate the behavior as well. Please let me know if you find any issues in keeping this code.

          Show
          pattipaka Subramanyam Pattipaka added a comment - Chris Nauroth , Regarding your question on synchronization and idiomatic usage of ThreadPoolExecutor Yes. But, it is very simple one and worth using it due to its benefits as explained below. 1. The input data set is final and doesn't change. To ensure this, I have changed input parameter to be final. Further, we don't have a code paths which can change this input list. 2. As the input array is final, each thread starts from index 0 and walks through till the end of the array. The synchronization is done through generating "next index" ATOMICALLY. This efficient way due to following reasons a) As the input set doesn't change, its overhead to convert array to synchronized queue. If number of files are in millions then having double memory is overkill. b) Each thread has to spend extra CPU to dequeue from synchronized queue which is costly compared to simple atomic increment. 3. With above benefits, this method still ensures that work doesn't get delayed even if some threads got stuck due to some reasons. Regarding your question on use of futures The reason for tracking lastException or operationStatus is to ensure a) Failure of operation on single file means overall operation is failed anyway. No point in processing further by anyt threads. bail them out right away. b) In case if threads are still yet to be submitted then bail out there is well. A probable case in case we use large number of threads like 128 and the very first file processing encountered issue. Is there any way to achieve this through futures? Regarding your question on RejectedExecutionException, After thinking deep, may be this exception will not be raised as threadPool.shutdown happens on the same thread inline after submit calls are done. All submit requests must be accepted. But, having this code will ensure that operation is completed cleanly even if this exception occurs due to any reason. I have unit tests to validate the behavior as well. Please let me know if you find any issues in keeping this code.
          Hide
          pattipaka Subramanyam Pattipaka added a comment -

          Uploading latest patch after fixing review comments, findbugs and checkstyle errors.

          Show
          pattipaka Subramanyam Pattipaka added a comment - Uploading latest patch after fixing review comments, findbugs and checkstyle errors.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 33s trunk passed
          +1 compile 0m 16s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 19s trunk passed
          +1 mvneclipse 1m 34s trunk passed
          +1 findbugs 0m 25s trunk passed
          +1 javadoc 0m 13s trunk passed
          +1 mvninstall 0m 15s the patch passed
          +1 compile 0m 14s the patch passed
          +1 javac 0m 14s the patch passed
          -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 2 new + 43 unchanged - 1 fixed = 45 total (was 44)
          +1 mvnsite 0m 17s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 0m 29s the patch passed
          +1 javadoc 0m 10s the patch passed
          +1 unit 1m 19s hadoop-azure in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          14m 22s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821152/HADOOP-13403-003.patch
          JIRA Issue HADOOP-13403
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1d2583fec78d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8d32bd8
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10128/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10128/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10128/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10128/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 33s trunk passed +1 compile 0m 16s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 19s trunk passed +1 mvneclipse 1m 34s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 15s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 2 new + 43 unchanged - 1 fixed = 45 total (was 44) +1 mvnsite 0m 17s the patch passed +1 mvneclipse 0m 9s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 0m 29s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 1m 19s hadoop-azure in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 14m 22s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821152/HADOOP-13403-003.patch JIRA Issue HADOOP-13403 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1d2583fec78d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8d32bd8 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10128/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10128/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10128/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10128/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thank you for sharing patch 003.

          If the reason for the unusual executor logic is optimization, then I suggest adding more comments in the executeParallel JavaDocs to explain that. I'm not sure that the memory optimization argument is true for the delete code path, where it still does a conversion from ArrayList to array.

          Is there any way to achieve this through futures?

          If the code had followed idiomatic usage, then the typical solution is to call ThreadPoolExecutor#submit for each task, track every returned Future in a list, and then iterate through the list and call Future#get on each one. If any individual task threw an exception, then the call to Future#get would propagate that exception. Then, that would give you an opportunity to call ThreadPoolExecutor#shutdownNow to cancel or interrupt all remaining tasks. With the current logic though, I don't really see a way to adapt this pattern.

          Repeating an earlier comment, I don't see any exceptions thrown from getThreadPool, so coding exception handling around it and tests for it looks unnecessary. If you check validity of deleteThreadCount and renameThreadCount in initialize (e.g. check for values <= 0) and fail fast by throwing an exception during initialization, then even unchecked exceptions will be impossible during calls to getThreadPool.

          I still see numerous test failures in TestFileSystemOperationsWithThreads. For the next patch revision, would you please ensure all tests pass?

          Show
          cnauroth Chris Nauroth added a comment - Thank you for sharing patch 003. If the reason for the unusual executor logic is optimization, then I suggest adding more comments in the executeParallel JavaDocs to explain that. I'm not sure that the memory optimization argument is true for the delete code path, where it still does a conversion from ArrayList to array. Is there any way to achieve this through futures? If the code had followed idiomatic usage, then the typical solution is to call ThreadPoolExecutor#submit for each task, track every returned Future in a list, and then iterate through the list and call Future#get on each one. If any individual task threw an exception, then the call to Future#get would propagate that exception. Then, that would give you an opportunity to call ThreadPoolExecutor#shutdownNow to cancel or interrupt all remaining tasks. With the current logic though, I don't really see a way to adapt this pattern. Repeating an earlier comment, I don't see any exceptions thrown from getThreadPool , so coding exception handling around it and tests for it looks unnecessary. If you check validity of deleteThreadCount and renameThreadCount in initialize (e.g. check for values <= 0) and fail fast by throwing an exception during initialization, then even unchecked exceptions will be impossible during calls to getThreadPool . I still see numerous test failures in TestFileSystemOperationsWithThreads . For the next patch revision, would you please ensure all tests pass?
          Hide
          pattipaka Subramanyam Pattipaka added a comment -

          Chris Nauroth, Thanks for your comments.

          I will update with more comments on executeParallel and generate another patch.

          I had refactored code for both delete and rename operations to use single interface. Rename already has array and the array is already used at other locations. If we use ConcurrentLinkedQueue and remove contents from it then after executeParallel call, there won't be any entries in the queue. In future, if we need this array contents for reuse then we have to regenerate the list of files. If use array, we do the job with out loosing entries and can be useful for other cases in future.

          Regarding futures, I hope you agree to keep current pattern and not to use futures.

          Regarding getThreadPool, we are doing new operation. This can potentially result in OutOfMemoryException if we give very large value as input. This could especially happen even for little big number if the current thread has already reached the maximum heap size due to object object allocations like fileMeataData array[]. Even if we think of restricting this to a max value like 1024 then still OutOfmemory can cause with remote possibility. Currently, I couldn't think of other scenario, but don't want to realize it later which can make operation fail. Instead, for any kind of exceptions raised as part of new ThreadPoolExecutor() operation, we want to take serial path. I have already included checks for basic cases like check for threadCount > 1 after going through user configurations etc.. This is extra safety check on top of that.

          I ran tests and all of them are passing. Can you please provide details on what errors you are seeing?

          Show
          pattipaka Subramanyam Pattipaka added a comment - Chris Nauroth , Thanks for your comments. I will update with more comments on executeParallel and generate another patch. I had refactored code for both delete and rename operations to use single interface. Rename already has array and the array is already used at other locations. If we use ConcurrentLinkedQueue and remove contents from it then after executeParallel call, there won't be any entries in the queue. In future, if we need this array contents for reuse then we have to regenerate the list of files. If use array, we do the job with out loosing entries and can be useful for other cases in future. Regarding futures, I hope you agree to keep current pattern and not to use futures. Regarding getThreadPool, we are doing new operation. This can potentially result in OutOfMemoryException if we give very large value as input. This could especially happen even for little big number if the current thread has already reached the maximum heap size due to object object allocations like fileMeataData array[]. Even if we think of restricting this to a max value like 1024 then still OutOfmemory can cause with remote possibility. Currently, I couldn't think of other scenario, but don't want to realize it later which can make operation fail. Instead, for any kind of exceptions raised as part of new ThreadPoolExecutor() operation, we want to take serial path. I have already included checks for basic cases like check for threadCount > 1 after going through user configurations etc.. This is extra safety check on top of that. I ran tests and all of them are passing. Can you please provide details on what errors you are seeing?
          Hide
          pattipaka Subramanyam Pattipaka added a comment -

          Chris Nauroth, I have reproduced the test issue on Linux machines. On windows machines, these tests are passing. I will fix these tests on Linux machine in next patch.

          Show
          pattipaka Subramanyam Pattipaka added a comment - Chris Nauroth , I have reproduced the test issue on Linux machines. On windows machines, these tests are passing. I will fix these tests on Linux machine in next patch.
          Hide
          cnauroth Chris Nauroth added a comment -

          Yes, I agree about not using futures.

          If available memory is so constrained that the JVM can't run the ThreadPoolExecutor constructor, then it's really already a lost cause. At that point, the OutOfMemoryError could come from just about any line of code. Even if we manage to fall back to serial execution after that, we'll likely either get another OutOfMemoryError or the JVM will be unresponsive due to GC churn. OutOfMemoryError generally is not something recoverable, no matter how hard you try.

          I cannot think of any possible error condition from the getThreadPoolExecutor method that can be recovered reasonably. If you really think it's important to keep this, then please comment that this is defensive coding despite the fact that there are no possible known recoverable error conditions right now.

          Show
          cnauroth Chris Nauroth added a comment - Yes, I agree about not using futures. If available memory is so constrained that the JVM can't run the ThreadPoolExecutor constructor, then it's really already a lost cause. At that point, the OutOfMemoryError could come from just about any line of code. Even if we manage to fall back to serial execution after that, we'll likely either get another OutOfMemoryError or the JVM will be unresponsive due to GC churn. OutOfMemoryError generally is not something recoverable, no matter how hard you try. I cannot think of any possible error condition from the getThreadPoolExecutor method that can be recovered reasonably. If you really think it's important to keep this, then please comment that this is defensive coding despite the fact that there are no possible known recoverable error conditions right now.
          Hide
          pattipaka Subramanyam Pattipaka added a comment -

          Chris Nauroth, I have included comment. Also, I have fixed unit tests to run by any user. fixed white space issue as well.

          Show
          pattipaka Subramanyam Pattipaka added a comment - Chris Nauroth , I have included comment. Also, I have fixed unit tests to run by any user. fixed white space issue as well.
          Hide
          pattipaka Subramanyam Pattipaka added a comment -

          Fixed white space issues, unit test issues added details for executeParallel method.

          Show
          pattipaka Subramanyam Pattipaka added a comment - Fixed white space issues, unit test issues added details for executeParallel method.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 38s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 0s trunk passed
          +1 compile 0m 17s trunk passed
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 19s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 24s trunk passed
          +1 javadoc 0m 12s trunk passed
          +1 mvninstall 0m 15s the patch passed
          +1 compile 0m 14s the patch passed
          +1 javac 0m 14s the patch passed
          +1 checkstyle 0m 10s hadoop-tools/hadoop-azure: The patch generated 0 new + 43 unchanged - 1 fixed = 43 total (was 44)
          +1 mvnsite 0m 17s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 30s the patch passed
          +1 javadoc 0m 9s the patch passed
          +1 unit 1m 18s hadoop-azure in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          13m 50s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821543/HADOOP-13403-004.patch
          JIRA Issue HADOOP-13403
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e987e0228780 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a5fb298
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10150/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10150/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 38s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 0s trunk passed +1 compile 0m 17s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 19s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 24s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 15s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed +1 checkstyle 0m 10s hadoop-tools/hadoop-azure: The patch generated 0 new + 43 unchanged - 1 fixed = 43 total (was 44) +1 mvnsite 0m 17s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 30s the patch passed +1 javadoc 0m 9s the patch passed +1 unit 1m 18s hadoop-azure in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 13m 50s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821543/HADOOP-13403-004.patch JIRA Issue HADOOP-13403 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e987e0228780 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a5fb298 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10150/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10150/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Subramanyam Pattipaka, patch 004 looks good. Thank you for incorporating the feedback. My only other suggestion is to add a link to the new documentation section in the table of contents at the top of the index.md file.

          I'm also marking HADOOP-13459 as a pre-requisite for this patch. While testing, I noticed that hadoop-azure test runs were taking a really long time. The root cause is that we are accidentally re-running some of the test cases multiple times. See my patch on HADOOP-13459 for full details. For the next patch revision, please change TestFileSystemOperationsWithThreads to subclass AbstractWasbTestBase instead of NativeAzureFileSystemBaseTest. You can test locally by applying my HADOOP-13459 patch.

          I'm going to click Cancel Patch now, because your next revision of the patch won't work against trunk until after HADOOP-13459 gets committed. I'll click Submit Patch again after HADOOP-13459 gets committed.

          Show
          cnauroth Chris Nauroth added a comment - Subramanyam Pattipaka , patch 004 looks good. Thank you for incorporating the feedback. My only other suggestion is to add a link to the new documentation section in the table of contents at the top of the index.md file. I'm also marking HADOOP-13459 as a pre-requisite for this patch. While testing, I noticed that hadoop-azure test runs were taking a really long time. The root cause is that we are accidentally re-running some of the test cases multiple times. See my patch on HADOOP-13459 for full details. For the next patch revision, please change TestFileSystemOperationsWithThreads to subclass AbstractWasbTestBase instead of NativeAzureFileSystemBaseTest . You can test locally by applying my HADOOP-13459 patch. I'm going to click Cancel Patch now, because your next revision of the patch won't work against trunk until after HADOOP-13459 gets committed. I'll click Submit Patch again after HADOOP-13459 gets committed.
          Hide
          cnauroth Chris Nauroth added a comment -

          Subramanyam Pattipaka, I have committed the HADOOP-13459 patch. You can pull the latest trunk and pick up the AbstractWasbTestCase class to use in your patch.

          Show
          cnauroth Chris Nauroth added a comment - Subramanyam Pattipaka , I have committed the HADOOP-13459 patch. You can pull the latest trunk and pick up the AbstractWasbTestCase class to use in your patch.
          Hide
          pattipaka Subramanyam Pattipaka added a comment -

          New patch after using using new base class for tests. Also fixed your latest comments. I have observed now that new test case execution takes only 30 sec.

          Show
          pattipaka Subramanyam Pattipaka added a comment - New patch after using using new base class for tests. Also fixed your latest comments. I have observed now that new test case execution takes only 30 sec.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 23s trunk passed
          +1 compile 0m 18s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 20s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 25s trunk passed
          +1 javadoc 0m 13s trunk passed
          +1 mvninstall 0m 16s the patch passed
          +1 compile 0m 14s the patch passed
          +1 javac 0m 14s the patch passed
          +1 checkstyle 0m 10s hadoop-tools/hadoop-azure: The patch generated 0 new + 43 unchanged - 1 fixed = 43 total (was 44)
          +1 mvnsite 0m 17s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 30s the patch passed
          +1 javadoc 0m 10s the patch passed
          +1 unit 1m 18s hadoop-azure in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          14m 1s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822369/HADOOP-13403-005.patch
          JIRA Issue HADOOP-13403
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 700d6849b033 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 131d58a
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10198/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10198/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 23s trunk passed +1 compile 0m 18s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 16s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed +1 checkstyle 0m 10s hadoop-tools/hadoop-azure: The patch generated 0 new + 43 unchanged - 1 fixed = 43 total (was 44) +1 mvnsite 0m 17s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 30s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 1m 18s hadoop-azure in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 14m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822369/HADOOP-13403-005.patch JIRA Issue HADOOP-13403 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 700d6849b033 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 131d58a Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10198/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10198/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Subramanyam Pattipaka, patch 005 looks great. Thank you for the update.

          Unfortunately, it looks like we'll need one more patch revision. I'll want to commit this to both trunk and branch-2. The patch works fine on trunk, but the test code fails to compile when applied to branch-2. (See below.)

          A potential explanation for this difference is that we build trunk with Java 8 and branch-2 with Java 7. You might need to put explicit type bounds on the Mockito calls to fix this.

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hadoop-azure: Compilation failure: Compilation failure:
          [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[376,99] incompatible types: java.lang.Object cannot be converted to java.lang.Runnable
          [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[424,118] incompatible types: java.lang.Object cannot be converted to java.lang.Runnable
          [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[452,69] incompatible types: java.lang.Object cannot be converted to java.lang.Runnable
          [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[571,99] incompatible types: java.lang.Object cannot be converted to java.lang.Runnable
          [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[618,118] incompatible types: java.lang.Object cannot be converted to java.lang.Runnable
          [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[649,69] incompatible types: java.lang.Object cannot be converted to java.lang.Runnable
          
          Show
          cnauroth Chris Nauroth added a comment - Subramanyam Pattipaka , patch 005 looks great. Thank you for the update. Unfortunately, it looks like we'll need one more patch revision. I'll want to commit this to both trunk and branch-2. The patch works fine on trunk, but the test code fails to compile when applied to branch-2. (See below.) A potential explanation for this difference is that we build trunk with Java 8 and branch-2 with Java 7. You might need to put explicit type bounds on the Mockito calls to fix this. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile ( default -testCompile) on project hadoop-azure: Compilation failure: Compilation failure: [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[376,99] incompatible types: java.lang. Object cannot be converted to java.lang. Runnable [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[424,118] incompatible types: java.lang. Object cannot be converted to java.lang. Runnable [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[452,69] incompatible types: java.lang. Object cannot be converted to java.lang. Runnable [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[571,99] incompatible types: java.lang. Object cannot be converted to java.lang. Runnable [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[618,118] incompatible types: java.lang. Object cannot be converted to java.lang. Runnable [ERROR] /Users/chris/git/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java:[649,69] incompatible types: java.lang. Object cannot be converted to java.lang. Runnable
          Hide
          pattipaka Subramanyam Pattipaka added a comment -

          Chris Nauroth, Thanks. I will upload another path after fixing this issue.

          Show
          pattipaka Subramanyam Pattipaka added a comment - Chris Nauroth , Thanks. I will upload another path after fixing this issue.
          Hide
          pattipaka Subramanyam Pattipaka added a comment -

          Latest patch tested on both trunk and branch-2. Verified all tests are passing.

          Show
          pattipaka Subramanyam Pattipaka added a comment - Latest patch tested on both trunk and branch-2. Verified all tests are passing.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 9m 18s trunk passed
          +1 compile 0m 19s trunk passed
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 21s trunk passed
          +1 mvneclipse 1m 6s trunk passed
          +1 findbugs 0m 35s trunk passed
          +1 javadoc 0m 14s trunk passed
          +1 mvninstall 0m 19s the patch passed
          +1 compile 0m 19s the patch passed
          +1 javac 0m 19s the patch passed
          +1 checkstyle 0m 14s hadoop-tools/hadoop-azure: The patch generated 0 new + 43 unchanged - 1 fixed = 43 total (was 44)
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 39s the patch passed
          +1 javadoc 0m 11s the patch passed
          +1 unit 1m 29s hadoop-azure in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          17m 42s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822628/HADOOP-13403-006.patch
          JIRA Issue HADOOP-13403
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8be9946d46ca 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6255859
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10201/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10201/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 9m 18s trunk passed +1 compile 0m 19s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 21s trunk passed +1 mvneclipse 1m 6s trunk passed +1 findbugs 0m 35s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 19s the patch passed +1 compile 0m 19s the patch passed +1 javac 0m 19s the patch passed +1 checkstyle 0m 14s hadoop-tools/hadoop-azure: The patch generated 0 new + 43 unchanged - 1 fixed = 43 total (was 44) +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 1m 29s hadoop-azure in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 17m 42s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822628/HADOOP-13403-006.patch JIRA Issue HADOOP-13403 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8be9946d46ca 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6255859 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10201/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10201/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Subramanyam Pattipaka, thank you again for revising the patch. +1 for patch 006. I have committed this to trunk and branch-2.

          Show
          cnauroth Chris Nauroth added a comment - Subramanyam Pattipaka , thank you again for revising the patch. +1 for patch 006. I have committed this to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10236 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10236/)
          HADOOP-13403. AzureNativeFileSystem rename/delete performance (cnauroth: rev 2ed58c40e5dcbf5c5303c00e85096085b1055f85)

          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureFileSystemThreadPoolExecutor.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureFileSystemThreadTask.java
          • hadoop-tools/hadoop-azure/src/site/markdown/index.md
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          • hadoop-tools/hadoop-azure/src/test/resources/log4j.properties
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10236 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10236/ ) HADOOP-13403 . AzureNativeFileSystem rename/delete performance (cnauroth: rev 2ed58c40e5dcbf5c5303c00e85096085b1055f85) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureFileSystemThreadPoolExecutor.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsWithThreads.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureFileSystemThreadTask.java hadoop-tools/hadoop-azure/src/site/markdown/index.md hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java hadoop-tools/hadoop-azure/src/test/resources/log4j.properties

            People

            • Assignee:
              pattipaka Subramanyam Pattipaka
              Reporter:
              pattipaka Subramanyam Pattipaka
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development