Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: tools
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      AzureStorageExceptions currently are logged as part of the WAB code which often is not too informative. AzureStorage SDK supports client side logging that can be enabled that logs relevant information w.r.t request made from the Storage client.

      This JIRA is created to enable Azure Storage Client Side logging at the Job submission level. User should be able to configure Client Side logging on a Per Job bases.

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2144 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2144/)
        HADOOP-11960. Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed)

        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
        • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2144 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2144/ ) HADOOP-11960 . Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #196 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/196/)
        HADOOP-11960. Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed)

        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #196 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/196/ ) HADOOP-11960 . Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #186 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/186/)
        HADOOP-11960. Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
        • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #186 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/186/ ) HADOOP-11960 . Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2126 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2126/)
        HADOOP-11960. Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed)

        • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java
        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2126 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2126/ ) HADOOP-11960 . Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #197 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/197/)
        HADOOP-11960. Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
        • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #197 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/197/ ) HADOOP-11960 . Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #928 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/928/)
        HADOOP-11960. Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed)

        • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java
        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #928 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/928/ ) HADOOP-11960 . Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7839 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7839/)
        HADOOP-11960. Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed)

        • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java
        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7839 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7839/ ) HADOOP-11960 . Enable Azure-Storage Client Side logging. Contributed by Dushyanth. (cnauroth: rev cb8e69a80cecb95abdfc93a787bea0bedef275ed) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        cnauroth Chris Nauroth added a comment -

        +1 for the patch. I committed this to trunk and branch-2. Dushyanth, thank you for contributing the patch.

        Show
        cnauroth Chris Nauroth added a comment - +1 for the patch. I committed this to trunk and branch-2. Dushyanth, thank you for contributing the patch.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 40s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 29s There were no new javac warning messages.
        +1 javadoc 9m 36s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 24s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 0m 39s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 1m 9s Tests passed in hadoop-azure.
            36m 30s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12733012/HADOOP-11960.002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 9a2a955
        hadoop-azure test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6698/artifact/patchprocess/testrun_hadoop-azure.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6698/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6698/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 24s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 39s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 1m 9s Tests passed in hadoop-azure.     36m 30s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733012/HADOOP-11960.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9a2a955 hadoop-azure test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6698/artifact/patchprocess/testrun_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6698/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6698/console This message was automatically generated.
        Hide
        dchickabasapa Dushyanth added a comment -

        Addressed Code review feedback and attached new patch with fixes.

        Show
        dchickabasapa Dushyanth added a comment - Addressed Code review feedback and attached new patch with fixes.
        Hide
        dchickabasapa Dushyanth added a comment -

        Chris Nauroth

        Thanks very much for the review. I have address your comments and attached a new patch with the fixes.

        HADOOP-11960.002.patch

        Show
        dchickabasapa Dushyanth added a comment - Chris Nauroth Thanks very much for the review. I have address your comments and attached a new patch with the fixes. HADOOP-11960 .002.patch
        Hide
        cnauroth Chris Nauroth added a comment -

        Hello Dushyanth. This patch is looking pretty good. I verified the tests running against the live service.

        Here are a few minor comments.

        1. Please add the standard Apache license header to the top of TestNativeAzureFileSystemClientLogging. That will fix the release audit warning from the last Jenkins run.
        2. To satisfy checkstyle, please change this...
              OperationContext.setLoggingEnabledByDefault(sessionConfiguration.getBoolean
                      (KEY_ENABLE_STORAGE_CLIENT_LOGGING, false));    
          

          ...to this...

              OperationContext.setLoggingEnabledByDefault(sessionConfiguration.getBoolean(
                      KEY_ENABLE_STORAGE_CLIENT_LOGGING, false));    
          
        3. Please try to remove trailing whitespace before uploading the next patch. If there is a problem, I'll handle it at commit time using git apply --whitespace=fix.

        For future patches, please take a look at the section of the contribution wiki about naming standards for patch files.
        https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch

        Show
        cnauroth Chris Nauroth added a comment - Hello Dushyanth . This patch is looking pretty good. I verified the tests running against the live service. Here are a few minor comments. Please add the standard Apache license header to the top of TestNativeAzureFileSystemClientLogging . That will fix the release audit warning from the last Jenkins run. To satisfy checkstyle, please change this... OperationContext.setLoggingEnabledByDefault(sessionConfiguration.getBoolean (KEY_ENABLE_STORAGE_CLIENT_LOGGING, false )); ...to this... OperationContext.setLoggingEnabledByDefault(sessionConfiguration.getBoolean( KEY_ENABLE_STORAGE_CLIENT_LOGGING, false )); Please try to remove trailing whitespace before uploading the next patch. If there is a problem, I'll handle it at commit time using git apply --whitespace=fix . For future patches, please take a look at the section of the contribution wiki about naming standards for patch files. https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 15m 18s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 46s There were no new javac warning messages.
        +1 javadoc 9m 55s There were no new javadoc warning messages.
        -1 release audit 0m 22s The applied patch generated 1 release audit warnings.
        -1 checkstyle 0m 22s The applied patch generated 1 new checkstyle issues (total was 34, now 35).
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 38s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 0m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 tools/hadoop tests 1m 9s Tests passed in hadoop-azure.
            37m 48s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12732364/0001-HADOOP-945-Enabling-WASB-Azure-Storage-Client-side-l.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / f24452d
        Release Audit https://builds.apache.org/job/PreCommit-HADOOP-Build/6673/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6673/artifact/patchprocess/diffcheckstylehadoop-azure.txt
        whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6673/artifact/patchprocess/whitespace.txt
        hadoop-azure test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6673/artifact/patchprocess/testrun_hadoop-azure.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6673/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6673/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 18s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 46s There were no new javac warning messages. +1 javadoc 9m 55s There were no new javadoc warning messages. -1 release audit 0m 22s The applied patch generated 1 release audit warnings. -1 checkstyle 0m 22s The applied patch generated 1 new checkstyle issues (total was 34, now 35). -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 0m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 1m 9s Tests passed in hadoop-azure.     37m 48s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732364/0001-HADOOP-945-Enabling-WASB-Azure-Storage-Client-side-l.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f24452d Release Audit https://builds.apache.org/job/PreCommit-HADOOP-Build/6673/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6673/artifact/patchprocess/diffcheckstylehadoop-azure.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6673/artifact/patchprocess/whitespace.txt hadoop-azure test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6673/artifact/patchprocess/testrun_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6673/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6673/console This message was automatically generated.
        Hide
        dchickabasapa Dushyanth added a comment -

        Patch Attached for review.

        Show
        dchickabasapa Dushyanth added a comment - Patch Attached for review.
        Hide
        dchickabasapa Dushyanth added a comment -

        Adding first review patch to enable Azure Storage client-side logging. The core-site config "fs.azure.storage.client.logging" acts as a knob to control logging. The default behavior is set to "False" disabling client side logging.

        Show
        dchickabasapa Dushyanth added a comment - Adding first review patch to enable Azure Storage client-side logging. The core-site config "fs.azure.storage.client.logging" acts as a knob to control logging. The default behavior is set to "False" disabling client side logging.

          People

          • Assignee:
            dchickabasapa Dushyanth
            Reporter:
            dchickabasapa Dushyanth
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development