Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14640

Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: fs/azure
    • Labels:

      Description

      Currently, WasbRemoteCallHelper can be configured to talk to comma separated list of URLs for authorization, SASKey generation and delegation token generation.
      To improve the performance, if service runs on the local machine, give it first preference over the other configured list of URLs.
      Currently, WasbRemoteCallHelper generates hadoop.auth cookie for every request by talking to the remote service, before making actual rest requests.
      The proposal is to reuse the hadoop.auth cookie for subsequent requests from same WasbRemoteCallHelper object until its expiry time.

      1. HADOOP-14640.1.patch
        21 kB
        Santhosh G Nayak
      2. HADOOP-14640.2.patch
        27 kB
        Santhosh G Nayak
      3. HADOOP-14640.3.patch
        28 kB
        Santhosh G Nayak
      4. HADOOP-14640.4.patch
        29 kB
        Santhosh G Nayak
      5. HADOOP-14640-branch-2.01.patch
        30 kB
        Jitendra Nath Pandey

        Activity

        Hide
        snayak Santhosh G Nayak added a comment -

        Attaching initial version of the patch containing proposed changes.

        • Introduced a new configuration fs.azure.enable.spnego.token.cache to enable/disable SPNEGO token cache per WasbRemoteCallHelper object.
        • If localhost URL is specified in the list of configured URLs, converting that into actual fully qualified actual hostname for Kerberos to work.
        Show
        snayak Santhosh G Nayak added a comment - Attaching initial version of the patch containing proposed changes. Introduced a new configuration fs.azure.enable.spnego.token.cache to enable/disable SPNEGO token cache per WasbRemoteCallHelper object. If localhost URL is specified in the list of configured URLs, converting that into actual fully qualified actual hostname for Kerberos to work.
        Hide
        jnp Jitendra Nath Pandey added a comment -

        Santhosh G Nayak, Thanks for the patch. A few comments:

        The AuthenthicationURL.Token doesn't provide any interface to determine validity or expiry time, and I believe that is the reason you are parsing out the expiry time from token string. It might be simpler if the SpnegoToken cache tracks its own cache-expiry time, set at the time of creation. We could typically configure it to be smaller than token expiry time. Keeping the token opaque at the client is a useful property.

        The check for expiry is expiryTime > System.currentTimeMillis() + 1000 * 60 * 5L. Are you adding 5 minutes just to guarantee that token is always accepted when client thinks it is valid? If that is the case, I think, it might be better to have a re-try where token is re-fetched if call fails due to token expiry.

        Minor:
        Checkstyle in a few place:
        1) Lines longer than 80 characters.
        2) Indentation where index of local url is calculated.

        Show
        jnp Jitendra Nath Pandey added a comment - Santhosh G Nayak , Thanks for the patch. A few comments: The AuthenthicationURL.Token doesn't provide any interface to determine validity or expiry time, and I believe that is the reason you are parsing out the expiry time from token string. It might be simpler if the SpnegoToken cache tracks its own cache-expiry time, set at the time of creation. We could typically configure it to be smaller than token expiry time. Keeping the token opaque at the client is a useful property. The check for expiry is expiryTime > System.currentTimeMillis() + 1000 * 60 * 5L . Are you adding 5 minutes just to guarantee that token is always accepted when client thinks it is valid? If that is the case, I think, it might be better to have a re-try where token is re-fetched if call fails due to token expiry. Minor: Checkstyle in a few place: 1) Lines longer than 80 characters. 2) Indentation where index of local url is calculated.
        Hide
        snayak Santhosh G Nayak added a comment - - edited

        Thanks Jitendra Nath Pandey for reviewing the patch.

        Yes. AuthenticatedURL.Token does not provide the expiry time of the token. So, introduced SpnegoToken class to represent SPNEGO token, which provides information about the validity of the token along with the token itself.
        Token used in hadoop.auth cookie follows SPNEGO standard and will have expiry time (represented as e) in it. So, I think it will be better, if we can reuse that information, instead of setting different expiry time on our end and treating it opaque.

        Regarding expiryTime > System.currentTimeMillis() + 1000 * 60 * 5L, adding 5 minutes buffer to proactively get new token to save additional network calls of requests which can definitely fail because of the token expiry and to eliminate any issues which can arise because of time difference between machines (if any).

        I have attached another patch addressing the checkstyle related comments and a bug fix where impersonation context is getting lost in SecureWasbRemoteCallHelper.getHttpRequest() as it is being invoked inside ugi.doAs().

        Tested with Azure South India Storage endpoint.

        PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn clean package
        [INFO] Scanning for projects...
        [INFO]
        [INFO] ------------------------------------------------------------------------
        [INFO] Building Apache Hadoop Azure support 3.0.0-beta1-SNAPSHOT
        [INFO] ------------------------------------------------------------------------
        [INFO]
        [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-azure ---
        [INFO] Deleting D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target
        [INFO] Deleting D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure (includes = [dependency-reduced-pom.xml], excludes = [])
        [INFO]
        [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-azure ---
        [INFO] Executing tasks
        
        main:
            [mkdir] Created dir: D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\test-dir
        [INFO] Executed tasks
        [INFO]
        [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-azure ---
        [INFO]
        [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hadoop-azure ---
        [INFO] Using 'UTF-8' encoding to copy filtered resources.
        [INFO] Copying 3 resources
        [INFO] Copying 2 resources
        [INFO]
        [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-azure ---
        [INFO] Compiling 57 source files to D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\classes
        [WARNING] /D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java: Some input files use or override a deprecated API.
        [WARNING] /D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java: Recompile with -Xlint:deprecation for details.
        [WARNING] /D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java: D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\src\main\java\org\apache\hadoop\fs\azure\security\TokenUtils.java uses unchecked or unsafe operations.
        [WARNING] /D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java: Recompile with -Xlint:unchecked for details.
        [INFO]
        [INFO] --- maven-dependency-plugin:2.10:list (deplist) @ hadoop-azure ---
        [INFO]
        [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hadoop-azure ---
        [INFO] Using 'UTF-8' encoding to copy filtered resources.
        [INFO] Copying 4 resources
        [INFO] Copying 2 resources
        [INFO]
        [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hadoop-azure ---
        [INFO] Compiling 58 source files to D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\test-classes
        [WARNING] /D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java: Some input files use or override a deprecated API.
        [WARNING] /D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java: Recompile with -Xlint:deprecation for details.
        [INFO]
        [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-azure ---
        [INFO] Surefire report directory: D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\surefire-reports
        
        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        
        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
        Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.525 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
        Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
        Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 0.467 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
        Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
        Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.386 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
        Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
        Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.386 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
        Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
        Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.455 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
        Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
        Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 0.386 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
        Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
        Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.401 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
        Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
        Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.35 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
        Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
        Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.544 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
        Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
        Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.093 sec - in org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
        Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.539 sec - in org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
        Running org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.031 sec - in org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
        Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.268 sec - in org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
        Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.328 sec - in org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
        Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.871 sec - in org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
        Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
        Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.568 sec - in org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
        Running org.apache.hadoop.fs.azure.TestBlobDataValidation
        Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.077 sec - in org.apache.hadoop.fs.azure.TestBlobDataValidation
        Running org.apache.hadoop.fs.azure.TestBlobMetadata
        Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.183 sec - in org.apache.hadoop.fs.azure.TestBlobMetadata
        Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.243 sec - in org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
        Running org.apache.hadoop.fs.azure.TestContainerChecks
        Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.482 sec - in org.apache.hadoop.fs.azure.TestContainerChecks
        Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
        Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.004 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
        Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage
        Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 398.648 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage
        Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
        Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.549 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
        Running org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
        Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 450.974 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
        Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 111.067 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.359 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
        Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.088 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
        Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.393 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
        Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.202 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.578 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.219 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.898 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
        Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 0.817 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
        Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 131.049 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
        Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.17 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
        Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 149.315 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
        Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.143 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
        Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 408.38 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
        Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.866 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
        Tests run: 50, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.472 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
        Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.147 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
        Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
        Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 422.47 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
        Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
        Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.373 sec - in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
        Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
        Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.348 sec - in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
        Running org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
        Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 127.899 sec - in org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
        Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.369 sec - in org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
        Running org.apache.hadoop.fs.azure.TestWasbFsck
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 1.102 sec - in org.apache.hadoop.fs.azure.TestWasbFsck
        Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
        Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 162.873 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
        Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
        Tests run: 18, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 7.954 sec - in org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
        
        Results :
        
        Tests run: 707, Failures: 0, Errors: 0, Skipped: 152
        
        [INFO]
        [INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-azure ---
        [INFO] Building jar: D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\hadoop-azure-3.0.0-beta1-SNAPSHOT.jar
        [INFO]
        [INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ hadoop-azure ---
        [INFO]
        [INFO] --- maven-jar-plugin:2.5:test-jar (default) @ hadoop-azure ---
        [INFO] Building jar: D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\hadoop-azure-3.0.0-beta1-SNAPSHOT-tests.jar
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD SUCCESS
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 46:44.589s
        [INFO] Finished at: Thu Jul 13 20:47:17 IST 2017
        [INFO] Final Memory: 35M/514M
        [INFO] ------------------------------------------------------------------------
        
        Show
        snayak Santhosh G Nayak added a comment - - edited Thanks Jitendra Nath Pandey for reviewing the patch. Yes. AuthenticatedURL.Token does not provide the expiry time of the token. So, introduced SpnegoToken class to represent SPNEGO token, which provides information about the validity of the token along with the token itself. Token used in hadoop.auth cookie follows SPNEGO standard and will have expiry time (represented as e ) in it. So, I think it will be better, if we can reuse that information, instead of setting different expiry time on our end and treating it opaque. Regarding expiryTime > System.currentTimeMillis() + 1000 * 60 * 5L , adding 5 minutes buffer to proactively get new token to save additional network calls of requests which can definitely fail because of the token expiry and to eliminate any issues which can arise because of time difference between machines (if any). I have attached another patch addressing the checkstyle related comments and a bug fix where impersonation context is getting lost in SecureWasbRemoteCallHelper.getHttpRequest() as it is being invoked inside ugi.doAs() . Tested with Azure South India Storage endpoint. PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn clean package [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Apache Hadoop Azure support 3.0.0-beta1-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean ( default -clean) @ hadoop-azure --- [INFO] Deleting D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target [INFO] Deleting D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure (includes = [dependency-reduced-pom.xml], excludes = []) [INFO] [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-azure --- [INFO] Executing tasks main: [mkdir] Created dir: D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\test-dir [INFO] Executed tasks [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process ( default ) @ hadoop-azure --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources ( default -resources) @ hadoop-azure --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 3 resources [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:compile ( default -compile) @ hadoop-azure --- [INFO] Compiling 57 source files to D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\classes [WARNING] /D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java: Some input files use or override a deprecated API. [WARNING] /D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java: Recompile with -Xlint:deprecation for details. [WARNING] /D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java: D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\src\main\java\org\apache\hadoop\fs\azure\security\TokenUtils.java uses unchecked or unsafe operations. [WARNING] /D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-dependency-plugin:2.10:list (deplist) @ hadoop-azure --- [INFO] [INFO] --- maven-resources-plugin:2.6:testResources ( default -testResources) @ hadoop-azure --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 4 resources [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile ( default -testCompile) @ hadoop-azure --- [INFO] Compiling 58 source files to D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\test-classes [WARNING] /D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java: Some input files use or override a deprecated API. [WARNING] /D:/code/hadoop1/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java: Recompile with -Xlint:deprecation for details. [INFO] [INFO] --- maven-surefire-plugin:2.17:test ( default -test) @ hadoop-azure --- [INFO] Surefire report directory: D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\surefire-reports ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.525 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 0.467 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.386 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.386 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.455 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 0.386 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.401 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.35 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.544 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.093 sec - in org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.539 sec - in org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater Running org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.031 sec - in org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.268 sec - in org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.328 sec - in org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.871 sec - in org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.568 sec - in org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions Running org.apache.hadoop.fs.azure.TestBlobDataValidation Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.077 sec - in org.apache.hadoop.fs.azure.TestBlobDataValidation Running org.apache.hadoop.fs.azure.TestBlobMetadata Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.183 sec - in org.apache.hadoop.fs.azure.TestBlobMetadata Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.243 sec - in org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference Running org.apache.hadoop.fs.azure.TestContainerChecks Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.482 sec - in org.apache.hadoop.fs.azure.TestContainerChecks Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.004 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 398.648 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.549 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Running org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 450.974 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 111.067 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.359 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.088 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.393 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.202 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.578 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.219 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.898 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 0.817 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 131.049 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.17 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 149.315 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.143 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 408.38 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.866 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked Tests run: 50, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.472 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.147 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 422.47 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.373 sec - in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.348 sec - in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive Running org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 127.899 sec - in org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.369 sec - in org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider Running org.apache.hadoop.fs.azure.TestWasbFsck Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 1.102 sec - in org.apache.hadoop.fs.azure.TestWasbFsck Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 162.873 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration Tests run: 18, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 7.954 sec - in org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration Results : Tests run: 707, Failures: 0, Errors: 0, Skipped: 152 [INFO] [INFO] --- maven-jar-plugin:2.5:jar ( default -jar) @ hadoop-azure --- [INFO] Building jar: D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\hadoop-azure-3.0.0-beta1-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ hadoop-azure --- [INFO] [INFO] --- maven-jar-plugin:2.5:test-jar ( default ) @ hadoop-azure --- [INFO] Building jar: D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure\target\hadoop-azure-3.0.0-beta1-SNAPSHOT-tests.jar [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 46:44.589s [INFO] Finished at: Thu Jul 13 20:47:17 IST 2017 [INFO] Final Memory: 35M/514M [INFO] ------------------------------------------------------------------------
        Hide
        jnp Jitendra Nath Pandey added a comment -

        Santhosh G Nayak, the only concern I have is that token parsing makes this code tightly coupled with the token format. In other words, if the token format changes this code will break, and, it is not easy to have client and server upgrade at the same time. Effectively, any token format change will become an incompatible change and hence unacceptable in a minor release.
        Do you believe, in this case token format is unlikely to change until next major release?

        As you rightly pointed out, that because of time differences in machines there is a possibility of using expired tokens, even with proactivity of 5 mins. Therefore, a retry is desirable, but that can be done as an improvement in a later jira.

        Thanks for fixing other issues.

        Show
        jnp Jitendra Nath Pandey added a comment - Santhosh G Nayak , the only concern I have is that token parsing makes this code tightly coupled with the token format. In other words, if the token format changes this code will break, and, it is not easy to have client and server upgrade at the same time. Effectively, any token format change will become an incompatible change and hence unacceptable in a minor release. Do you believe, in this case token format is unlikely to change until next major release? As you rightly pointed out, that because of time differences in machines there is a possibility of using expired tokens, even with proactivity of 5 mins. Therefore, a retry is desirable, but that can be done as an improvement in a later jira. Thanks for fixing other issues.
        Hide
        snayak Santhosh G Nayak added a comment - - edited

        Thanks Jitendra Nath Pandey.

        I agree that token parsing makes code tightly coupled with the token format, which could result in parsing errors when token format changes. It is better to treat token as opaque as you suggested.

        Attaching v3 version of the patch containing following changes -

        • Token is treated as opaque and client sets the expiry time to be 60 mins.
        • If request fails because of the expired or invalid token, client retries the request after re-fetching the token.
        Show
        snayak Santhosh G Nayak added a comment - - edited Thanks Jitendra Nath Pandey . I agree that token parsing makes code tightly coupled with the token format, which could result in parsing errors when token format changes. It is better to treat token as opaque as you suggested. Attaching v3 version of the patch containing following changes - Token is treated as opaque and client sets the expiry time to be 60 mins. If request fails because of the expired or invalid token, client retries the request after re-fetching the token.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 10s trunk passed
        +1 compile 0m 19s trunk passed
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 21s trunk passed
        +1 findbugs 0m 27s trunk passed
        +1 javadoc 0m 14s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 17s the patch passed
        +1 compile 0m 16s the patch passed
        +1 javac 0m 16s the patch passed
        -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
        +1 mvnsite 0m 18s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 0m 33s hadoop-tools/hadoop-azure generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
        +1 javadoc 0m 11s the patch passed
              Other Tests
        +1 unit 1m 22s hadoop-azure in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        19m 36s



        Reason Tests
        FindBugs module:hadoop-tools/hadoop-azure
          Inconsistent synchronization of org.apache.hadoop.fs.azure.SecureWasbRemoteCallHelper.delegationToken; locked 71% of time Unsynchronized access at SecureWasbRemoteCallHelper.java:71% of time Unsynchronized access at SecureWasbRemoteCallHelper.java:[line 167]
          Possible null pointer dereference of statusLine in org.apache.hadoop.fs.azure.WasbRemoteCallHelper.retryableRequest(String[], String, List, String) Dereferenced at WasbRemoteCallHelper.java:statusLine in org.apache.hadoop.fs.azure.WasbRemoteCallHelper.retryableRequest(String[], String, List, String) Dereferenced at WasbRemoteCallHelper.java:[line 156]
          Nullcheck of statusLine at line 156 of value previously dereferenced in org.apache.hadoop.fs.azure.WasbRemoteCallHelper.retryableRequest(String[], String, List, String) At WasbRemoteCallHelper.java:156 of value previously dereferenced in org.apache.hadoop.fs.azure.WasbRemoteCallHelper.retryableRequest(String[], String, List, String) At WasbRemoteCallHelper.java:[line 156]



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14640
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877281/HADOOP-14640.3.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b8b803491a79 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f413ee3
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12788/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
        findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12788/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-azure.html
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12788/testReport/
        modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12788/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 10s trunk passed +1 compile 0m 19s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 21s trunk passed +1 findbugs 0m 27s trunk passed +1 javadoc 0m 14s trunk passed       Patch Compile Tests +1 mvninstall 0m 17s the patch passed +1 compile 0m 16s the patch passed +1 javac 0m 16s the patch passed -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 mvnsite 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 33s hadoop-tools/hadoop-azure generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) +1 javadoc 0m 11s the patch passed       Other Tests +1 unit 1m 22s hadoop-azure in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 19m 36s Reason Tests FindBugs module:hadoop-tools/hadoop-azure   Inconsistent synchronization of org.apache.hadoop.fs.azure.SecureWasbRemoteCallHelper.delegationToken; locked 71% of time Unsynchronized access at SecureWasbRemoteCallHelper.java:71% of time Unsynchronized access at SecureWasbRemoteCallHelper.java: [line 167]   Possible null pointer dereference of statusLine in org.apache.hadoop.fs.azure.WasbRemoteCallHelper.retryableRequest(String[], String, List, String) Dereferenced at WasbRemoteCallHelper.java:statusLine in org.apache.hadoop.fs.azure.WasbRemoteCallHelper.retryableRequest(String[], String, List, String) Dereferenced at WasbRemoteCallHelper.java: [line 156]   Nullcheck of statusLine at line 156 of value previously dereferenced in org.apache.hadoop.fs.azure.WasbRemoteCallHelper.retryableRequest(String[], String, List, String) At WasbRemoteCallHelper.java:156 of value previously dereferenced in org.apache.hadoop.fs.azure.WasbRemoteCallHelper.retryableRequest(String[], String, List, String) At WasbRemoteCallHelper.java: [line 156] Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14640 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877281/HADOOP-14640.3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b8b803491a79 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f413ee3 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12788/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12788/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-azure.html Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12788/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12788/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jnp Jitendra Nath Pandey added a comment - - edited

        This seems like a good catch by findbugs. It will cause NPE if statusLine is null.

                if (statusLine == null
                    || statusLine.getStatusCode() != HttpStatus.SC_OK) {
                  requiresNewAuth =
                      (statusLine.getStatusCode() == HttpStatus.SC_UNAUTHORIZED);
        
        Show
        jnp Jitendra Nath Pandey added a comment - - edited This seems like a good catch by findbugs. It will cause NPE if statusLine is null. if (statusLine == null || statusLine.getStatusCode() != HttpStatus.SC_OK) { requiresNewAuth = (statusLine.getStatusCode() == HttpStatus.SC_UNAUTHORIZED);
        Hide
        jnp Jitendra Nath Pandey added a comment -
        • The other findbug issue for delegationToken is also worth looking at. Why does makeRemoteRequest tries to lookup a delegationToken, but getHttpRequest doesn't ?
        • I see a change to format of License Header. In some azure files the license link is wrapped inside <p>..<p>. Lets be consistent with rest of hadoop. For other files which are already committed, we can address it in a follow up jira.
        Show
        jnp Jitendra Nath Pandey added a comment - The other findbug issue for delegationToken is also worth looking at. Why does makeRemoteRequest tries to lookup a delegationToken, but getHttpRequest doesn't ? I see a change to format of License Header. In some azure files the license link is wrapped inside <p>..<p> . Lets be consistent with rest of hadoop. For other files which are already committed, we can address it in a follow up jira.
        Hide
        snayak Santhosh G Nayak added a comment -

        Thanks Jitendra Nath Pandey.
        Attached v4 version of the patch containing findbugs, checkstyle and licence header fixes.

        Show
        snayak Santhosh G Nayak added a comment - Thanks Jitendra Nath Pandey . Attached v4 version of the patch containing findbugs, checkstyle and licence header fixes.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 25s trunk passed
        +1 compile 0m 19s trunk passed
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 21s trunk passed
        +1 findbugs 0m 33s trunk passed
        +1 javadoc 0m 14s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 17s the patch passed
        +1 javac 0m 17s the patch passed
        +1 checkstyle 0m 11s the patch passed
        +1 mvnsite 0m 20s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 42s the patch passed
        +1 javadoc 0m 12s the patch passed
              Other Tests
        +1 unit 1m 32s hadoop-azure in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        20m 28s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14640
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877478/HADOOP-14640.4.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a46ad9e6d702 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b778887
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12795/testReport/
        modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12795/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 25s trunk passed +1 compile 0m 19s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 21s trunk passed +1 findbugs 0m 33s trunk passed +1 javadoc 0m 14s trunk passed       Patch Compile Tests +1 mvninstall 0m 22s the patch passed +1 compile 0m 17s the patch passed +1 javac 0m 17s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 42s the patch passed +1 javadoc 0m 12s the patch passed       Other Tests +1 unit 1m 32s hadoop-azure in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 20m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14640 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877478/HADOOP-14640.4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a46ad9e6d702 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b778887 Default Java 1.8.0_131 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12795/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12795/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jnp Jitendra Nath Pandey added a comment -

        +1 The latest patch looks good to me.
        Thanks for addressing the comments Santhosh G Nayak.

        Show
        jnp Jitendra Nath Pandey added a comment - +1 The latest patch looks good to me. Thanks for addressing the comments Santhosh G Nayak .
        Hide
        jnp Jitendra Nath Pandey added a comment -

        I have committed this to trunk. Keeping the jira open for commit to branch-2.

        Show
        jnp Jitendra Nath Pandey added a comment - I have committed this to trunk. Keeping the jira open for commit to branch-2.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12020 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12020/)
        HADOOP-14640. Azure: Support affinity for service running on localhost (jitendra: rev b0e78ae085928c82ae63a05a29a628c2e289c0fc)

        • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteWasbAuthorizerImpl.java
        • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/Constants.java
        • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureWasbRemoteCallHelper.java
        • (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestWasbRemoteCallHelper.java
        • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/RemoteWasbDelegationTokenManager.java
        • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/SpnegoToken.java
        • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbRemoteCallHelper.java
        • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteSASKeyGeneratorImpl.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12020 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12020/ ) HADOOP-14640 . Azure: Support affinity for service running on localhost (jitendra: rev b0e78ae085928c82ae63a05a29a628c2e289c0fc) (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteWasbAuthorizerImpl.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/Constants.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureWasbRemoteCallHelper.java (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestWasbRemoteCallHelper.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/RemoteWasbDelegationTokenManager.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/SpnegoToken.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbRemoteCallHelper.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteSASKeyGeneratorImpl.java
        Hide
        jnp Jitendra Nath Pandey added a comment -

        The patch applies cleanly to branch-2. Attaching.

        Show
        jnp Jitendra Nath Pandey added a comment - The patch applies cleanly to branch-2. Attaching.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2 Compile Tests
        +1 mvninstall 9m 58s branch-2 passed
        +1 compile 0m 16s branch-2 passed with JDK v1.8.0_131
        +1 compile 0m 19s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 16s branch-2 passed
        +1 mvnsite 0m 25s branch-2 passed
        +1 findbugs 0m 37s branch-2 passed
        +1 javadoc 0m 15s branch-2 passed with JDK v1.8.0_131
        +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_131
              Patch Compile Tests
        +1 mvninstall 0m 18s the patch passed
        +1 compile 0m 14s the patch passed with JDK v1.8.0_131
        +1 javac 0m 14s the patch passed
        +1 compile 0m 16s the patch passed with JDK v1.7.0_131
        +1 javac 0m 16s the patch passed
        +1 checkstyle 0m 12s the patch passed
        +1 mvnsite 0m 23s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 46s the patch passed
        +1 javadoc 0m 12s the patch passed with JDK v1.8.0_131
        +1 javadoc 0m 13s the patch passed with JDK v1.7.0_131
              Other Tests
        +1 unit 1m 35s hadoop-azure in the patch passed with JDK v1.7.0_131.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        20m 12s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue HADOOP-14640
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877701/HADOOP-14640-branch-2.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1d042625b838 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 02627f8c
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12805/testReport/
        modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12805/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2 Compile Tests +1 mvninstall 9m 58s branch-2 passed +1 compile 0m 16s branch-2 passed with JDK v1.8.0_131 +1 compile 0m 19s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 16s branch-2 passed +1 mvnsite 0m 25s branch-2 passed +1 findbugs 0m 37s branch-2 passed +1 javadoc 0m 15s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 18s the patch passed +1 compile 0m 14s the patch passed with JDK v1.8.0_131 +1 javac 0m 14s the patch passed +1 compile 0m 16s the patch passed with JDK v1.7.0_131 +1 javac 0m 16s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 46s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_131       Other Tests +1 unit 1m 35s hadoop-azure in the patch passed with JDK v1.7.0_131. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 20m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue HADOOP-14640 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877701/HADOOP-14640-branch-2.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1d042625b838 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 02627f8c Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12805/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12805/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jnp Jitendra Nath Pandey added a comment -

        Committed to branch-2 as well.
        Thanks Santhosh G Nayak for the contribution.

        Show
        jnp Jitendra Nath Pandey added a comment - Committed to branch-2 as well. Thanks Santhosh G Nayak for the contribution.

          People

          • Assignee:
            snayak Santhosh G Nayak
            Reporter:
            snayak Santhosh G Nayak
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development