Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4395

Possible NPE at ClientDistributedCacheManager#determineTimestamps

    Details

      Description

      ClientDistributedCacheManager#determineTimestamps
      URI[] tfiles = DistributedCache.getCacheFiles(job);
      

      It may be possible that tfiles array contains null as it's entry, and subsequently leads to NPE.

      1. MAPREDUCE-4395-1.patch
        2 kB
        Bhallamudi Venkata Siva Kamesh
      2. MAPREDUCE-4395.patch
        2 kB
        Bhallamudi Venkata Siva Kamesh

        Activity

        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Here is an example.

        Exception in thread "main" java.lang.NullPointerException
        	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:282)
        	at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:205)
        	at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:77)
        	at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:245)
        	at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:283)
        	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:354)
        
        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Here is an example. Exception in thread "main" java.lang.NullPointerException at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:282) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:205) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:77) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:245) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:283) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:354)
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        I think, when we pass the DC entries through options like -files -libjars etc..., and, if there is any exception during URI construction, it will be thrown right away. However when we pass the same entries either through config or through programatically, exception during URI construction is ignored and the corresponding URI value has been set as null.

        StringUtils#stringToURI
            for (int i = 0; i < str.length;i++){
              try{
                uris[i] = new URI(str[i]);
              }catch(URISyntaxException ur){
                System.out.println("Exception in specified URI's " + StringUtils.stringifyException(ur));
                //making sure its asssigned to null in case of an error
                uris[i] = null;
              }
            }
        

        I think fix is, throwing an Illegal argument exception, rather than setting it as null.

        Any comments?

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - I think , when we pass the DC entries through options like -files -libjars etc..., and, if there is any exception during URI construction, it will be thrown right away. However when we pass the same entries either through config or through programatically, exception during URI construction is ignored and the corresponding URI value has been set as null . StringUtils#stringToURI for ( int i = 0; i < str.length;i++){ try { uris[i] = new URI(str[i]); } catch (URISyntaxException ur){ System .out.println( "Exception in specified URI's " + StringUtils.stringifyException(ur)); //making sure its asssigned to null in case of an error uris[i] = null ; } } I think fix is, throwing an Illegal argument exception, rather than setting it as null . Any comments?
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Submitting patch, please review.

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Submitting patch, please review.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12536357/MAPREDUCE-4395.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.io.file.tfile.TestTFileByteArrays
        org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2586//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2586//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536357/MAPREDUCE-4395.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.io.file.tfile.TestTFileByteArrays org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2586//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2586//console This message is automatically generated.
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        test failures are unrelated to this patch.

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - test failures are unrelated to this patch.
        Hide
        Robert Joseph Evans added a comment -

        Looking at the patch it looks OK to me. I looked and the only other thing that uses this API is streaming, when it is setting up a JobConf, so I am OK with it blowing up when the URI is not valid. My only comment is that I would like the Javadocs for this method updated to explain what it does and also indicate what happens in the failure case. because well

           /**
            *
            * @param str
            */
        

        is completely useless.

        Show
        Robert Joseph Evans added a comment - Looking at the patch it looks OK to me. I looked and the only other thing that uses this API is streaming, when it is setting up a JobConf, so I am OK with it blowing up when the URI is not valid. My only comment is that I would like the Javadocs for this method updated to explain what it does and also indicate what happens in the failure case. because well /** * * @param str */ is completely useless.
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Thanks Bobby for looking into the patch. Added javadoc. Please go through the patch.

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Thanks Bobby for looking into the patch. Added javadoc. Please go through the patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12536550/MAPREDUCE-4395-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays
        org.apache.hadoop.io.file.tfile.TestTFileByteArrays

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2593//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2593//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536550/MAPREDUCE-4395-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays org.apache.hadoop.io.file.tfile.TestTFileByteArrays +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2593//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2593//console This message is automatically generated.
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        test failures are unrelated to this patch.

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - test failures are unrelated to this patch.
        Hide
        Robert Joseph Evans added a comment -

        +1 the patch looks good to me.

        Show
        Robert Joseph Evans added a comment - +1 the patch looks good to me.
        Hide
        Robert Joseph Evans added a comment -

        Thanks Bhallamudi,

        I put this into trunk and branch-2/

        Show
        Robert Joseph Evans added a comment - Thanks Bhallamudi, I put this into trunk and branch-2/
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2543 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2543/)
        MAPREDUCE-4395. Possible NPE at ClientDistributedCacheManager#determineTimestamps (Bhallamudi via bobby) (Revision 1362052)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362052
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2543 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2543/ ) MAPREDUCE-4395 . Possible NPE at ClientDistributedCacheManager#determineTimestamps (Bhallamudi via bobby) (Revision 1362052) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362052 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2478 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2478/)
        MAPREDUCE-4395. Possible NPE at ClientDistributedCacheManager#determineTimestamps (Bhallamudi via bobby) (Revision 1362052)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362052
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2478 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2478/ ) MAPREDUCE-4395 . Possible NPE at ClientDistributedCacheManager#determineTimestamps (Bhallamudi via bobby) (Revision 1362052) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362052 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2498 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2498/)
        MAPREDUCE-4395. Possible NPE at ClientDistributedCacheManager#determineTimestamps (Bhallamudi via bobby) (Revision 1362052)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362052
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2498 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2498/ ) MAPREDUCE-4395 . Possible NPE at ClientDistributedCacheManager#determineTimestamps (Bhallamudi via bobby) (Revision 1362052) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362052 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Hide
        Bhallamudi Venkata Siva Kamesh added a comment -

        Thanks Bobby

        Show
        Bhallamudi Venkata Siva Kamesh added a comment - Thanks Bobby
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1106 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1106/)
        MAPREDUCE-4395. Possible NPE at ClientDistributedCacheManager#determineTimestamps (Bhallamudi via bobby) (Revision 1362052)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362052
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1106 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1106/ ) MAPREDUCE-4395 . Possible NPE at ClientDistributedCacheManager#determineTimestamps (Bhallamudi via bobby) (Revision 1362052) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362052 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1139 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1139/)
        MAPREDUCE-4395. Possible NPE at ClientDistributedCacheManager#determineTimestamps (Bhallamudi via bobby) (Revision 1362052)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362052
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1139 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1139/ ) MAPREDUCE-4395 . Possible NPE at ClientDistributedCacheManager#determineTimestamps (Bhallamudi via bobby) (Revision 1362052) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362052 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #317 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/317/)
        svn merge -c 1362052 FIXES: MAPREDUCE-4395. Possible NPE at ClientDistributedCacheManager#determineTimestamps (Bhallamudi via bobby) (Revision 1362549)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362549
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #317 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/317/ ) svn merge -c 1362052 FIXES: MAPREDUCE-4395 . Possible NPE at ClientDistributedCacheManager#determineTimestamps (Bhallamudi via bobby) (Revision 1362549) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362549 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt

          People

          • Assignee:
            Bhallamudi Venkata Siva Kamesh
            Reporter:
            Bhallamudi Venkata Siva Kamesh
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development