Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3583

ProcfsBasedProcessTree#constructProcessInfo() may throw NumberFormatException

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 1.0.2, 0.23.2
    • Component/s: None
    • Labels:
      None
    • Environment:

      64-bit Linux:
      asf011.sp2.ygridcore.net
      Linux asf011.sp2.ygridcore.net 2.6.32-33-server #71-Ubuntu SMP Wed Jul 20 17:42:25 UTC 2011 x86_64 GNU/Linux

    • Hadoop Flags:
      Reviewed

      Description

      HBase PreCommit builds frequently gave us NumberFormatException.

      From https://builds.apache.org/job/PreCommit-HBASE-Build/553//testReport/org.apache.hadoop.hbase.mapreduce/TestHFileOutputFormat/testMRIncrementalLoad/:

      2011-12-20 01:44:01,180 WARN  [main] mapred.JobClient(784): No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
      java.lang.NumberFormatException: For input string: "18446743988060683582"
      	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
      	at java.lang.Long.parseLong(Long.java:422)
      	at java.lang.Long.parseLong(Long.java:468)
      	at org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:413)
      	at org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:148)
      	at org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401)
      	at org.apache.hadoop.mapred.Task.initialize(Task.java:536)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
      	at org.apache.hadoop.mapred.Child.main(Child.java:249)
      

      From hadoop 0.20.205 source code, looks like ppid was 18446743988060683582, causing NFE:

              // Set (name) (ppid) (pgrpId) (session) (utime) (stime) (vsize) (rss)
               pinfo.updateProcessInfo(m.group(2), Integer.parseInt(m.group(3)),
      

      You can find information on the OS at the beginning of https://builds.apache.org/job/PreCommit-HBASE-Build/553/console:

      asf011.sp2.ygridcore.net
      Linux asf011.sp2.ygridcore.net 2.6.32-33-server #71-Ubuntu SMP Wed Jul 20 17:42:25 UTC 2011 x86_64 GNU/Linux
      core file size          (blocks, -c) 0
      data seg size           (kbytes, -d) unlimited
      scheduling priority             (-e) 20
      file size               (blocks, -f) unlimited
      pending signals                 (-i) 16382
      max locked memory       (kbytes, -l) 64
      max memory size         (kbytes, -m) unlimited
      open files                      (-n) 60000
      pipe size            (512 bytes, -p) 8
      POSIX message queues     (bytes, -q) 819200
      real-time priority              (-r) 0
      stack size              (kbytes, -s) 8192
      cpu time               (seconds, -t) unlimited
      max user processes              (-u) 2048
      virtual memory          (kbytes, -v) unlimited
      file locks                      (-x) unlimited
      60000
      Running in Jenkins mode
      

      From Nicolas Sze:

      It looks like that the ppid is a 64-bit positive integer but Java long is signed and so only works with 63-bit positive integers.  In your case,
      
        2^64 > 18446743988060683582 > 2^63.
      
      Therefore, there is a NFE. 
      

      I propose changing allProcessInfo to Map<String, ProcessInfo> so that we don't encounter this problem by avoiding parsing large integer.

      1. mapreduce-3583.txt
        8 kB
        ramkrishna.s.vasudevan
      2. mapreduce-3583-v2.txt
        9 kB
        Ted Yu
      3. mapreduce-3583-v3.txt
        9 kB
        Ted Yu
      4. mapreduce-3583-v4.txt
        9 kB
        Ted Yu
      5. mapreduce-3583-v5.txt
        9 kB
        Ted Yu
      6. mapreduce-3583-trunk.txt
        22 kB
        Ted Yu
      7. mapreduce-3583-trunk-v2.txt
        22 kB
        Ted Yu
      8. mapreduce-3583-trunk-v2.txt
        22 kB
        Ted Yu
      9. mapreduce-3583-trunk-v3.txt
        22 kB
        Ted Yu
      10. mapreduce-3583-trunk-v4.txt
        22 kB
        Ted Yu
      11. mapreduce-3583-trunk-v5.txt
        23 kB
        Ted Yu
      12. mapreduce-3583-trunk-v6.txt
        23 kB
        Ted Yu
      13. mapreduce-3583-trunk-v7.txt
        24 kB
        Ted Yu
      14. mapreduce-3583-v6.txt
        9 kB
        Ted Yu
      15. mapreduce-3583-v7.txt
        10 kB
        Ted Yu

        Issue Links

          Activity

          Ted Yu made changes -
          Link This issue relates to MAPREDUCE-5715 [ MAPREDUCE-5715 ]
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Matt Foley made changes -
          Fix Version/s 0.24.0 [ 12317654 ]
          Fix Version/s 1.1.0 [ 12317960 ]
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 1.1.0 [ 12317960 ]
          Fix Version/s 1.0.2 [ 12320047 ]
          Resolution Fixed [ 1 ]
          Ted Yu made changes -
          Attachment mapreduce-3583-v7.txt [ 12515597 ]
          Ted Yu made changes -
          Attachment mapreduce-3583-v6.txt [ 12515499 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue duplicates MAPREDUCE-3836 [ MAPREDUCE-3836 ]
          Tsz Wo Nicholas Sze made changes -
          Fix Version/s 0.24.0 [ 12317654 ]
          Fix Version/s 0.23.2 [ 12319851 ]
          Ted Yu made changes -
          Attachment mapreduce-3583-trunk-v7.txt [ 12514740 ]
          Tsz Wo Nicholas Sze made changes -
          Assignee ramkrishna.s.vasudevan [ ram_krish ] Zhihong Yu [ zhihyu@ebaysf.com ]
          Ted Yu made changes -
          Attachment mapreduce-3583-trunk-v6.txt [ 12514733 ]
          Ted Yu made changes -
          Attachment mapreduce-3583-trunk-v5.txt [ 12514728 ]
          Ted Yu made changes -
          Attachment mapreduce-3583-trunk-v4.txt [ 12514704 ]
          Ted Yu made changes -
          Attachment mapreduce-3583-trunk-v3.txt [ 12514689 ]
          Ted Yu made changes -
          Attachment mapreduce-3583-trunk-v2.txt [ 12514626 ]
          Ted Yu made changes -
          Attachment mapreduce-3583-trunk-v2.txt [ 12514624 ]
          Ted Yu made changes -
          Attachment mapreduce-3583-trunk.txt [ 12514572 ]
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Ted Yu made changes -
          Attachment mapreduce-3583-v5.txt [ 12514440 ]
          Ted Yu made changes -
          Attachment mapreduce-3583-v4.txt [ 12514427 ]
          Ted Yu made changes -
          Attachment mapreduce-3583-v3.txt [ 12513534 ]
          Ted Yu made changes -
          Attachment mapreduce-3583-v2.txt [ 12513194 ]
          Tom White made changes -
          Link This issue is related to MAPREDUCE-1201 [ MAPREDUCE-1201 ]
          Evan Pollan made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Mahadev konar made changes -
          Assignee ramkrishna.s.vasudevan [ ram_krish ]
          ramkrishna.s.vasudevan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          ramkrishna.s.vasudevan made changes -
          Field Original Value New Value
          Attachment mapreduce-3583.txt [ 12508247 ]
          Ted Yu created issue -

            People

            • Assignee:
              Ted Yu
              Reporter:
              Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development