Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1880

"java.lang.ArithmeticException: Non-terminating decimal expansion; no exact representable decimal result." while running "hadoop jar hadoop-0.20.1+169.89-examples.jar pi 4 30"

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.20.3, 0.21.0, 0.22.0
    • Component/s: examples
    • Environment:

      ubuntu 10.04 lucid
      sun jdk 6
      hadoop packaged by cloudera

    • Hadoop Flags:
      Reviewed

      Description

      If I run "hadoop jar hadoop-0.20.1+169.89-examples.jar pi 4 30", I get the following output:

      Number of Maps = 4
      Samples per Map = 30
      Wrote input for Map #0
      Wrote input for Map #1
      Wrote input for Map #2
      Wrote input for Map #3
      Starting Job
      10/06/19 21:50:34 INFO mapred.FileInputFormat: Total input paths to process : 4
      10/06/19 21:50:34 INFO mapred.JobClient: Running job: job_201006192101_0003
      10/06/19 21:50:35 INFO mapred.JobClient: map 0% reduce 0%
      10/06/19 21:50:45 INFO mapred.JobClient: map 50% reduce 0%
      10/06/19 21:50:51 INFO mapred.JobClient: map 100% reduce 0%
      10/06/19 21:51:00 INFO mapred.JobClient: map 100% reduce 100%
      10/06/19 21:51:02 INFO mapred.JobClient: Job complete: job_201006192101_0003
      10/06/19 21:51:02 INFO mapred.JobClient: Counters: 18
      10/06/19 21:51:02 INFO mapred.JobClient: Job Counters
      10/06/19 21:51:02 INFO mapred.JobClient: Launched reduce tasks=1
      10/06/19 21:51:02 INFO mapred.JobClient: Launched map tasks=4
      10/06/19 21:51:02 INFO mapred.JobClient: Data-local map tasks=4
      10/06/19 21:51:02 INFO mapred.JobClient: FileSystemCounters
      10/06/19 21:51:02 INFO mapred.JobClient: FILE_BYTES_READ=94
      10/06/19 21:51:02 INFO mapred.JobClient: HDFS_BYTES_READ=472
      10/06/19 21:51:02 INFO mapred.JobClient: FILE_BYTES_WRITTEN=334
      10/06/19 21:51:02 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=215
      10/06/19 21:51:02 INFO mapred.JobClient: Map-Reduce Framework
      10/06/19 21:51:02 INFO mapred.JobClient: Reduce input groups=8
      10/06/19 21:51:02 INFO mapred.JobClient: Combine output records=0
      10/06/19 21:51:02 INFO mapred.JobClient: Map input records=4
      10/06/19 21:51:02 INFO mapred.JobClient: Reduce shuffle bytes=112
      10/06/19 21:51:02 INFO mapred.JobClient: Reduce output records=0
      10/06/19 21:51:02 INFO mapred.JobClient: Spilled Records=16
      10/06/19 21:51:02 INFO mapred.JobClient: Map output bytes=72
      10/06/19 21:51:02 INFO mapred.JobClient: Map input bytes=96
      10/06/19 21:51:02 INFO mapred.JobClient: Combine input records=0
      10/06/19 21:51:02 INFO mapred.JobClient: Map output records=8
      10/06/19 21:51:02 INFO mapred.JobClient: Reduce input records=8
      Job Finished in 28.593 seconds
      java.lang.ArithmeticException: Non-terminating decimal expansion; no exact representable decimal result.
      at java.math.BigDecimal.divide(BigDecimal.java:1603)
      at org.apache.hadoop.examples.PiEstimator.estimate(PiEstimator.java:313)
      at org.apache.hadoop.examples.PiEstimator.run(PiEstimator.java:342)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.examples.PiEstimator.main(PiEstimator.java:351)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
      at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
      at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

      "hadoop jar hadoop-0.20.1+169.89-examples.jar pi 2 10" finishes fine

      1. m1880_20100619.patch
        1.0 kB
        Tsz Wo Nicholas Sze
      2. m1880_20100619b.patch
        2 kB
        Tsz Wo Nicholas Sze
      3. m1880_20100619_0.20.patch
        0.9 kB
        Tsz Wo Nicholas Sze

        Activity

        Hide
        Tsz Wo Nicholas Sze added a comment -

        I am able to reproduce this.

        Show
        Tsz Wo Nicholas Sze added a comment - I am able to reproduce this.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Thanks, Victor for reporting this.

        m1880_20100619_0.20.patch: need RoundingMode.HALF_UP in BigDecimal.divide(..)

        Show
        Tsz Wo Nicholas Sze added a comment - Thanks, Victor for reporting this. m1880_20100619_0.20.patch: need RoundingMode.HALF_UP in BigDecimal.divide(..)
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Tested manually, the patch works.

        -bash-3.1$ hadoop jar ~/hadoop-examples-0.20.3-SNAPSHOT.jar pi  -Dmapred.job.queue.name=xxx 4 30
        Number of Maps  = 4
        Samples per Map = 30
        Wrote input for Map #0
        Wrote input for Map #1
        Wrote input for Map #2
        Wrote input for Map #3
        Starting Job
        10/06/19 21:14:31 INFO mapred.FileInputFormat: Total input paths to process : 4
        10/06/19 21:14:31 INFO mapred.JobClient: Running job: job_201006110238_70224
        10/06/19 21:14:32 INFO mapred.JobClient:  map 0% reduce 0%
        10/06/19 21:14:38 INFO mapred.JobClient:  map 25% reduce 0%
        10/06/19 21:14:40 INFO mapred.JobClient:  map 100% reduce 0%
        10/06/19 21:18:43 INFO mapred.JobClient:  map 100% reduce 100%
        10/06/19 21:18:44 INFO mapred.JobClient: Job complete: job_201006110238_70224
        10/06/19 21:18:44 INFO mapred.JobClient: Counters: 22
        10/06/19 21:18:44 INFO mapred.JobClient:   Job Counters 
        10/06/19 21:18:44 INFO mapred.JobClient:     Launched reduce tasks=1
        10/06/19 21:18:44 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=11243
        10/06/19 21:18:44 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
        10/06/19 21:18:44 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
        10/06/19 21:18:44 INFO mapred.JobClient:     Rack-local map tasks=1
        10/06/19 21:18:44 INFO mapred.JobClient:     Launched map tasks=4
        10/06/19 21:18:44 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=14263
        10/06/19 21:18:44 INFO mapred.JobClient:   FileSystemCounters
        10/06/19 21:18:44 INFO mapred.JobClient:     FILE_BYTES_READ=68
        10/06/19 21:18:44 INFO mapred.JobClient:     HDFS_BYTES_READ=472
        10/06/19 21:18:44 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=348
        10/06/19 21:18:44 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=215
        10/06/19 21:18:44 INFO mapred.JobClient:   Map-Reduce Framework
        10/06/19 21:18:44 INFO mapred.JobClient:     Reduce input groups=8
        10/06/19 21:18:44 INFO mapred.JobClient:     Combine output records=0
        10/06/19 21:18:44 INFO mapred.JobClient:     Map input records=4
        10/06/19 21:18:44 INFO mapred.JobClient:     Reduce shuffle bytes=152
        10/06/19 21:18:44 INFO mapred.JobClient:     Reduce output records=0
        10/06/19 21:18:44 INFO mapred.JobClient:     Spilled Records=16
        10/06/19 21:18:44 INFO mapred.JobClient:     Map output bytes=72
        10/06/19 21:18:44 INFO mapred.JobClient:     Map input bytes=96
        10/06/19 21:18:44 INFO mapred.JobClient:     Combine input records=0
        10/06/19 21:18:44 INFO mapred.JobClient:     Map output records=8
        10/06/19 21:18:44 INFO mapred.JobClient:     Reduce input records=8
        Job Finished in 253.059 seconds
        Estimated value of Pi is 3.13333333333333333333
        
        Show
        Tsz Wo Nicholas Sze added a comment - Tested manually, the patch works. -bash-3.1$ hadoop jar ~/hadoop-examples-0.20.3-SNAPSHOT.jar pi -Dmapred.job.queue.name=xxx 4 30 Number of Maps = 4 Samples per Map = 30 Wrote input for Map #0 Wrote input for Map #1 Wrote input for Map #2 Wrote input for Map #3 Starting Job 10/06/19 21:14:31 INFO mapred.FileInputFormat: Total input paths to process : 4 10/06/19 21:14:31 INFO mapred.JobClient: Running job: job_201006110238_70224 10/06/19 21:14:32 INFO mapred.JobClient: map 0% reduce 0% 10/06/19 21:14:38 INFO mapred.JobClient: map 25% reduce 0% 10/06/19 21:14:40 INFO mapred.JobClient: map 100% reduce 0% 10/06/19 21:18:43 INFO mapred.JobClient: map 100% reduce 100% 10/06/19 21:18:44 INFO mapred.JobClient: Job complete: job_201006110238_70224 10/06/19 21:18:44 INFO mapred.JobClient: Counters: 22 10/06/19 21:18:44 INFO mapred.JobClient: Job Counters 10/06/19 21:18:44 INFO mapred.JobClient: Launched reduce tasks=1 10/06/19 21:18:44 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=11243 10/06/19 21:18:44 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 10/06/19 21:18:44 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 10/06/19 21:18:44 INFO mapred.JobClient: Rack-local map tasks=1 10/06/19 21:18:44 INFO mapred.JobClient: Launched map tasks=4 10/06/19 21:18:44 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=14263 10/06/19 21:18:44 INFO mapred.JobClient: FileSystemCounters 10/06/19 21:18:44 INFO mapred.JobClient: FILE_BYTES_READ=68 10/06/19 21:18:44 INFO mapred.JobClient: HDFS_BYTES_READ=472 10/06/19 21:18:44 INFO mapred.JobClient: FILE_BYTES_WRITTEN=348 10/06/19 21:18:44 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=215 10/06/19 21:18:44 INFO mapred.JobClient: Map-Reduce Framework 10/06/19 21:18:44 INFO mapred.JobClient: Reduce input groups=8 10/06/19 21:18:44 INFO mapred.JobClient: Combine output records=0 10/06/19 21:18:44 INFO mapred.JobClient: Map input records=4 10/06/19 21:18:44 INFO mapred.JobClient: Reduce shuffle bytes=152 10/06/19 21:18:44 INFO mapred.JobClient: Reduce output records=0 10/06/19 21:18:44 INFO mapred.JobClient: Spilled Records=16 10/06/19 21:18:44 INFO mapred.JobClient: Map output bytes=72 10/06/19 21:18:44 INFO mapred.JobClient: Map input bytes=96 10/06/19 21:18:44 INFO mapred.JobClient: Combine input records=0 10/06/19 21:18:44 INFO mapred.JobClient: Map output records=8 10/06/19 21:18:44 INFO mapred.JobClient: Reduce input records=8 Job Finished in 253.059 seconds Estimated value of Pi is 3.13333333333333333333
        Hide
        Tsz Wo Nicholas Sze added a comment -

        m1880_20100619b.patch: patch for trunk; also added a test.

        Show
        Tsz Wo Nicholas Sze added a comment - m1880_20100619b.patch: patch for trunk; also added a test.
        Hide
        Todd Lipcon added a comment -

        +1 to fix. Do we need to edit TestMiniMRLocalFS though? Seems out of place.

        (sad to see this bug go, though - I always thought it was rather cute that it complained, given pi itself is a non-terminating decimal expansion with no exact respresentable decimal result )

        Show
        Todd Lipcon added a comment - +1 to fix. Do we need to edit TestMiniMRLocalFS though? Seems out of place. (sad to see this bug go, though - I always thought it was rather cute that it complained, given pi itself is a non-terminating decimal expansion with no exact respresentable decimal result )
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > Do we need to edit TestMiniMRLocalFS though? Seems out of place.
        Sure, let's revert the change. Thanks Todd for reviewing it.

        m1880_20100619.patch: removed the test.

        Show
        Tsz Wo Nicholas Sze added a comment - > Do we need to edit TestMiniMRLocalFS though? Seems out of place. Sure, let's revert the change. Thanks Todd for reviewing it. m1880_20100619.patch: removed the test.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12447536/m1880_20100619b.patch
        against trunk revision 956171.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/254/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/254/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/254/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/254/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447536/m1880_20100619b.patch against trunk revision 956171. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/254/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/254/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/254/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/254/console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this to 0.20, 0.20-append and above.

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this to 0.20, 0.20-append and above.

          People

          • Assignee:
            Tsz Wo Nicholas Sze
            Reporter:
            Victor Pakhomov
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development