Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-659

gridmix2 not compiling under mapred module trunk/src/benchmarks/gridmix2

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: build
    • Labels:
      None
    • Environment:

      latest trunk

      Description

      When build is tried in gridmix2, it fails

      trunk/src/benchmarks/gridmix2 $ ant
      Buildfile: build.xml

      init:

      compile:
      [javac] Compiling 3 source files to /home/iyappans/new_trunk1/mapreduce/trunk/src/benchmarks/gridmix2/build
      [javac] /home/iyappans/new_trunk1/mapreduce/trunk/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred/GridMixRunner.java:40: package org.apache.hadoop.streaming does not exist
      [javac] import org.apache.hadoop.streaming.StreamJob;
      [javac] ^
      [javac] /home/iyappans/new_trunk1/mapreduce/trunk/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred/GridMixRunner.java:123: cannot find symbol
      [javac] symbol: variable StreamJob
      [javac] JobConf jobconf = StreamJob.createJob(args);
      [javac] ^
      [javac] Note: Some input files use or override a deprecated API.
      [javac] Note: Recompile with -Xlint:deprecation for details.
      [javac] 2 errors

      BUILD FAILED
      /home/iyappans/new_trunk1/mapreduce/trunk/src/benchmarks/gridmix2/build.xml:27: Compile failed; see the compiler error output for details.

      Total time: 1 second

      1. MAPREDUCE-659-V2.PATCH
        2 kB
        Giridharan Kesavan
      2. MAPREDUCE-659.PATCH
        0.6 kB
        Giridharan Kesavan
      3. 659-1.patch
        1 kB
        Chris Douglas

        Activity

        Hide
        Giridharan Kesavan added a comment -

        this fixes the gridmix2 build failure.

        Steps:
        checkout mapred trunk

        ant package -Dforrest.home=<forrest.home> -Djava5.home=<java5.home>
        cd build/hadoop-mapred-0.21.0-dev/src/benchmark/gridmix2
        ant

        This will generate gridmix2 jar

        tnx!

        Show
        Giridharan Kesavan added a comment - this fixes the gridmix2 build failure. Steps: checkout mapred trunk ant package -Dforrest.home=<forrest.home> -Djava5.home=<java5.home> cd build/hadoop-mapred-0.21.0-dev/src/benchmark/gridmix2 ant This will generate gridmix2 jar tnx!
        Hide
        Iyappan Srinivasan added a comment -

        I checked outa fresh trunk, applied this patch and compiled as mentioned. The gridmix.jar file got generated successfully after a successful compilation.

        Show
        Iyappan Srinivasan added a comment - I checked outa fresh trunk, applied this patch and compiled as mentioned. The gridmix.jar file got generated successfully after a successful compilation.
        Hide
        Chris Douglas added a comment -

        For developers, it's often easier if it picks up jars in the build dir first; running the package target takes a long time.

        Show
        Chris Douglas added a comment - For developers, it's often easier if it picks up jars in the build dir first; running the package target takes a long time.
        Hide
        Giridharan Kesavan added a comment -

        $

        {hadoop.dir}/build directory doesnt exist in tar.gz package.

        When someone tries to build gridmix2 by untarring hadoop-mapred-0.21.0-dev.tar.gz
        ${hadoop.dir}

        /build doesn't exit and build would fail.

        Show
        Giridharan Kesavan added a comment - $ {hadoop.dir}/build directory doesnt exist in tar.gz package. When someone tries to build gridmix2 by untarring hadoop-mapred-0.21.0-dev.tar.gz ${hadoop.dir} /build doesn't exit and build would fail.
        Hide
        Amareshwari Sriramadasu added a comment -

        +1 for Chris's patch.
        I did ant tar.
        cd src/benchmarks/gridmix2
        ant
        This created gridmix.jar

        With Giri's patch, though I was able to build jar, following the steps described by Giri, when I run I was facing problems with runtime ClassNotFoundException.

        Show
        Amareshwari Sriramadasu added a comment - +1 for Chris's patch. I did ant tar. cd src/benchmarks/gridmix2 ant This created gridmix.jar With Giri's patch, though I was able to build jar, following the steps described by Giri, when I run I was facing problems with runtime ClassNotFoundException.
        Hide
        Amareshwari Sriramadasu added a comment -

        ClassNotFoundException is due to bin/hadoop script after project split. It is nothing to do with patches.

        For developers, it's often easier if it picks up jars in the build dir first; running the package target takes a long time.

        +1
        I think, we should modify gridmix2/build.xml to pickup jar files properly either it is in src directory or build directory.

        Show
        Amareshwari Sriramadasu added a comment - ClassNotFoundException is due to bin/hadoop script after project split. It is nothing to do with patches. For developers, it's often easier if it picks up jars in the build dir first; running the package target takes a long time. +1 I think, we should modify gridmix2/build.xml to pickup jar files properly either it is in src directory or build directory.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12412038/659-1.patch
        against trunk revision 790543.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/336/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/336/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/336/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/336/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12412038/659-1.patch against trunk revision 790543. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/336/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/336/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/336/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/336/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Increasing the priority as gridmix compilation is broken

        Show
        Amareshwari Sriramadasu added a comment - Increasing the priority as gridmix compilation is broken
        Hide
        Amareshwari Sriramadasu added a comment -

        To catch the compilation failures early, gridmix should be added to the binary/package target itself.

        User should be able to build the jar just by doing in ant in the src/benchmarks/gridmix2 directory. If there are any dependencies with streaming and test, they should be added to build.xml

        Show
        Amareshwari Sriramadasu added a comment - To catch the compilation failures early, gridmix should be added to the binary/package target itself. User should be able to build the jar just by doing in ant in the src/benchmarks/gridmix2 directory. If there are any dependencies with streaming and test, they should be added to build.xml
        Hide
        Chris Douglas added a comment -

        Sorry, I hadn't placed a watch on this issue.

        When someone tries to build gridmix2 by untarring hadoop-mapred-0.21.0-dev.tar.gz
        ${hadoop.dir}/build doesn't exit and build would fail.

        The change was a superset of the original patch; it only adds the build dir to the classpath, so gridmix2 will be linked against those classes/jars if they exist.

        To catch the compilation failures early, gridmix should be added to the binary/package target itself.

        User should be able to build the jar just by doing in ant in the src/benchmarks/gridmix2 directory. If there are any dependencies with streaming and test, they should be added to build.xml

        This is reasonable, but gridmix2 shouldn't be a top-level target, it has too many dependencies to model cleanly in our current setup, and it's not a critical component. We could move it to contrib, but we don't have a way to model dependencies between contrib modules. I guess it could go into tools, but that's a stretch. For now, fixing the compilation error seems sufficient.

        Show
        Chris Douglas added a comment - Sorry, I hadn't placed a watch on this issue. When someone tries to build gridmix2 by untarring hadoop-mapred-0.21.0-dev.tar.gz ${hadoop.dir}/build doesn't exit and build would fail. The change was a superset of the original patch; it only adds the build dir to the classpath, so gridmix2 will be linked against those classes/jars if they exist. To catch the compilation failures early, gridmix should be added to the binary/package target itself. User should be able to build the jar just by doing in ant in the src/benchmarks/gridmix2 directory. If there are any dependencies with streaming and test, they should be added to build.xml This is reasonable, but gridmix2 shouldn't be a top-level target, it has too many dependencies to model cleanly in our current setup, and it's not a critical component. We could move it to contrib, but we don't have a way to model dependencies between contrib modules. I guess it could go into tools, but that's a stretch. For now, fixing the compilation error seems sufficient.
        Hide
        Giridharan Kesavan added a comment -

        created a different target for developers
        dev-build

        cd trunk
        ant all-jars - this would create all the jars of mapreduce
        cd src/benchmarks/gridmix2
        ant dev-build

        this would create the gridmix2 jar.

        others can continue using the same target as before.

        Show
        Giridharan Kesavan added a comment - created a different target for developers dev-build cd trunk ant all-jars - this would create all the jars of mapreduce cd src/benchmarks/gridmix2 ant dev-build this would create the gridmix2 jar. others can continue using the same target as before.
        Hide
        Giridharan Kesavan added a comment -

        I just committed this.
        Thanks for the comments.

        Show
        Giridharan Kesavan added a comment - I just committed this. Thanks for the comments.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #40 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/40/)
        . Fix gridmix2 compilation. Contributed by Giridharan Kesavan.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #40 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/40/ ) . Fix gridmix2 compilation. Contributed by Giridharan Kesavan.

          People

          • Assignee:
            Giridharan Kesavan
            Reporter:
            Iyappan Srinivasan
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development