Whirr
  1. Whirr
  2. WHIRR-582

Yarn service does not build in IntelliJ because of missing Hadoop test dependencies

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.2, 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      This is related to the recent mailinglist discussion about problem with Yarn, Hadoop versions and Maven 3.

      I can't build the project in IntelliJ as the Yarn service misses dependencies on all sorts of Hadoop classes. A mvn clean install from the command line does work however.

      1. WHIRR-582.patch
        10 kB
        Frank Scholten

        Activity

        Hide
        Andrei Savu added a comment -

        How are you building the project in IntelliJ? I'm able to run tests as expected.

        Show
        Andrei Savu added a comment - How are you building the project in IntelliJ? I'm able to run tests as expected.
        Hide
        Frank Scholten added a comment -

        I get compilation errors cause AbstractHadoopServiceTest cannot find imports such as Configuration and other Hadoop classes.

        Show
        Frank Scholten added a comment - I get compilation errors cause AbstractHadoopServiceTest cannot find imports such as Configuration and other Hadoop classes.
        Hide
        Frank Scholten added a comment - - edited

        The following imports cannot be found in AbstractHadoopServiceTest:

        import org.apache.hadoop.conf.Configuration;
        import org.apache.hadoop.fs.FSDataInputStream;
        import org.apache.hadoop.fs.FileSystem;
        import org.apache.hadoop.fs.Path;
        import org.apache.hadoop.io.LongWritable;
        import org.apache.hadoop.io.Text;
        import org.apache.hadoop.mapred.FileInputFormat;
        import org.apache.hadoop.mapred.FileOutputFormat;
        import org.apache.hadoop.mapred.JobClient;
        import org.apache.hadoop.mapred.JobConf;
        import org.apache.hadoop.mapred.lib.LongSumReducer;
        import org.apache.hadoop.mapred.lib.TokenCountMapper;
        

        And these imports cannot be found in HadoopServiceController:

         
        import org.apache.hadoop.conf.Configuration;
        import org.apache.hadoop.fs.FileSystem;
        import org.apache.hadoop.hdfs.DistributedFileSystem;
        import org.apache.hadoop.hdfs.protocol.FSConstants;
        import org.apache.hadoop.mapred.ClusterStatus;
        import org.apache.hadoop.mapred.JobClient;
        import org.apache.hadoop.mapred.JobConf;
        

        The yarn pom contains

            <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-client</artifactId>
              <version>0.23.1</version>
              <scope>test</scope>
            </dependency>
        

        Which Hadoop dependencies are needed here?

        Show
        Frank Scholten added a comment - - edited The following imports cannot be found in AbstractHadoopServiceTest: import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapred.FileInputFormat; import org.apache.hadoop.mapred.FileOutputFormat; import org.apache.hadoop.mapred.JobClient; import org.apache.hadoop.mapred.JobConf; import org.apache.hadoop.mapred.lib.LongSumReducer; import org.apache.hadoop.mapred.lib.TokenCountMapper; And these imports cannot be found in HadoopServiceController: import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.hdfs.DistributedFileSystem; import org.apache.hadoop.hdfs.protocol.FSConstants; import org.apache.hadoop.mapred.ClusterStatus; import org.apache.hadoop.mapred.JobClient; import org.apache.hadoop.mapred.JobConf; The yarn pom contains <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>0.23.1</version> <scope>test</scope> </dependency> Which Hadoop dependencies are needed here?
        Hide
        Frank Scholten added a comment -

        Under 'External Libraries' in the project view in IntelliJ I now have the following Hadoop libraries:

        Maven: org.apache.hadoop:avro:1.3.2
        Maven: org.apache.hadoop:hadoop-common:0.23.0-cdh4b1
        Maven: org.apache.hadoop:hadoop-core:0.20.2
        Maven: org.apache.hadoop:hadoop-core:0.20.2-cdh3u2
        Maven: org.apache.hadoop:hadoop-core:0.20.205.0
        Maven: org.apache.hadoop:hadoop-examples:0.20.205.0
        Maven: org.apache.hadoop:hadoop-hdfs:0.23.0-cdh4b1
        Maven: org.apache.hadoop:hadoop-mapreduce-client-common:0.23.0-cdh4b1
        Maven: org.apache.hadoop:hadoop-mapreduce-client-core:0.23.0-cdh4b1
        Maven: org.apache.hadoop:hadoop-mapreduce-client-jobclient:0.23.0-cdh4b1
        Maven: org.apache.hadoop:hadoop-test:0.20.2
        Maven: org.apache.hadoop:hadoop-test:0.20.205.0
        
        Show
        Frank Scholten added a comment - Under 'External Libraries' in the project view in IntelliJ I now have the following Hadoop libraries: Maven: org.apache.hadoop:avro:1.3.2 Maven: org.apache.hadoop:hadoop-common:0.23.0-cdh4b1 Maven: org.apache.hadoop:hadoop-core:0.20.2 Maven: org.apache.hadoop:hadoop-core:0.20.2-cdh3u2 Maven: org.apache.hadoop:hadoop-core:0.20.205.0 Maven: org.apache.hadoop:hadoop-examples:0.20.205.0 Maven: org.apache.hadoop:hadoop-hdfs:0.23.0-cdh4b1 Maven: org.apache.hadoop:hadoop-mapreduce-client-common:0.23.0-cdh4b1 Maven: org.apache.hadoop:hadoop-mapreduce-client-core:0.23.0-cdh4b1 Maven: org.apache.hadoop:hadoop-mapreduce-client-jobclient:0.23.0-cdh4b1 Maven: org.apache.hadoop:hadoop-test:0.20.2 Maven: org.apache.hadoop:hadoop-test:0.20.205.0
        Hide
        Frank Scholten added a comment -

        Also curious if Eclipse users are having problems with this. It looks to me that we have some conflicting dependencies. I don't know which ones we can pull out, upgrade or include however.

        Show
        Frank Scholten added a comment - Also curious if Eclipse users are having problems with this. It looks to me that we have some conflicting dependencies. I don't know which ones we can pull out, upgrade or include however.
        Hide
        Frank Scholten added a comment -

        Tom, since you wrote the Yarn service do you know which of the above dependencies can be replaced or upgraded to fix building in IntelliJ?

        Show
        Frank Scholten added a comment - Tom, since you wrote the Yarn service do you know which of the above dependencies can be replaced or upgraded to fix building in IntelliJ?
        Hide
        Frank Scholten added a comment -

        The same problem occurs with HBase. The whirr-cdh module has HBase 0.92.0-cdh4b1 as a depedendency while whirr-hbase has 0.89.20100924-28 as a dependency.

        Show
        Frank Scholten added a comment - The same problem occurs with HBase. The whirr-cdh module has HBase 0.92.0-cdh4b1 as a depedendency while whirr-hbase has 0.89.20100924-28 as a dependency.
        Hide
        Frank Scholten added a comment -

        Building via Maven from IntelliJ still works, but not via CTRL-F9

        Show
        Frank Scholten added a comment - Building via Maven from IntelliJ still works, but not via CTRL-F9
        Hide
        Frank Scholten added a comment - - edited

        I just saw that that CdhHBaseServiceTest didn't compile at all and was excluded in the cdh pom. See WHIRR-525

        This patch makes the test compile by wrapping each byte array in a ByteBuffer.

        Explicit Hadoop 0.23.1 test dependencies are added to the pom to fix AbstractHadoopServiceTest. Even though hadoop-client 0.23.1 pom includes these dependencies they have to be added explicitly to make the build work in IntelliJ.

        Show
        Frank Scholten added a comment - - edited I just saw that that CdhHBaseServiceTest didn't compile at all and was excluded in the cdh pom. See WHIRR-525 This patch makes the test compile by wrapping each byte array in a ByteBuffer. Explicit Hadoop 0.23.1 test dependencies are added to the pom to fix AbstractHadoopServiceTest. Even though hadoop-client 0.23.1 pom includes these dependencies they have to be added explicitly to make the build work in IntelliJ.
        Hide
        Andrei Savu added a comment -

        +1

        Show
        Andrei Savu added a comment - +1
        Hide
        Frank Scholten added a comment -

        Committed to trunk.

        Show
        Frank Scholten added a comment - Committed to trunk.
        Hide
        Frank Scholten added a comment -

        Committed to 0.7 branch.

        Show
        Frank Scholten added a comment - Committed to 0.7 branch.

          People

          • Assignee:
            Frank Scholten
            Reporter:
            Frank Scholten
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development