Apache Whirr (retired)
  1. Apache Whirr (retired)
  2. WHIRR-582

Yarn service does not build in IntelliJ because of missing Hadoop test dependencies

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.2, 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      This is related to the recent mailinglist discussion about problem with Yarn, Hadoop versions and Maven 3.

      I can't build the project in IntelliJ as the Yarn service misses dependencies on all sorts of Hadoop classes. A mvn clean install from the command line does work however.

      1. WHIRR-582.patch
        10 kB
        Frank Scholten

        Activity

        Frank Scholten created issue -
        Frank Scholten made changes -
        Field Original Value New Value
        Summary Yarn service does not build in IntelliJ becuase of missing Hadoop test dependencies Yarn service does not build in IntelliJ because of missing Hadoop test dependencies
        Hide
        Andrei Savu added a comment -

        How are you building the project in IntelliJ? I'm able to run tests as expected.

        Show
        Andrei Savu added a comment - How are you building the project in IntelliJ? I'm able to run tests as expected.
        Hide
        Frank Scholten added a comment -

        I get compilation errors cause AbstractHadoopServiceTest cannot find imports such as Configuration and other Hadoop classes.

        Show
        Frank Scholten added a comment - I get compilation errors cause AbstractHadoopServiceTest cannot find imports such as Configuration and other Hadoop classes.
        Hide
        Frank Scholten added a comment - - edited

        The following imports cannot be found in AbstractHadoopServiceTest:

        import org.apache.hadoop.conf.Configuration;
        import org.apache.hadoop.fs.FSDataInputStream;
        import org.apache.hadoop.fs.FileSystem;
        import org.apache.hadoop.fs.Path;
        import org.apache.hadoop.io.LongWritable;
        import org.apache.hadoop.io.Text;
        import org.apache.hadoop.mapred.FileInputFormat;
        import org.apache.hadoop.mapred.FileOutputFormat;
        import org.apache.hadoop.mapred.JobClient;
        import org.apache.hadoop.mapred.JobConf;
        import org.apache.hadoop.mapred.lib.LongSumReducer;
        import org.apache.hadoop.mapred.lib.TokenCountMapper;
        

        And these imports cannot be found in HadoopServiceController:

         
        import org.apache.hadoop.conf.Configuration;
        import org.apache.hadoop.fs.FileSystem;
        import org.apache.hadoop.hdfs.DistributedFileSystem;
        import org.apache.hadoop.hdfs.protocol.FSConstants;
        import org.apache.hadoop.mapred.ClusterStatus;
        import org.apache.hadoop.mapred.JobClient;
        import org.apache.hadoop.mapred.JobConf;
        

        The yarn pom contains

            <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-client</artifactId>
              <version>0.23.1</version>
              <scope>test</scope>
            </dependency>
        

        Which Hadoop dependencies are needed here?

        Show
        Frank Scholten added a comment - - edited The following imports cannot be found in AbstractHadoopServiceTest: import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapred.FileInputFormat; import org.apache.hadoop.mapred.FileOutputFormat; import org.apache.hadoop.mapred.JobClient; import org.apache.hadoop.mapred.JobConf; import org.apache.hadoop.mapred.lib.LongSumReducer; import org.apache.hadoop.mapred.lib.TokenCountMapper; And these imports cannot be found in HadoopServiceController: import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.hdfs.DistributedFileSystem; import org.apache.hadoop.hdfs.protocol.FSConstants; import org.apache.hadoop.mapred.ClusterStatus; import org.apache.hadoop.mapred.JobClient; import org.apache.hadoop.mapred.JobConf; The yarn pom contains <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>0.23.1</version> <scope>test</scope> </dependency> Which Hadoop dependencies are needed here?
        Hide
        Frank Scholten added a comment -

        Under 'External Libraries' in the project view in IntelliJ I now have the following Hadoop libraries:

        Maven: org.apache.hadoop:avro:1.3.2
        Maven: org.apache.hadoop:hadoop-common:0.23.0-cdh4b1
        Maven: org.apache.hadoop:hadoop-core:0.20.2
        Maven: org.apache.hadoop:hadoop-core:0.20.2-cdh3u2
        Maven: org.apache.hadoop:hadoop-core:0.20.205.0
        Maven: org.apache.hadoop:hadoop-examples:0.20.205.0
        Maven: org.apache.hadoop:hadoop-hdfs:0.23.0-cdh4b1
        Maven: org.apache.hadoop:hadoop-mapreduce-client-common:0.23.0-cdh4b1
        Maven: org.apache.hadoop:hadoop-mapreduce-client-core:0.23.0-cdh4b1
        Maven: org.apache.hadoop:hadoop-mapreduce-client-jobclient:0.23.0-cdh4b1
        Maven: org.apache.hadoop:hadoop-test:0.20.2
        Maven: org.apache.hadoop:hadoop-test:0.20.205.0
        
        Show
        Frank Scholten added a comment - Under 'External Libraries' in the project view in IntelliJ I now have the following Hadoop libraries: Maven: org.apache.hadoop:avro:1.3.2 Maven: org.apache.hadoop:hadoop-common:0.23.0-cdh4b1 Maven: org.apache.hadoop:hadoop-core:0.20.2 Maven: org.apache.hadoop:hadoop-core:0.20.2-cdh3u2 Maven: org.apache.hadoop:hadoop-core:0.20.205.0 Maven: org.apache.hadoop:hadoop-examples:0.20.205.0 Maven: org.apache.hadoop:hadoop-hdfs:0.23.0-cdh4b1 Maven: org.apache.hadoop:hadoop-mapreduce-client-common:0.23.0-cdh4b1 Maven: org.apache.hadoop:hadoop-mapreduce-client-core:0.23.0-cdh4b1 Maven: org.apache.hadoop:hadoop-mapreduce-client-jobclient:0.23.0-cdh4b1 Maven: org.apache.hadoop:hadoop-test:0.20.2 Maven: org.apache.hadoop:hadoop-test:0.20.205.0
        Hide
        Frank Scholten added a comment -

        Also curious if Eclipse users are having problems with this. It looks to me that we have some conflicting dependencies. I don't know which ones we can pull out, upgrade or include however.

        Show
        Frank Scholten added a comment - Also curious if Eclipse users are having problems with this. It looks to me that we have some conflicting dependencies. I don't know which ones we can pull out, upgrade or include however.
        Hide
        Frank Scholten added a comment -

        Tom, since you wrote the Yarn service do you know which of the above dependencies can be replaced or upgraded to fix building in IntelliJ?

        Show
        Frank Scholten added a comment - Tom, since you wrote the Yarn service do you know which of the above dependencies can be replaced or upgraded to fix building in IntelliJ?
        Hide
        Frank Scholten added a comment -

        The same problem occurs with HBase. The whirr-cdh module has HBase 0.92.0-cdh4b1 as a depedendency while whirr-hbase has 0.89.20100924-28 as a dependency.

        Show
        Frank Scholten added a comment - The same problem occurs with HBase. The whirr-cdh module has HBase 0.92.0-cdh4b1 as a depedendency while whirr-hbase has 0.89.20100924-28 as a dependency.
        Hide
        Frank Scholten added a comment -

        Building via Maven from IntelliJ still works, but not via CTRL-F9

        Show
        Frank Scholten added a comment - Building via Maven from IntelliJ still works, but not via CTRL-F9
        Hide
        Frank Scholten added a comment - - edited

        I just saw that that CdhHBaseServiceTest didn't compile at all and was excluded in the cdh pom. See WHIRR-525

        This patch makes the test compile by wrapping each byte array in a ByteBuffer.

        Explicit Hadoop 0.23.1 test dependencies are added to the pom to fix AbstractHadoopServiceTest. Even though hadoop-client 0.23.1 pom includes these dependencies they have to be added explicitly to make the build work in IntelliJ.

        Show
        Frank Scholten added a comment - - edited I just saw that that CdhHBaseServiceTest didn't compile at all and was excluded in the cdh pom. See WHIRR-525 This patch makes the test compile by wrapping each byte array in a ByteBuffer. Explicit Hadoop 0.23.1 test dependencies are added to the pom to fix AbstractHadoopServiceTest. Even though hadoop-client 0.23.1 pom includes these dependencies they have to be added explicitly to make the build work in IntelliJ.
        Frank Scholten made changes -
        Attachment WHIRR-582.patch [ 12526336 ]
        Hide
        Andrei Savu added a comment -

        +1

        Show
        Andrei Savu added a comment - +1
        Hide
        Frank Scholten added a comment -

        Committed to trunk.

        Show
        Frank Scholten added a comment - Committed to trunk.
        Frank Scholten made changes -
        Fix Version/s 0.8.0 [ 12318880 ]
        Hide
        Frank Scholten added a comment -

        Committed to 0.7 branch.

        Show
        Frank Scholten added a comment - Committed to 0.7 branch.
        Frank Scholten made changes -
        Fix Version/s 0.7.2 [ 12320175 ]
        Frank Scholten made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Frank Scholten made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Assignee Frank Scholten [ frankscholten ]
        Tom White made changes -
        Fix Version/s 0.8.0 [ 12321649 ]
        Fix Version/s 0.8.0-alpha.1 [ 12318880 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        33d 6h 37m 1 Frank Scholten 10/May/12 21:38
        Resolved Resolved Closed Closed
        12s 1 Frank Scholten 10/May/12 21:38

          People

          • Assignee:
            Frank Scholten
            Reporter:
            Frank Scholten
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development