Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2282

MapReduce tests don't compile following HDFS-1561

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestMRServerPorts depends on TestHDFSServerPorts which was changed by HDFS-1561, resulting in a compilation failure.

      1. TestMRPorts.patch
        4 kB
        Konstantin Shvachko

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #580 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/580/)
        MAPREDUCE-2282. Fix TestMRServerPorts for the changes in TestHDFSServerPorts. Contributed by shv

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #580 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/580/ ) MAPREDUCE-2282 . Fix TestMRServerPorts for the changes in TestHDFSServerPorts. Contributed by shv
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Konstantin!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Konstantin!
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        I also tested it manually. It fixes the build and TestMRServerPorts.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good. I also tested it manually. It fixes the build and TestMRServerPorts.
        Hide
        Konstantin Shvachko added a comment -

        This should fix the build.
        It used to be one test before project split therefore the dependency.
        I reduced the dependencies, but did not eliminate them completely.
        Basically it needs to start and stop NN and DN, which can be done using MiniDFSCluter instead of using TestHDFSServerPorts methods. Another jira?

        Show
        Konstantin Shvachko added a comment - This should fix the build. It used to be one test before project split therefore the dependency. I reduced the dependencies, but did not eliminate them completely. Basically it needs to start and stop NN and DN, which can be done using MiniDFSCluter instead of using TestHDFSServerPorts methods. Another jira?
        Hide
        Tom White added a comment -

        Thanks Nicholas. I don't think I'll have time to produce a fix today, so if someone else wants to pick this up that would be good.

        We could detect these kinds of errors by getting hudson to compile all three subprojects whenever it runs tests for one of them. Does that sound like a good change to make?

        Show
        Tom White added a comment - Thanks Nicholas. I don't think I'll have time to produce a fix today, so if someone else wants to pick this up that would be good. We could detect these kinds of errors by getting hudson to compile all three subprojects whenever it runs tests for one of them. Does that sound like a good change to make?
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Sure, either way is good.

        BTW, thanks for reporting this bug.

        Show
        Tsz Wo Nicholas Sze added a comment - Sure, either way is good. BTW, thanks for reporting this bug.
        Hide
        Tom White added a comment -

        It's certainly possible to rely only on the public HDFS API by duplicating some of the functionality of TestHDFSServerPorts in TestMRServerPorts (such as startNameNode()). Alternatively, we could mark these methods as LimitedPrivate like we did in MiniDFSCluster. I'd be happy with either approach.

        Show
        Tom White added a comment - It's certainly possible to rely only on the public HDFS API by duplicating some of the functionality of TestHDFSServerPorts in TestMRServerPorts (such as startNameNode()). Alternatively, we could mark these methods as LimitedPrivate like we did in MiniDFSCluster. I'd be happy with either approach.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I had a quick look and TestMRServerPorts tests things such as port clashes between the namenode and jobtracker, so I'm sure how feasible it is to remove the dependency.

        Use the public APIs in HDFS but not the unstable HDFS test codes. Does it sound impossible?

        Show
        Tsz Wo Nicholas Sze added a comment - I had a quick look and TestMRServerPorts tests things such as port clashes between the namenode and jobtracker, so I'm sure how feasible it is to remove the dependency. Use the public APIs in HDFS but not the unstable HDFS test codes. Does it sound impossible?
        Hide
        Tom White added a comment -

        Sorry, meant to say "I'm not sure how feasible it is to remove the dependency".

        Show
        Tom White added a comment - Sorry, meant to say "I'm not sure how feasible it is to remove the dependency".
        Hide
        Tom White added a comment -

        I had a quick look and TestMRServerPorts tests things such as port clashes between the namenode and jobtracker, so I'm sure how feasible it is to remove the dependency.

        Show
        Tom White added a comment - I had a quick look and TestMRServerPorts tests things such as port clashes between the namenode and jobtracker, so I'm sure how feasible it is to remove the dependency.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        It would be great if we can remove the dependence.

        Show
        Tsz Wo Nicholas Sze added a comment - It would be great if we can remove the dependence.

          People

          • Assignee:
            Konstantin Shvachko
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development