Hadoop Common
  1. Hadoop Common
  2. HADOOP-7143

Hive Hadoop20SShims depends on removed HadoopArchives

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.203.0
    • Fix Version/s: 0.20.203.0
    • Component/s: fs
    • Labels:
      None
    • Environment:

      RHEL5u4

    • Hadoop Flags:
      Reviewed
    • Tags:
      Hive HarFileSystem

      Description

      Compiling (Hive 0.6 + HIVE-1264) or Hive-trunk against 0.20.100 fails compilation.
      /hive/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java depends on o.a.h.tools.HadoopArchives which was removed from 0.20.100.

      HadoopArchives in turn depends on src/core/o.a.h/fs/HarFileSystem.java which was also removed from 0.20.100.

      1. hadoop-7143.patch
        52 kB
        Joep Rottinghuis

        Activity

        Hide
        Joep Rottinghuis added a comment -

        Not sure why HadoopArchives was removed, nor why HarFileSystem was removed. Was the latter replaced by something else?

        As a stop-gap I've just resusitated both classes (see patch). Is that a good idea, or is there an alternative to HadoopArchives and/or HarFileSystem that should be used instead.

        Wrt. unit test, I could not locate the accompanying test class for HadoopArchives just before HADOOP-7108 was applied, although I did find a copy here:
        http://svn.apache.org/viewvc?view=revision&revision=1040414
        although the package structure does not match 1-to-1 probably due to mapreduce split from core.
        Would it be appropriate to apply this unit test as part of this patch?

        Ditto for TestHarFileSystem found on same URL as listed above. Should I add this back in as well as part of patch for this bug?

        Show
        Joep Rottinghuis added a comment - Not sure why HadoopArchives was removed, nor why HarFileSystem was removed. Was the latter replaced by something else? As a stop-gap I've just resusitated both classes (see patch). Is that a good idea, or is there an alternative to HadoopArchives and/or HarFileSystem that should be used instead. Wrt. unit test, I could not locate the accompanying test class for HadoopArchives just before HADOOP-7108 was applied, although I did find a copy here: http://svn.apache.org/viewvc?view=revision&revision=1040414 although the package structure does not match 1-to-1 probably due to mapreduce split from core. Would it be appropriate to apply this unit test as part of this patch? Ditto for TestHarFileSystem found on same URL as listed above. Should I add this back in as well as part of patch for this bug?
        Hide
        Joep Rottinghuis added a comment -

        Patch adding back in HadoopArchives and HarFileSystem.

        Show
        Joep Rottinghuis added a comment - Patch adding back in HadoopArchives and HarFileSystem.
        Hide
        Mahadev konar added a comment -

        Joep, We had removed archives to allow folks to use multiple versions of it without being deployed on the cluster. With a patch we can add it back. Where are your getting the har classes from in your patch?

        Show
        Mahadev konar added a comment - Joep, We had removed archives to allow folks to use multiple versions of it without being deployed on the cluster. With a patch we can add it back. Where are your getting the har classes from in your patch?
        Hide
        Joep Rottinghuis added a comment -

        The intent was to get them from branch-0.20-security just before HADOOP-7108 was applied.
        When double-checking I did find fs/HarFileSystem.java is indeed the same.
        HadoopArchives however does show differences:

        678,681c678,680
        <     int ret = 0;
        < 
        <     try{
        <       ret = ToolRunner.run(harchives, args);
        ---
        >     try {
        >       int res = harchives.run(args);
        >       System.exit(res);
        683,684d681
        <       LOG.debug("Exception in archives  ", e);
        <       System.err.println("Exception in archives");
        686d682
        <       System.exit(1);
        688d683
        <     System.exit(ret);
        690c685
        < }
        ---
        > }
        \ No newline at end of file
        

        I'll look into that and rectify with new version of patch.

        Show
        Joep Rottinghuis added a comment - The intent was to get them from branch-0.20-security just before HADOOP-7108 was applied. When double-checking I did find fs/HarFileSystem.java is indeed the same. HadoopArchives however does show differences: 678,681c678,680 < int ret = 0; < < try { < ret = ToolRunner.run(harchives, args); --- > try { > int res = harchives.run(args); > System .exit(res); 683,684d681 < LOG.debug( "Exception in archives " , e); < System .err.println( "Exception in archives" ); 686d682 < System .exit(1); 688d683 < System .exit(ret); 690c685 < } --- > } \ No newline at end of file I'll look into that and rectify with new version of patch.
        Hide
        Joep Rottinghuis added a comment -

        I take that back. Both files are branch-0.20-security just before HADOOP-7108 was applied.

        Mahadev, the different versions you refer to are from HADOOP-6591 and MAPREDUCE-1585.
        How do you think we should move forward with this?
        1) Add original version of HadoopArchives back in as per this patch?
        2) Move HadoopArchives to Hive so that it has its own copy with its own version?
        3) something else

        Show
        Joep Rottinghuis added a comment - I take that back. Both files are branch-0.20-security just before HADOOP-7108 was applied. Mahadev, the different versions you refer to are from HADOOP-6591 and MAPREDUCE-1585 . How do you think we should move forward with this? 1) Add original version of HadoopArchives back in as per this patch? 2) Move HadoopArchives to Hive so that it has its own copy with its own version? 3) something else
        Hide
        Mahadev konar added a comment -

        joep, we should probably add the original version from the branch-0.20-security before hadoop-7108 was applied.

        Show
        Mahadev konar added a comment - joep, we should probably add the original version from the branch-0.20-security before hadoop-7108 was applied.
        Hide
        Owen O'Malley added a comment -

        I just committed this to 203.

        Show
        Owen O'Malley added a comment - I just committed this to 203.
        Hide
        Owen O'Malley added a comment -

        Closing for 0.20.203.0

        Show
        Owen O'Malley added a comment - Closing for 0.20.203.0

          People

          • Assignee:
            Joep Rottinghuis
            Reporter:
            Joep Rottinghuis
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development