Hive
  1. Hive
  2. HIVE-1526

Hive should depend on a release version of Thrift

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Tags:
      thrift ivy

      Description

      Hive should depend on a release version of Thrift, and ideally it should use Ivy to resolve this dependency.

      The Thrift folks are working on adding Thrift artifacts to a maven repository here: https://issues.apache.org/jira/browse/THRIFT-363

      1. thrift-fb303-0.5.0.jar
        170 kB
        Carl Steinbach
      2. thrift-0.5.0.jar
        256 kB
        Carl Steinbach
      3. test.log
        1.15 MB
        Ning Zhang
      4. svn_rm.sh
        2 kB
        Carl Steinbach
      5. serde2_test.patch
        2 kB
        Ashutosh Chauhan
      6. libthrift.jar
        227 kB
        Todd Lipcon
      7. libfb303.jar
        172 kB
        Todd Lipcon
      8. HIVE-1526-no-codegen.8.patch.txt
        489 kB
        Carl Steinbach
      9. HIVE-1526-no-codegen.7.patch.txt
        489 kB
        Carl Steinbach
      10. HIVE-1526-no-codegen.6.patch.txt
        488 kB
        Carl Steinbach
      11. HIVE-1526-no-codegen.5.patch.txt
        480 kB
        Carl Steinbach
      12. HIVE-1526-no-codegen.3.patch.txt
        478 kB
        Carl Steinbach
      13. HIVE-1526-complete.8.patch.txt
        7.15 MB
        Carl Steinbach
      14. HIVE-1526-complete.7.patch.txt
        7.13 MB
        Carl Steinbach
      15. HIVE-1526-complete.6.patch.txt
        7.13 MB
        Carl Steinbach
      16. HIVE-1526-complete.5.patch.txt
        4.21 MB
        Carl Steinbach
      17. hive-1526.txt
        3.47 MB
        Todd Lipcon
      18. HIVE-1526.3.patch.txt
        4.15 MB
        Carl Steinbach
      19. HIVE-1526.2.patch.txt
        3.93 MB
        Carl Steinbach
      20. compile.err
        24 kB
        Ning Zhang
      21. ASF.LICENSE.NOT.GRANTED--HIVE-1526-no-codegen.4.patch.txt
        480 kB
        Carl Steinbach
      22. ASF.LICENSE.NOT.GRANTED--HIVE-1526-complete.4.patch.txt
        4.21 MB
        Carl Steinbach

        Issue Links

          Activity

          Hide
          Carl Steinbach added a comment -

          @Ashutosh: Thanks for reminding me. I'll see that this gets updated.

          Show
          Carl Steinbach added a comment - @Ashutosh: Thanks for reminding me. I'll see that this gets updated.
          Hide
          Ashutosh Chauhan added a comment -

          Great !

          Carl, you noted above:

          Need to update this page once this change is committed: http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code

          Now that its in, the wiki page can now be updated as well. Otherwise, similar number of iterations will take place when we upgrade to next version of thrift

          Show
          Ashutosh Chauhan added a comment - Great ! Carl, you noted above: Need to update this page once this change is committed: http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code Now that its in, the wiki page can now be updated as well. Otherwise, similar number of iterations will take place when we upgrade to next version of thrift
          Hide
          Ning Zhang added a comment -

          Committed. Thanks Carl, Todd and Ashutosh!

          Show
          Ning Zhang added a comment - Committed. Thanks Carl, Todd and Ashutosh!
          Hide
          Ning Zhang added a comment -

          Sorry I missed Carl's last update. I'm now testing the new patch.

          Show
          Ning Zhang added a comment - Sorry I missed Carl's last update. I'm now testing the new patch.
          Hide
          Namit Jain added a comment -

          @Ning, can you take care of this ?
          So many other patches are waiting for this ?

          Show
          Namit Jain added a comment - @Ning, can you take care of this ? So many other patches are waiting for this ?
          Hide
          Carl Steinbach added a comment -

          Attaching a new version of the patch that incorporates the recent Thrift metastore changes.
          I verified that all tests pass.

          Show
          Carl Steinbach added a comment - Attaching a new version of the patch that incorporates the recent Thrift metastore changes. I verified that all tests pass.
          Hide
          Ning Zhang added a comment -

          Thanks Ashutosh and Carl! The changes look good and all unit tests have passed. However, there are conflicts after another JIRA is committed. Carl, can you pelase regenerate the patch yet another time? I'll try my best to test and commit ASAP to avoid conflicts again.

          Show
          Ning Zhang added a comment - Thanks Ashutosh and Carl! The changes look good and all unit tests have passed. However, there are conflicts after another JIRA is committed. Carl, can you pelase regenerate the patch yet another time? I'll try my best to test and commit ASAP to avoid conflicts again.
          Hide
          Carl Steinbach added a comment -

          Attaching a new patch that fixes the cpp ant targets and the various dynamic serde tests.

          Show
          Carl Steinbach added a comment - Attaching a new patch that fixes the cpp ant targets and the various dynamic serde tests.
          Hide
          Carl Steinbach added a comment -

          @Ashutosh: Thanks for the tip! I'll roll this into the next version of the patch (coming soon).

          Show
          Carl Steinbach added a comment - @Ashutosh: Thanks for the tip! I'll roll this into the next version of the patch (coming soon).
          Hide
          Ashutosh Chauhan added a comment -

          Also there are still exceptions on dynamic serde. When you run 'ant -Dthrift.home=... -Dtestcase=TestDynamicSerde' it throws org.apache.thrift.transport.TTransportException ...

          This was because of a change in Thrift from 0.3 to 0.5 in TIOStreamTransport where earlier it was returning bytesRead as -1 in case of EOF but now throws Exception. It can be fixed with following:

          Index: serde/src/java/org/apache/hadoop/hive/serde2/thrift/TCTLSeparatedProtocol.java
          ===================================================================
          --- serde/src/java/org/apache/hadoop/hive/serde2/thrift/TCTLSeparatedProtocol.java      (revision 1042747)
          +++ serde/src/java/org/apache/hadoop/hive/serde2/thrift/TCTLSeparatedProtocol.java      (working copy)
          @@ -239,6 +240,10 @@
                   }
                   tokenizer = new StringTokenizer(row, separator, true);
                 } catch (TTransportException e) {
          +        if(e.getType() == TTransportException.END_OF_FILE){
          +          tokenizer = new StringTokenizer("", separator, true);
          +          return false;
          +        }
                   e.printStackTrace();
                   tokenizer = null;
                   return false;
          

          Carl, can you include this fix in your next patch?

          Show
          Ashutosh Chauhan added a comment - Also there are still exceptions on dynamic serde. When you run 'ant -Dthrift.home=... -Dtestcase=TestDynamicSerde' it throws org.apache.thrift.transport.TTransportException ... This was because of a change in Thrift from 0.3 to 0.5 in TIOStreamTransport where earlier it was returning bytesRead as -1 in case of EOF but now throws Exception. It can be fixed with following: Index: serde/src/java/org/apache/hadoop/hive/serde2/thrift/TCTLSeparatedProtocol.java =================================================================== --- serde/src/java/org/apache/hadoop/hive/serde2/thrift/TCTLSeparatedProtocol.java (revision 1042747) +++ serde/src/java/org/apache/hadoop/hive/serde2/thrift/TCTLSeparatedProtocol.java (working copy) @@ -239,6 +240,10 @@ } tokenizer = new StringTokenizer(row, separator, true ); } catch (TTransportException e) { + if (e.getType() == TTransportException.END_OF_FILE){ + tokenizer = new StringTokenizer("", separator, true ); + return false ; + } e.printStackTrace(); tokenizer = null ; return false ; Carl, can you include this fix in your next patch?
          Hide
          Ning Zhang added a comment -

          Hi Carl, the latest patch has a compilation error building odbc (ant -Dthrift.home=... compile-cpp). I think it is due to the fact that the thrift generated code are under src/gen/thrift/gen-cpp etc. You may need to change the Makefile to reflect that.

          Also there are still exceptions on dynamic serde. When you run 'ant -Dthrift.home=... -Dtestcase=TestDynamicSerde' it throws org.apache.thrift.transport.TTransportException (they also appear in the test.log I uploaded before) although at last ant showed 'BUILD SUCCESSFUL'. In the trunk there is no such exception.

          I agree that dynamic serde itself may not as important and we are considering to deprecate it, but it is one of the few (if not the only) test cases in our unit tests that uses the thrift library. I think we should find out the reason why there are exceptions before we can commit the changes.

          Show
          Ning Zhang added a comment - Hi Carl, the latest patch has a compilation error building odbc (ant -Dthrift.home=... compile-cpp). I think it is due to the fact that the thrift generated code are under src/gen/thrift/gen-cpp etc. You may need to change the Makefile to reflect that. Also there are still exceptions on dynamic serde. When you run 'ant -Dthrift.home=... -Dtestcase=TestDynamicSerde' it throws org.apache.thrift.transport.TTransportException (they also appear in the test.log I uploaded before) although at last ant showed 'BUILD SUCCESSFUL'. In the trunk there is no such exception. I agree that dynamic serde itself may not as important and we are considering to deprecate it, but it is one of the few (if not the only) test cases in our unit tests that uses the thrift library. I think we should find out the reason why there are exceptions before we can commit the changes.
          Hide
          Ashutosh Chauhan added a comment -

          With the latest patch, I see no failures for any of bucketmapjoin6.q, join_filters.q, sample6.q and TestSemanticAnalyzerHookLoading. So, no known failures. For TestSemanticAnalyzerHookLoading, Carl did you see "FAILED: Error in semantic analysis: CTAS not supported." and assumed it failed? if so, thats a negative test case and that print is actually valid.

          Show
          Ashutosh Chauhan added a comment - With the latest patch, I see no failures for any of bucketmapjoin6.q, join_filters.q, sample6.q and TestSemanticAnalyzerHookLoading. So, no known failures. For TestSemanticAnalyzerHookLoading, Carl did you see "FAILED: Error in semantic analysis: CTAS not supported." and assumed it failed? if so, thats a negative test case and that print is actually valid.
          Hide
          Carl Steinbach added a comment -

          I observed a timeout in sample6.q and a failure in TestSemanticAnalyzerHookLoading, but was unable to reproduce these errors by running the tests individually. I'm rerunning everything again now.

          Show
          Carl Steinbach added a comment - I observed a timeout in sample6.q and a failure in TestSemanticAnalyzerHookLoading, but was unable to reproduce these errors by running the tests individually. I'm rerunning everything again now.
          Hide
          Carl Steinbach added a comment -

          @Ning: The errors in join_filters.q and bucketmapjoin6.q are due to the change in
          SMBMapJoinOperator.getType(), which I have reverted in the latest patch.

          I am currently running tests.

          Show
          Carl Steinbach added a comment - @Ning: The errors in join_filters.q and bucketmapjoin6.q are due to the change in SMBMapJoinOperator.getType(), which I have reverted in the latest patch. I am currently running tests.
          Hide
          Ashutosh Chauhan added a comment -

          Looking at the logs Ning posted, I can only see two failures in his log and they reproduce in my environment as well.

          ant test -Dtestcase=TestMinimrCliDriver -Dqfile=bucketmapjoin6.q
          ant test -Dtestcase=TestCliDriver -Dqfile=join_filters.q

          I verified that these two don't fail on current trunk. So, looks like an issue with a patch. Looks like both fail with a similar root cause of assert failure in source code (not in testcase)

          [junit] Exception: null
           [junit] java.lang.AssertionError
           [junit]     at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.initPlan(GenMapRedUtils.java:119)
           [junit]     at org.apache.hadoop.hive.ql.optimizer.GenMRRedSink1.process(GenMRRedSink1.java:76)
           [junit]     at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89)
           [junit]     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88)
           [junit]     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:55)
           [junit]     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67)
           [junit]     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67)
           [junit]     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67)
           [junit]     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67)
           [junit]     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67)
           [junit]     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67)
           [junit]     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67)
           [junit]     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67)
           [junit]     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102)
           [junit]     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:6274)
           [junit]     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6553)
           [junit]     at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
           [junit]     at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:332)
           [junit]     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:686)
           [junit]     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:142)
           [junit]     at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:216)
           [junit]     at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:589)
           [junit]     at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters(TestCliDriver.java:107)
          
          Show
          Ashutosh Chauhan added a comment - Looking at the logs Ning posted, I can only see two failures in his log and they reproduce in my environment as well. ant test -Dtestcase=TestMinimrCliDriver -Dqfile=bucketmapjoin6.q ant test -Dtestcase=TestCliDriver -Dqfile=join_filters.q I verified that these two don't fail on current trunk. So, looks like an issue with a patch. Looks like both fail with a similar root cause of assert failure in source code (not in testcase) [junit] Exception: null [junit] java.lang.AssertionError [junit] at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.initPlan(GenMapRedUtils.java:119) [junit] at org.apache.hadoop.hive.ql.optimizer.GenMRRedSink1.process(GenMRRedSink1.java:76) [junit] at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89) [junit] at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88) [junit] at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:55) [junit] at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) [junit] at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) [junit] at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) [junit] at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) [junit] at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) [junit] at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) [junit] at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) [junit] at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) [junit] at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102) [junit] at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:6274) [junit] at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6553) [junit] at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238) [junit] at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:332) [junit] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:686) [junit] at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:142) [junit] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:216) [junit] at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:589) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters(TestCliDriver.java:107)
          Hide
          Ning Zhang added a comment -

          Carl, I figured out why there is a thrift compilation error: the serde/build.xml:70 need to changed from

          executable="thrift" to executable="$

          {thrift.home}

          /bin/thrift".

          I have a different version of thrift installed on /usr/local/bin, and that was used to generate.

          Even after resolving this the unit tests failed with the same error (dynamic serde). I'm attaching the full log file here.

          Show
          Ning Zhang added a comment - Carl, I figured out why there is a thrift compilation error: the serde/build.xml:70 need to changed from executable="thrift" to executable="$ {thrift.home} /bin/thrift". I have a different version of thrift installed on /usr/local/bin, and that was used to generate. Even after resolving this the unit tests failed with the same error (dynamic serde). I'm attaching the full log file here.
          Hide
          Carl Steinbach added a comment -

          A new problem is that if you run ant -Dthrift.home=<thrift-0.5-home> thriftif and then run ant package, there will be compilation error.

          I was unable to reproduce the compilation error. Are you sure that you're using Thrift 0.5.0? Did you apply the complete patch before executing the thriftif target? Which files were modified as a result of executing the thriftif target?

          A similar problem can be reproduced by removing all svn deleted files under serde/src and run ant package.

          I don't think I understand what you did. How can you remove files that were already svn deleted? Or are you saying that you did something similar to "rm rf serde/src/gen*"? If so that's going to produce errors because in the process you also blew away protobuf generated code located in the serde/src/gen-java directory.

          I debugged it a little bit and found it may be related to the thrift generated java and java:bean under serde.

          The code in serde/src/gen-java is protobuf code, not Thrift code. This directory used to contain Thrift generated code, but that code now gets placed in the gen-javabean directory. If this directory contains any Thrift code then I think the patch was not applied correctly.

          Show
          Carl Steinbach added a comment - A new problem is that if you run ant -Dthrift.home=<thrift-0.5-home> thriftif and then run ant package, there will be compilation error. I was unable to reproduce the compilation error. Are you sure that you're using Thrift 0.5.0? Did you apply the complete patch before executing the thriftif target? Which files were modified as a result of executing the thriftif target? A similar problem can be reproduced by removing all svn deleted files under serde/src and run ant package. I don't think I understand what you did. How can you remove files that were already svn deleted? Or are you saying that you did something similar to "rm rf serde/src/gen *"? If so that's going to produce errors because in the process you also blew away protobuf generated code located in the serde/src/gen-java directory. I debugged it a little bit and found it may be related to the thrift generated java and java:bean under serde. The code in serde/src/gen-java is protobuf code, not Thrift code. This directory used to contain Thrift generated code, but that code now gets placed in the gen-javabean directory. If this directory contains any Thrift code then I think the patch was not applied correctly.
          Hide
          Ning Zhang added a comment -

          I debugged it a little bit and found it may be related to the thrift generated java and java:bean under serde.

          A new problem is that if you run ant -Dthrift.home=<thrift-0.5-home> thriftif and then run ant package, there will be compilation error.

          A similar problem can be reproduced by removing all svn deleted files under serde/src and run ant package.

          So I think the serde/build.xml should be fixed.

          Show
          Ning Zhang added a comment - I debugged it a little bit and found it may be related to the thrift generated java and java:bean under serde. A new problem is that if you run ant -Dthrift.home=<thrift-0.5-home> thriftif and then run ant package, there will be compilation error. A similar problem can be reproduced by removing all svn deleted files under serde/src and run ant package. So I think the serde/build.xml should be fixed.
          Hide
          Carl Steinbach added a comment -

          The test passes for me, but while running it does generate a stack trace that gets directed to stderr.
          I'm not sure if this should be a blocker, especially since DynamicSerde is deprecated and slated for removal.

          Show
          Carl Steinbach added a comment - The test passes for me, but while running it does generate a stack trace that gets directed to stderr. I'm not sure if this should be a blocker, especially since DynamicSerde is deprecated and slated for removal.
          Hide
          Ashutosh Chauhan added a comment -

          Ning, I don't see the failure.

          $ ant test -Dtestcase=TestCliDriver -Dqfile=implicit_cast1.q
          [junit] Done query: implicit_cast1.q
          [junit] Cleaning up TestCliDriver
          [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 17.215 sec
          

          Not sure, what might have caused the failures. Did you get any other test failures?

          Show
          Ashutosh Chauhan added a comment - Ning, I don't see the failure. $ ant test -Dtestcase=TestCliDriver -Dqfile=implicit_cast1.q [junit] Done query: implicit_cast1.q [junit] Cleaning up TestCliDriver [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 17.215 sec Not sure, what might have caused the failures. Did you get any other test failures?
          Hide
          Ning Zhang added a comment -

          The unit test failed on implicit_cast1.q (reproducible by the command ant test -Dtestcase=TestCliDriver -Dqfile=implicit_cast1.q). It seems to be caused by dynamic_serde which uses thrift.

          Carl, can you take a look? Ashutosh, did you see the same?

          Show
          Ning Zhang added a comment - The unit test failed on implicit_cast1.q (reproducible by the command ant test -Dtestcase=TestCliDriver -Dqfile=implicit_cast1.q). It seems to be caused by dynamic_serde which uses thrift. Carl, can you take a look? Ashutosh, did you see the same?
          Hide
          Ning Zhang added a comment -

          Thanks for debugging for me Ashutosh! It was due to my fault: somehow my thrift-0.5.0.jar is the same as thrift-fb303-0.5.0.jar. Now it compiles correctly and I started testing.

          Show
          Ning Zhang added a comment - Thanks for debugging for me Ashutosh! It was due to my fault: somehow my thrift-0.5.0.jar is the same as thrift-fb303-0.5.0.jar. Now it compiles correctly and I started testing.
          Hide
          Ashutosh Chauhan added a comment -

          I greped for one of the classes which are reported as being not found in your compile.err

          $ jar -tvf lib/thrift-0.5.0.jar | grep  TBaseHelper.class
           6739 Sun Nov 07 23:34:36 PST 2010 org/apache/thrift/TBaseHelper.class
          $ jar -tvf libthrift.jar | grep  TBaseHelper.class
          $
          

          So, it seems like in your class path still libthrift.jar is getting picked up somehow instead of thrift-0.5.0.jar Can you try completely deleting libthrift.jar and libfb303.jar from your $

          {HIVE_ROOT}

          / dir and then just have thrift-0.5.0.jar and thrift-fb303-0.5.0.jar in your lib/ dir and then build again.

          Show
          Ashutosh Chauhan added a comment - I greped for one of the classes which are reported as being not found in your compile.err $ jar -tvf lib/thrift-0.5.0.jar | grep TBaseHelper.class 6739 Sun Nov 07 23:34:36 PST 2010 org/apache/thrift/TBaseHelper.class $ jar -tvf libthrift.jar | grep TBaseHelper.class $ So, it seems like in your class path still libthrift.jar is getting picked up somehow instead of thrift-0.5.0.jar Can you try completely deleting libthrift.jar and libfb303.jar from your $ {HIVE_ROOT} / dir and then just have thrift-0.5.0.jar and thrift-fb303-0.5.0.jar in your lib/ dir and then build again.
          Hide
          Ning Zhang added a comment -

          Attached the complete output of 'ant package: compile.err. Also note that the patch still include the lib/libthrift.jar and lib/libfb303.jar which are the old versions. But the error is the same even though replacing libthrft.jar and libfb303.jar with thrift-0.5.0.jar and thrift-fb303-0.5.0.jar.

          Show
          Ning Zhang added a comment - Attached the complete output of 'ant package: compile.err. Also note that the patch still include the lib/libthrift.jar and lib/libfb303.jar which are the old versions. But the error is the same even though replacing libthrft.jar and libfb303.jar with thrift-0.5.0.jar and thrift-fb303-0.5.0.jar.
          Hide
          Ashutosh Chauhan added a comment -

          Hey Ning,

          Can you paste the full stack trace? I did the following and everything worked as expected:

          $ svn co http://svn.apache.org/repos/asf/hive/trunk .
          $ curl https://issues.apache.org/jira/secure/attachment/12464952/HIVE-1526-complete.5.patch.txt | patch -p0
          $ cd lib/
          $ wget https://issues.apache.org/jira/secure/attachment/12459901/thrift-fb303-0.5.0.jar 
          $ wget https://issues.apache.org/jira/secure/attachment/12459901/thrift-fb303-0.5.0.jar
          $ cd ..
          $ ant package
          BUILD SUCCESSFUL
          $ ant test
          

          Now running the tests. Havent seen any failure yet. Will post back when tests finish.

          Show
          Ashutosh Chauhan added a comment - Hey Ning, Can you paste the full stack trace? I did the following and everything worked as expected: $ svn co http: //svn.apache.org/repos/asf/hive/trunk . $ curl https: //issues.apache.org/jira/secure/attachment/12464952/HIVE-1526-complete.5.patch.txt | patch -p0 $ cd lib/ $ wget https: //issues.apache.org/jira/secure/attachment/12459901/thrift-fb303-0.5.0.jar $ wget https: //issues.apache.org/jira/secure/attachment/12459901/thrift-fb303-0.5.0.jar $ cd .. $ ant package BUILD SUCCESSFUL $ ant test Now running the tests. Havent seen any failure yet. Will post back when tests finish.
          Hide
          Ning Zhang added a comment -

          hi Carl, I tried compiling based on your complete patch and it gave compilation error related to thrift (something like serde/src/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java:29: type org.apache.thrift.TBase does not take parameters). I did copied the thrift and fb303 jars to lib.

          I also tried to generate thrift files from your non-generated patch. The compilation also got some error. So do you need to do settings with the patches?

          Show
          Ning Zhang added a comment - hi Carl, I tried compiling based on your complete patch and it gave compilation error related to thrift (something like serde/src/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java:29: type org.apache.thrift.TBase does not take parameters). I did copied the thrift and fb303 jars to lib. I also tried to generate thrift files from your non-generated patch. The compilation also got some error. So do you need to do settings with the patches?
          Hide
          Ning Zhang added a comment -

          Thanks Carl. I'll start testing tonight.

          Show
          Ning Zhang added a comment - Thanks Carl. I'll start testing tonight.
          Hide
          Carl Steinbach added a comment -

          @Ning: I had the same problem downloading the patch so it must be corrupted. I attached new copies and verified that I can download them. Please try again.

          Show
          Carl Steinbach added a comment - @Ning: I had the same problem downloading the patch so it must be corrupted. I attached new copies and verified that I can download them. Please try again.
          Hide
          Ning Zhang added a comment -

          The patch looks good from the cloudera review board. However there's system errors when I tried to download the latest patches. Carl, I'm not sure if this is a transient error, but if it persist tomorrow, can you send me the patch by email (nzhang@fb.com)?

          Show
          Ning Zhang added a comment - The patch looks good from the cloudera review board. However there's system errors when I tried to download the latest patches. Carl, I'm not sure if this is a transient error, but if it persist tomorrow, can you send me the patch by email (nzhang@fb.com)?
          Hide
          HBase Review Board added a comment -

          Message from: "Carl Steinbach" <carl@cloudera.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.cloudera.org/r/1242/
          -----------------------------------------------------------

          (Updated 2010-11-29 02:30:23.538215)

          Review request for Hive Developers.

          Changes
          -------

          Patch updated with changes suggested by Ning and Ashutosh.

          Summary
          -------

          Review request for HIVE-1526: upgrade to Thrift 0.5.0

          This review request does not include the code generated by the Thrift compiler.

          This addresses bug HIVE-1526.
          http://issues.apache.org/jira/browse/HIVE-1526

          Diffs (updated)


          .classpath._hbase 15fd1c5
          .gitignore e54804f
          build-common.xml 53fd1b1
          build.xml dc20725
          eclipse-templates/.classpath d06d6bd
          eclipse-templates/.classpath._hbase b81c24a
          ivy/libraries.properties fdd54c8
          lib/README 72d90b7
          lib/libfb303.LICENSE 6b5c3b1
          lib/libfb303.jar 516b597
          lib/libthrift.LICENSE 6b5c3b1
          lib/libthrift.jar 7e7f90e
          lib/thrift-0.5.0.jar PRE-CREATION
          lib/thrift-fb303-0.5.0.jar PRE-CREATION
          lib/thrift-fb303.LICENSE PRE-CREATION
          lib/thrift.LICENSE PRE-CREATION
          metastore/build.xml ffb86c1
          metastore/if/hive_metastore.thrift 0e47312
          metastore/ivy.xml 2e39eb8
          metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 96ef5b7
          metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 4c977b8
          odbc/Makefile 816ffd4
          odbc/build.xml 64cf988
          odbc/src/cpp/HiveRowSet.cpp b4dc2c0
          odbc/src/cpp/hiveclienthelper.cpp e2d48f3
          ql/build.xml a025a2b
          ql/if/queryplan.thrift aca0e8f
          ql/ivy.xml e72a450
          ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 5ccb909
          ql/src/java/org/apache/hadoop/hive/ql/exec/CollectOperator.java 7daea2d
          ql/src/java/org/apache/hadoop/hive/ql/exec/ConditionalTask.java bf7b7a5
          ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 46b8bf4
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 7168884
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 49e708c
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java a572bcb
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java 7bf8886
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java de2c42d
          ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java d510f95
          ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java 554bf25
          ql/src/java/org/apache/hadoop/hive/ql/exec/ForwardOperator.java 2a35c96
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 1e6941e
          ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 8423178
          ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableDummyOperator.java 87fc61c
          ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 4cf055f
          ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java e16df36
          ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewForwardOperator.java c070aca
          ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java 5eb9098
          ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java da5dbb9
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java c9f99ac
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 887ad30
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java 0fab63c
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java c5aa3d6
          ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7fbc586
          ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 6c6ea89
          ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 592f4f7
          ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 001d1f0
          ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 4f8b4bd
          ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java d644a33
          ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 47adbed
          ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 96e63b8
          ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 893530c
          ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java 682f38a
          ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 2c42ed7
          ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 0d60507
          ql/src/java/org/apache/hadoop/hive/ql/index/compact/IndexMetadataChangeTask.java 734bd72
          ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 5d33550
          ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 7167196
          serde/build.xml b64b142
          serde/ivy.xml da69342
          serde/src/java/org/apache/hadoop/hive/serde2/thrift/TBinarySortableProtocol.java 961c24f
          serde/src/java/org/apache/hadoop/hive/serde2/thrift/TCTLSeparatedProtocol.java e781a06
          serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorUtils.java 4bdcf88
          serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestThriftObjectInspectors.java 99b0e90
          service/build.xml ffd1854
          service/if/hive_service.thrift db9fab6
          service/include/thrift/TLogging.h 2df82dd
          service/include/thrift/TProcessor.h f2d5279
          service/include/thrift/TReflectionLocal.h e83e475
          service/include/thrift/Thrift.h 27a6476
          service/include/thrift/concurrency/Exception.h ec46629
          service/include/thrift/concurrency/FunctionRunner.h 2216927
          service/include/thrift/concurrency/Monitor.h 234bf32
          service/include/thrift/concurrency/Mutex.h 73c73e0
          service/include/thrift/concurrency/PosixThreadFactory.h d6d83a3
          service/include/thrift/concurrency/Thread.h d4282ad
          service/include/thrift/concurrency/ThreadManager.h 6e5a178
          service/include/thrift/concurrency/TimerManager.h dfbf0ea
          service/include/thrift/concurrency/Util.h 25fcc20
          service/include/thrift/config.h 70264f2
          service/include/thrift/fb303/FacebookBase.h fd169e6
          service/include/thrift/fb303/FacebookService.h dcd843f
          service/include/thrift/fb303/FacebookService.cpp 4a34362
          service/include/thrift/fb303/ServiceTracker.h 9304386
          service/include/thrift/fb303/fb303_constants.h 19405bc
          service/include/thrift/fb303/fb303_constants.cpp 66b8782
          service/include/thrift/fb303/fb303_types.h 18936f9
          service/include/thrift/fb303/fb303_types.cpp f290125
          service/include/thrift/fb303/if/fb303.thrift 66c8315
          service/include/thrift/if/reflection_limited.thrift 993c46e
          service/include/thrift/processor/PeekProcessor.h 0f7c016
          service/include/thrift/processor/StatsProcessor.h 820b3ad
          service/include/thrift/protocol/TBase64Utils.h 3def733
          service/include/thrift/protocol/TBinaryProtocol.h 7fd3de6
          service/include/thrift/protocol/TCompactProtocol.h b4e06f0
          service/include/thrift/protocol/TDebugProtocol.h ab69e0c
          service/include/thrift/protocol/TDenseProtocol.h 7655a47
          service/include/thrift/protocol/TJSONProtocol.h 2df499a
          service/include/thrift/protocol/TOneWayProtocol.h 6f08fe1
          service/include/thrift/protocol/TProtocol.h 4025827
          service/include/thrift/protocol/TProtocolException.h 33011b3
          service/include/thrift/protocol/TProtocolTap.h 5580216
          service/include/thrift/reflection_limited_types.h 677cb4a
          service/include/thrift/server/TNonblockingServer.h 8506507
          service/include/thrift/server/TServer.h 5c4c588
          service/include/thrift/server/TSimpleServer.h c4fc91c
          service/include/thrift/server/TThreadPoolServer.h 7b7e906
          service/include/thrift/server/TThreadedServer.h 4d0811a
          service/include/thrift/transport/TBufferTransports.h 1908205
          service/include/thrift/transport/TFDTransport.h bda5d82
          service/include/thrift/transport/TFileTransport.h b08c5c8
          service/include/thrift/transport/THttpClient.h f4be4c1
          service/include/thrift/transport/TServerSocket.h a6be017
          service/include/thrift/transport/TServerTransport.h 40bbc6c
          service/include/thrift/transport/TShortReadTransport.h 3df8a57
          service/include/thrift/transport/TSimpleFileTransport.h 6cc52ea
          service/include/thrift/transport/TSocket.h b0f445a
          service/include/thrift/transport/TSocketPool.h 8c50669
          service/include/thrift/transport/TTransport.h eb0d5df
          service/include/thrift/transport/TTransportException.h 330785c
          service/include/thrift/transport/TTransportUtils.h d65c916
          service/include/thrift/transport/TZlibTransport.h 1439d9d
          service/src/java/org/apache/hadoop/hive/service/HiveServer.java 89cf336

          Diff: http://review.cloudera.org/r/1242/diff

          Testing
          -------

          Thanks,

          Carl

          Show
          HBase Review Board added a comment - Message from: "Carl Steinbach" <carl@cloudera.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/1242/ ----------------------------------------------------------- (Updated 2010-11-29 02:30:23.538215) Review request for Hive Developers. Changes ------- Patch updated with changes suggested by Ning and Ashutosh. Summary ------- Review request for HIVE-1526 : upgrade to Thrift 0.5.0 This review request does not include the code generated by the Thrift compiler. This addresses bug HIVE-1526 . http://issues.apache.org/jira/browse/HIVE-1526 Diffs (updated) .classpath._hbase 15fd1c5 .gitignore e54804f build-common.xml 53fd1b1 build.xml dc20725 eclipse-templates/.classpath d06d6bd eclipse-templates/.classpath._hbase b81c24a ivy/libraries.properties fdd54c8 lib/README 72d90b7 lib/libfb303.LICENSE 6b5c3b1 lib/libfb303.jar 516b597 lib/libthrift.LICENSE 6b5c3b1 lib/libthrift.jar 7e7f90e lib/thrift-0.5.0.jar PRE-CREATION lib/thrift-fb303-0.5.0.jar PRE-CREATION lib/thrift-fb303.LICENSE PRE-CREATION lib/thrift.LICENSE PRE-CREATION metastore/build.xml ffb86c1 metastore/if/hive_metastore.thrift 0e47312 metastore/ivy.xml 2e39eb8 metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 96ef5b7 metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 4c977b8 odbc/Makefile 816ffd4 odbc/build.xml 64cf988 odbc/src/cpp/HiveRowSet.cpp b4dc2c0 odbc/src/cpp/hiveclienthelper.cpp e2d48f3 ql/build.xml a025a2b ql/if/queryplan.thrift aca0e8f ql/ivy.xml e72a450 ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 5ccb909 ql/src/java/org/apache/hadoop/hive/ql/exec/CollectOperator.java 7daea2d ql/src/java/org/apache/hadoop/hive/ql/exec/ConditionalTask.java bf7b7a5 ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 46b8bf4 ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 7168884 ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 49e708c ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java a572bcb ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java 7bf8886 ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java de2c42d ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java d510f95 ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java 554bf25 ql/src/java/org/apache/hadoop/hive/ql/exec/ForwardOperator.java 2a35c96 ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 1e6941e ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 8423178 ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableDummyOperator.java 87fc61c ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 4cf055f ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java e16df36 ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewForwardOperator.java c070aca ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java 5eb9098 ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java da5dbb9 ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java c9f99ac ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 887ad30 ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java 0fab63c ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java c5aa3d6 ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7fbc586 ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 6c6ea89 ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 592f4f7 ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 001d1f0 ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 4f8b4bd ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java d644a33 ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 47adbed ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 96e63b8 ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 893530c ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java 682f38a ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 2c42ed7 ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 0d60507 ql/src/java/org/apache/hadoop/hive/ql/index/compact/IndexMetadataChangeTask.java 734bd72 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 5d33550 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 7167196 serde/build.xml b64b142 serde/ivy.xml da69342 serde/src/java/org/apache/hadoop/hive/serde2/thrift/TBinarySortableProtocol.java 961c24f serde/src/java/org/apache/hadoop/hive/serde2/thrift/TCTLSeparatedProtocol.java e781a06 serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorUtils.java 4bdcf88 serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestThriftObjectInspectors.java 99b0e90 service/build.xml ffd1854 service/if/hive_service.thrift db9fab6 service/include/thrift/TLogging.h 2df82dd service/include/thrift/TProcessor.h f2d5279 service/include/thrift/TReflectionLocal.h e83e475 service/include/thrift/Thrift.h 27a6476 service/include/thrift/concurrency/Exception.h ec46629 service/include/thrift/concurrency/FunctionRunner.h 2216927 service/include/thrift/concurrency/Monitor.h 234bf32 service/include/thrift/concurrency/Mutex.h 73c73e0 service/include/thrift/concurrency/PosixThreadFactory.h d6d83a3 service/include/thrift/concurrency/Thread.h d4282ad service/include/thrift/concurrency/ThreadManager.h 6e5a178 service/include/thrift/concurrency/TimerManager.h dfbf0ea service/include/thrift/concurrency/Util.h 25fcc20 service/include/thrift/config.h 70264f2 service/include/thrift/fb303/FacebookBase.h fd169e6 service/include/thrift/fb303/FacebookService.h dcd843f service/include/thrift/fb303/FacebookService.cpp 4a34362 service/include/thrift/fb303/ServiceTracker.h 9304386 service/include/thrift/fb303/fb303_constants.h 19405bc service/include/thrift/fb303/fb303_constants.cpp 66b8782 service/include/thrift/fb303/fb303_types.h 18936f9 service/include/thrift/fb303/fb303_types.cpp f290125 service/include/thrift/fb303/if/fb303.thrift 66c8315 service/include/thrift/if/reflection_limited.thrift 993c46e service/include/thrift/processor/PeekProcessor.h 0f7c016 service/include/thrift/processor/StatsProcessor.h 820b3ad service/include/thrift/protocol/TBase64Utils.h 3def733 service/include/thrift/protocol/TBinaryProtocol.h 7fd3de6 service/include/thrift/protocol/TCompactProtocol.h b4e06f0 service/include/thrift/protocol/TDebugProtocol.h ab69e0c service/include/thrift/protocol/TDenseProtocol.h 7655a47 service/include/thrift/protocol/TJSONProtocol.h 2df499a service/include/thrift/protocol/TOneWayProtocol.h 6f08fe1 service/include/thrift/protocol/TProtocol.h 4025827 service/include/thrift/protocol/TProtocolException.h 33011b3 service/include/thrift/protocol/TProtocolTap.h 5580216 service/include/thrift/reflection_limited_types.h 677cb4a service/include/thrift/server/TNonblockingServer.h 8506507 service/include/thrift/server/TServer.h 5c4c588 service/include/thrift/server/TSimpleServer.h c4fc91c service/include/thrift/server/TThreadPoolServer.h 7b7e906 service/include/thrift/server/TThreadedServer.h 4d0811a service/include/thrift/transport/TBufferTransports.h 1908205 service/include/thrift/transport/TFDTransport.h bda5d82 service/include/thrift/transport/TFileTransport.h b08c5c8 service/include/thrift/transport/THttpClient.h f4be4c1 service/include/thrift/transport/TServerSocket.h a6be017 service/include/thrift/transport/TServerTransport.h 40bbc6c service/include/thrift/transport/TShortReadTransport.h 3df8a57 service/include/thrift/transport/TSimpleFileTransport.h 6cc52ea service/include/thrift/transport/TSocket.h b0f445a service/include/thrift/transport/TSocketPool.h 8c50669 service/include/thrift/transport/TTransport.h eb0d5df service/include/thrift/transport/TTransportException.h 330785c service/include/thrift/transport/TTransportUtils.h d65c916 service/include/thrift/transport/TZlibTransport.h 1439d9d service/src/java/org/apache/hadoop/hive/service/HiveServer.java 89cf336 Diff: http://review.cloudera.org/r/1242/diff Testing ------- Thanks, Carl
          Hide
          HBase Review Board added a comment -

          Message from: "Carl Steinbach" <carl@cloudera.com>

          On 2010-11-22 14:37:20, Ning Zhang wrote:

          > ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java, line 416

          > <http://review.cloudera.org/r/1242/diff/1/?file=17504#file17504line416>

          >

          > This should not be MAPREDLOCAL, but MAPRED. Since MapRedTask extends ExecDriver which implements getType(), this function should not be here in MapRedTask.

          Removed.

          On 2010-11-22 14:37:20, Ning Zhang wrote:

          > ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java, line 592

          > <http://review.cloudera.org/r/1242/diff/1/?file=17509#file17509line592>

          >

          > It would be better to define a new OperatorType for SMBMapJoinOperator.

          Done.

          On 2010-11-22 14:37:20, Ning Zhang wrote:

          > ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java, line 348

          > <http://review.cloudera.org/r/1242/diff/1/?file=17514#file17514line348>

          >

          > commnet: -1 -> null

          I made this method abstract.

          • Carl

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.cloudera.org/r/1242/#review1958
          -----------------------------------------------------------

          Show
          HBase Review Board added a comment - Message from: "Carl Steinbach" <carl@cloudera.com> On 2010-11-22 14:37:20, Ning Zhang wrote: > ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java, line 416 > < http://review.cloudera.org/r/1242/diff/1/?file=17504#file17504line416 > > > This should not be MAPREDLOCAL, but MAPRED. Since MapRedTask extends ExecDriver which implements getType(), this function should not be here in MapRedTask. Removed. On 2010-11-22 14:37:20, Ning Zhang wrote: > ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java, line 592 > < http://review.cloudera.org/r/1242/diff/1/?file=17509#file17509line592 > > > It would be better to define a new OperatorType for SMBMapJoinOperator. Done. On 2010-11-22 14:37:20, Ning Zhang wrote: > ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java, line 348 > < http://review.cloudera.org/r/1242/diff/1/?file=17514#file17514line348 > > > commnet: -1 -> null I made this method abstract. Carl ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/1242/#review1958 -----------------------------------------------------------
          Hide
          Carl Steinbach added a comment -

          Updated patch with changes suggested by Ning and Ashutosh.

          Show
          Carl Steinbach added a comment - Updated patch with changes suggested by Ning and Ashutosh.
          Hide
          Carl Steinbach added a comment -

          @Ning: working on it now.

          Show
          Carl Steinbach added a comment - @Ning: working on it now.
          Hide
          Ning Zhang added a comment -

          Carl, can you upload a new patch taking consideration of my other comments? I'll start test.

          Show
          Ning Zhang added a comment - Carl, can you upload a new patch taking consideration of my other comments? I'll start test.
          Hide
          He Yongqiang added a comment -

          ok. i am fine to get this committed before HIVE-78.

          Show
          He Yongqiang added a comment - ok. i am fine to get this committed before HIVE-78 .
          Hide
          Devaraj Das added a comment -

          Guys, it will be really appreciated if this patch can be committed now. This blocks the other jiras on Security - HIVE-842 and HIVE-1696. Since Carl and others have done so much work on the thrift patch already, I think it makes sense to have this patch committed now. Thanks!

          Show
          Devaraj Das added a comment - Guys, it will be really appreciated if this patch can be committed now. This blocks the other jiras on Security - HIVE-842 and HIVE-1696 . Since Carl and others have done so much work on the thrift patch already, I think it makes sense to have this patch committed now. Thanks!
          Hide
          Carl Steinbach added a comment -

          @Yongqiang: that's fine with me, assuming that HIVE-78 is close to getting committed. I think this ticket is blocking some of the security work.

          Show
          Carl Steinbach added a comment - @Yongqiang: that's fine with me, assuming that HIVE-78 is close to getting committed. I think this ticket is blocking some of the security work.
          Hide
          He Yongqiang added a comment -

          can we get this in after HIVE-78 since this is mostly auto-generated files?

          Show
          He Yongqiang added a comment - can we get this in after HIVE-78 since this is mostly auto-generated files?
          Hide
          Ning Zhang added a comment -

          I see. That's OK then.

          Show
          Ning Zhang added a comment - I see. That's OK then.
          Hide
          Ning Zhang added a comment -

          I see. That's OK then.

          Show
          Ning Zhang added a comment - I see. That's OK then.
          Hide
          HBase Review Board added a comment -

          Message from: "Carl Steinbach" <carl@cloudera.com>

          On 2010-11-22 14:37:20, Ning Zhang wrote:

          > .classpath._hbase, line 1

          > <http://review.cloudera.org/r/1242/diff/1/?file=17455#file17455line1>

          >

          > Is deleting of this file because of we eliminated hadoop pre-0.20 versions?

          >

          > John, can you confirm if this file can be deleted?

          This is an eclipse project classpath file. It should not have been checked in.

          • Carl

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.cloudera.org/r/1242/#review1958
          -----------------------------------------------------------

          Show
          HBase Review Board added a comment - Message from: "Carl Steinbach" <carl@cloudera.com> On 2010-11-22 14:37:20, Ning Zhang wrote: > .classpath._hbase, line 1 > < http://review.cloudera.org/r/1242/diff/1/?file=17455#file17455line1 > > > Is deleting of this file because of we eliminated hadoop pre-0.20 versions? > > John, can you confirm if this file can be deleted? This is an eclipse project classpath file. It should not have been checked in. Carl ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/1242/#review1958 -----------------------------------------------------------
          Hide
          HBase Review Board added a comment -

          Message from: "Ning Zhang" <n.ning.z@gmail.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.cloudera.org/r/1242/#review1958
          -----------------------------------------------------------

          .classpath._hbase
          <http://review.cloudera.org/r/1242/#comment6177>

          Is deleting of this file because of we eliminated hadoop pre-0.20 versions?

          John, can you confirm if this file can be deleted?

          ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java
          <http://review.cloudera.org/r/1242/#comment6207>

          This should not be MAPREDLOCAL, but MAPRED. Since MapRedTask extends ExecDriver which implements getType(), this function should not be here in MapRedTask.

          ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
          <http://review.cloudera.org/r/1242/#comment6208>

          It would be better to define a new OperatorType for SMBMapJoinOperator.

          ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
          <http://review.cloudera.org/r/1242/#comment6209>

          commnet: -1 -> null

          • Ning
          Show
          HBase Review Board added a comment - Message from: "Ning Zhang" <n.ning.z@gmail.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/1242/#review1958 ----------------------------------------------------------- .classpath._hbase < http://review.cloudera.org/r/1242/#comment6177 > Is deleting of this file because of we eliminated hadoop pre-0.20 versions? John, can you confirm if this file can be deleted? ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java < http://review.cloudera.org/r/1242/#comment6207 > This should not be MAPREDLOCAL, but MAPRED. Since MapRedTask extends ExecDriver which implements getType(), this function should not be here in MapRedTask. ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java < http://review.cloudera.org/r/1242/#comment6208 > It would be better to define a new OperatorType for SMBMapJoinOperator. ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java < http://review.cloudera.org/r/1242/#comment6209 > commnet: -1 -> null Ning
          Hide
          Ashutosh Chauhan added a comment -

          I was testing this patch. It seems that fields defined in structs in thrift idl, thrift now generates them as private instead of as public in previous releases. As a result, tests in serde2 don't compile. This additional patch is required to compile the test sources in serde2/ .

          Show
          Ashutosh Chauhan added a comment - I was testing this patch. It seems that fields defined in structs in thrift idl, thrift now generates them as private instead of as public in previous releases. As a result, tests in serde2 don't compile. This additional patch is required to compile the test sources in serde2/ .
          Hide
          Ning Zhang added a comment -

          Thanks Carl for the work. I'll be returning to Hive the coming week. I'll review this JIRA as the highest priority then.

          Show
          Ning Zhang added a comment - Thanks Carl for the work. I'll be returning to Hive the coming week. I'll review this JIRA as the highest priority then.
          Hide
          Carl Steinbach added a comment -

          @Ning: Can you please take a look at this? Thanks!

          Show
          Carl Steinbach added a comment - @Ning: Can you please take a look at this? Thanks!
          Hide
          HBase Review Board added a comment -

          Message from: "Carl Steinbach" <carl@cloudera.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.cloudera.org/r/1242/
          -----------------------------------------------------------

          Review request for Hive Developers.

          Summary
          -------

          Review request for HIVE-1526: upgrade to Thrift 0.5.0

          This review request does not include the code generated by the Thrift compiler.

          This addresses bug HIVE-1526.
          http://issues.apache.org/jira/browse/HIVE-1526

          Diffs


          .classpath._hbase 15fd1c5
          build-common.xml 53fd1b1
          build.xml dc20725
          eclipse-templates/.classpath d06d6bd
          eclipse-templates/.classpath._hbase b81c24a
          ivy/libraries.properties fdd54c8
          lib/README 72d90b7
          lib/libfb303.LICENSE 6b5c3b1
          lib/libfb303.jar 516b597
          lib/libthrift.LICENSE 6b5c3b1
          lib/libthrift.jar 7e7f90e
          lib/thrift-0.5.0.jar PRE-CREATION
          lib/thrift-fb303-0.5.0.jar PRE-CREATION
          lib/thrift-fb303.LICENSE PRE-CREATION
          lib/thrift.LICENSE PRE-CREATION
          metastore/build.xml ffb86c1
          metastore/if/hive_metastore.thrift 9c314cc
          metastore/ivy.xml 2e39eb8
          metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java a4f1853
          metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 4f5623c
          odbc/Makefile 816ffd4
          odbc/build.xml 64cf988
          odbc/src/cpp/HiveRowSet.cpp b4dc2c0
          odbc/src/cpp/hiveclienthelper.cpp e2d48f3
          ql/build.xml a025a2b
          ql/if/queryplan.thrift aca0e8f
          ql/ivy.xml e72a450
          ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 5ccb909
          ql/src/java/org/apache/hadoop/hive/ql/exec/CollectOperator.java 7daea2d
          ql/src/java/org/apache/hadoop/hive/ql/exec/ConditionalTask.java bf7b7a5
          ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 46b8bf4
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 0a9cdf4
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java af6e0af
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java a572bcb
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java 7bf8886
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java de2c42d
          ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java d510f95
          ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java 554bf25
          ql/src/java/org/apache/hadoop/hive/ql/exec/ForwardOperator.java 2a35c96
          ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 1e6941e
          ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 8423178
          ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableDummyOperator.java 87fc61c
          ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 9fe35ec
          ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java e16df36
          ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewForwardOperator.java c070aca
          ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java 5eb9098
          ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java da5dbb9
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 98571dd
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 887ad30
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java 0fab63c
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java c5aa3d6
          ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7fbc586
          ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 6c6ea89
          ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 592f4f7
          ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 001d1f0
          ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 4f8b4bd
          ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java d644a33
          ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 47adbed
          ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 96e63b8
          ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 893530c
          ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java 682f38a
          ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 2c42ed7
          ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 0d60507
          ql/src/java/org/apache/hadoop/hive/ql/index/compact/IndexMetadataChangeTask.java 734bd72
          ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c5fb22b
          ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 7167196
          serde/build.xml b64b142
          serde/ivy.xml da69342
          serde/src/java/org/apache/hadoop/hive/serde2/thrift/TBinarySortableProtocol.java 961c24f
          serde/src/java/org/apache/hadoop/hive/serde2/thrift/TCTLSeparatedProtocol.java e781a06
          service/build.xml ffd1854
          service/if/hive_service.thrift db9fab6
          service/include/thrift/TLogging.h 2df82dd
          service/include/thrift/TProcessor.h f2d5279
          service/include/thrift/TReflectionLocal.h e83e475
          service/include/thrift/Thrift.h 27a6476
          service/include/thrift/concurrency/Exception.h ec46629
          service/include/thrift/concurrency/FunctionRunner.h 2216927
          service/include/thrift/concurrency/Monitor.h 234bf32
          service/include/thrift/concurrency/Mutex.h 73c73e0
          service/include/thrift/concurrency/PosixThreadFactory.h d6d83a3
          service/include/thrift/concurrency/Thread.h d4282ad
          service/include/thrift/concurrency/ThreadManager.h 6e5a178
          service/include/thrift/concurrency/TimerManager.h dfbf0ea
          service/include/thrift/concurrency/Util.h 25fcc20
          service/include/thrift/config.h 70264f2
          service/include/thrift/fb303/FacebookBase.h fd169e6
          service/include/thrift/fb303/FacebookService.h dcd843f
          service/include/thrift/fb303/FacebookService.cpp 4a34362
          service/include/thrift/fb303/ServiceTracker.h 9304386
          service/include/thrift/fb303/fb303_constants.h 19405bc
          service/include/thrift/fb303/fb303_constants.cpp 66b8782
          service/include/thrift/fb303/fb303_types.h 18936f9
          service/include/thrift/fb303/fb303_types.cpp f290125
          service/include/thrift/fb303/if/fb303.thrift 66c8315
          service/include/thrift/if/reflection_limited.thrift 993c46e
          service/include/thrift/processor/PeekProcessor.h 0f7c016
          service/include/thrift/processor/StatsProcessor.h 820b3ad
          service/include/thrift/protocol/TBase64Utils.h 3def733
          service/include/thrift/protocol/TBinaryProtocol.h 7fd3de6
          service/include/thrift/protocol/TCompactProtocol.h b4e06f0
          service/include/thrift/protocol/TDebugProtocol.h ab69e0c
          service/include/thrift/protocol/TDenseProtocol.h 7655a47
          service/include/thrift/protocol/TJSONProtocol.h 2df499a
          service/include/thrift/protocol/TOneWayProtocol.h 6f08fe1
          service/include/thrift/protocol/TProtocol.h 4025827
          service/include/thrift/protocol/TProtocolException.h 33011b3
          service/include/thrift/protocol/TProtocolTap.h 5580216
          service/include/thrift/reflection_limited_types.h 677cb4a
          service/include/thrift/server/TNonblockingServer.h 8506507
          service/include/thrift/server/TServer.h 5c4c588
          service/include/thrift/server/TSimpleServer.h c4fc91c
          service/include/thrift/server/TThreadPoolServer.h 7b7e906
          service/include/thrift/server/TThreadedServer.h 4d0811a
          service/include/thrift/transport/TBufferTransports.h 1908205
          service/include/thrift/transport/TFDTransport.h bda5d82
          service/include/thrift/transport/TFileTransport.h b08c5c8
          service/include/thrift/transport/THttpClient.h f4be4c1
          service/include/thrift/transport/TServerSocket.h a6be017
          service/include/thrift/transport/TServerTransport.h 40bbc6c
          service/include/thrift/transport/TShortReadTransport.h 3df8a57
          service/include/thrift/transport/TSimpleFileTransport.h 6cc52ea
          service/include/thrift/transport/TSocket.h b0f445a
          service/include/thrift/transport/TSocketPool.h 8c50669
          service/include/thrift/transport/TTransport.h eb0d5df
          service/include/thrift/transport/TTransportException.h 330785c
          service/include/thrift/transport/TTransportUtils.h d65c916
          service/include/thrift/transport/TZlibTransport.h 1439d9d
          service/src/java/org/apache/hadoop/hive/service/HiveServer.java 89cf336

          Diff: http://review.cloudera.org/r/1242/diff

          Testing
          -------

          Thanks,

          Carl

          Show
          HBase Review Board added a comment - Message from: "Carl Steinbach" <carl@cloudera.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/1242/ ----------------------------------------------------------- Review request for Hive Developers. Summary ------- Review request for HIVE-1526 : upgrade to Thrift 0.5.0 This review request does not include the code generated by the Thrift compiler. This addresses bug HIVE-1526 . http://issues.apache.org/jira/browse/HIVE-1526 Diffs .classpath._hbase 15fd1c5 build-common.xml 53fd1b1 build.xml dc20725 eclipse-templates/.classpath d06d6bd eclipse-templates/.classpath._hbase b81c24a ivy/libraries.properties fdd54c8 lib/README 72d90b7 lib/libfb303.LICENSE 6b5c3b1 lib/libfb303.jar 516b597 lib/libthrift.LICENSE 6b5c3b1 lib/libthrift.jar 7e7f90e lib/thrift-0.5.0.jar PRE-CREATION lib/thrift-fb303-0.5.0.jar PRE-CREATION lib/thrift-fb303.LICENSE PRE-CREATION lib/thrift.LICENSE PRE-CREATION metastore/build.xml ffb86c1 metastore/if/hive_metastore.thrift 9c314cc metastore/ivy.xml 2e39eb8 metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java a4f1853 metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 4f5623c odbc/Makefile 816ffd4 odbc/build.xml 64cf988 odbc/src/cpp/HiveRowSet.cpp b4dc2c0 odbc/src/cpp/hiveclienthelper.cpp e2d48f3 ql/build.xml a025a2b ql/if/queryplan.thrift aca0e8f ql/ivy.xml e72a450 ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 5ccb909 ql/src/java/org/apache/hadoop/hive/ql/exec/CollectOperator.java 7daea2d ql/src/java/org/apache/hadoop/hive/ql/exec/ConditionalTask.java bf7b7a5 ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 46b8bf4 ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 0a9cdf4 ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java af6e0af ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java a572bcb ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java 7bf8886 ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java de2c42d ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java d510f95 ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java 554bf25 ql/src/java/org/apache/hadoop/hive/ql/exec/ForwardOperator.java 2a35c96 ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 1e6941e ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 8423178 ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableDummyOperator.java 87fc61c ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 9fe35ec ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java e16df36 ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewForwardOperator.java c070aca ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java 5eb9098 ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java da5dbb9 ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 98571dd ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 887ad30 ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java 0fab63c ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java c5aa3d6 ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7fbc586 ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 6c6ea89 ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 592f4f7 ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 001d1f0 ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 4f8b4bd ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java d644a33 ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 47adbed ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 96e63b8 ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 893530c ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java 682f38a ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 2c42ed7 ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 0d60507 ql/src/java/org/apache/hadoop/hive/ql/index/compact/IndexMetadataChangeTask.java 734bd72 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c5fb22b ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 7167196 serde/build.xml b64b142 serde/ivy.xml da69342 serde/src/java/org/apache/hadoop/hive/serde2/thrift/TBinarySortableProtocol.java 961c24f serde/src/java/org/apache/hadoop/hive/serde2/thrift/TCTLSeparatedProtocol.java e781a06 service/build.xml ffd1854 service/if/hive_service.thrift db9fab6 service/include/thrift/TLogging.h 2df82dd service/include/thrift/TProcessor.h f2d5279 service/include/thrift/TReflectionLocal.h e83e475 service/include/thrift/Thrift.h 27a6476 service/include/thrift/concurrency/Exception.h ec46629 service/include/thrift/concurrency/FunctionRunner.h 2216927 service/include/thrift/concurrency/Monitor.h 234bf32 service/include/thrift/concurrency/Mutex.h 73c73e0 service/include/thrift/concurrency/PosixThreadFactory.h d6d83a3 service/include/thrift/concurrency/Thread.h d4282ad service/include/thrift/concurrency/ThreadManager.h 6e5a178 service/include/thrift/concurrency/TimerManager.h dfbf0ea service/include/thrift/concurrency/Util.h 25fcc20 service/include/thrift/config.h 70264f2 service/include/thrift/fb303/FacebookBase.h fd169e6 service/include/thrift/fb303/FacebookService.h dcd843f service/include/thrift/fb303/FacebookService.cpp 4a34362 service/include/thrift/fb303/ServiceTracker.h 9304386 service/include/thrift/fb303/fb303_constants.h 19405bc service/include/thrift/fb303/fb303_constants.cpp 66b8782 service/include/thrift/fb303/fb303_types.h 18936f9 service/include/thrift/fb303/fb303_types.cpp f290125 service/include/thrift/fb303/if/fb303.thrift 66c8315 service/include/thrift/if/reflection_limited.thrift 993c46e service/include/thrift/processor/PeekProcessor.h 0f7c016 service/include/thrift/processor/StatsProcessor.h 820b3ad service/include/thrift/protocol/TBase64Utils.h 3def733 service/include/thrift/protocol/TBinaryProtocol.h 7fd3de6 service/include/thrift/protocol/TCompactProtocol.h b4e06f0 service/include/thrift/protocol/TDebugProtocol.h ab69e0c service/include/thrift/protocol/TDenseProtocol.h 7655a47 service/include/thrift/protocol/TJSONProtocol.h 2df499a service/include/thrift/protocol/TOneWayProtocol.h 6f08fe1 service/include/thrift/protocol/TProtocol.h 4025827 service/include/thrift/protocol/TProtocolException.h 33011b3 service/include/thrift/protocol/TProtocolTap.h 5580216 service/include/thrift/reflection_limited_types.h 677cb4a service/include/thrift/server/TNonblockingServer.h 8506507 service/include/thrift/server/TServer.h 5c4c588 service/include/thrift/server/TSimpleServer.h c4fc91c service/include/thrift/server/TThreadPoolServer.h 7b7e906 service/include/thrift/server/TThreadedServer.h 4d0811a service/include/thrift/transport/TBufferTransports.h 1908205 service/include/thrift/transport/TFDTransport.h bda5d82 service/include/thrift/transport/TFileTransport.h b08c5c8 service/include/thrift/transport/THttpClient.h f4be4c1 service/include/thrift/transport/TServerSocket.h a6be017 service/include/thrift/transport/TServerTransport.h 40bbc6c service/include/thrift/transport/TShortReadTransport.h 3df8a57 service/include/thrift/transport/TSimpleFileTransport.h 6cc52ea service/include/thrift/transport/TSocket.h b0f445a service/include/thrift/transport/TSocketPool.h 8c50669 service/include/thrift/transport/TTransport.h eb0d5df service/include/thrift/transport/TTransportException.h 330785c service/include/thrift/transport/TTransportUtils.h d65c916 service/include/thrift/transport/TZlibTransport.h 1439d9d service/src/java/org/apache/hadoop/hive/service/HiveServer.java 89cf336 Diff: http://review.cloudera.org/r/1242/diff Testing ------- Thanks, Carl
          Hide
          Carl Steinbach added a comment -

          @Pradeep: working on it now

          Show
          Carl Steinbach added a comment - @Pradeep: working on it now
          Hide
          Pradeep Kamath added a comment -

          Hi Carl - just wondering if you have had a chance to look at this..

          Show
          Pradeep Kamath added a comment - Hi Carl - just wondering if you have had a chance to look at this..
          Hide
          Pradeep Kamath added a comment -

          Hi Carl - just wondering if you have had a chance to look at this - a new patch for this issue will help me create a patch for HIVE-1696 (I suspect we will need to redo HIVE-842 as well - I can take a stab at that once this patch is ready).

          Show
          Pradeep Kamath added a comment - Hi Carl - just wondering if you have had a chance to look at this - a new patch for this issue will help me create a patch for HIVE-1696 (I suspect we will need to redo HIVE-842 as well - I can take a stab at that once this patch is ready).
          Hide
          Carl Steinbach added a comment -

          Need to update this page once this change is committed: http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code

          Show
          Carl Steinbach added a comment - Need to update this page once this change is committed: http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
          Hide
          Carl Steinbach added a comment -

          We discussed this at the contributors meeting yesterday. I'm going to rebase the patch,
          modify it to use Thrift 0.5.0, and then make the patch easier to review by removing the
          the thrift generated code. I plan to get to this sometime in the next couple of days.

          Show
          Carl Steinbach added a comment - We discussed this at the contributors meeting yesterday. I'm going to rebase the patch, modify it to use Thrift 0.5.0, and then make the patch easier to review by removing the the thrift generated code. I plan to get to this sometime in the next couple of days.
          Hide
          Pradeep Kamath added a comment -

          Any update on this? I would like to submit a patch for HIVE-1696 which depends on this and HIVE-842. Since this is currently broken against trunk, am waiting for the new patch (based off thrift-0.5 ?) so I can generate a patch for HIVE-1696.

          Show
          Pradeep Kamath added a comment - Any update on this? I would like to submit a patch for HIVE-1696 which depends on this and HIVE-842 . Since this is currently broken against trunk, am waiting for the new patch (based off thrift-0.5 ?) so I can generate a patch for HIVE-1696 .
          Hide
          Carl Steinbach added a comment -

          I'll be at the contributors meeting. John, if you have time lets meet afterwards to
          review this patch.

          Show
          Carl Steinbach added a comment - I'll be at the contributors meeting. John, if you have time lets meet afterwards to review this patch.
          Hide
          Todd Lipcon added a comment -

          Hey John. I'm actually headed to Tokyo for the next two weeks so won't be at the contributors meeting. Perhaps Carl can look at this with you. Note that we should update the change to Thrift 0.5.0 release before committing, but the review can happen on current code.

          Show
          Todd Lipcon added a comment - Hey John. I'm actually headed to Tokyo for the next two weeks so won't be at the contributors meeting. Perhaps Carl can look at this with you. Note that we should update the change to Thrift 0.5.0 release before committing, but the review can happen on current code.
          Hide
          John Sichi added a comment -

          Ning is currently working on some non-Hive tasks, so we'll get someone else to take a look. If you have time after next Monday's contributor meeting, maybe we can do a quick review then in person.

          Show
          John Sichi added a comment - Ning is currently working on some non-Hive tasks, so we'll get someone else to take a look. If you have time after next Monday's contributor meeting, maybe we can do a quick review then in person.
          Hide
          Todd Lipcon added a comment -

          Hey Ning. Can you take a look at this change? It's no longer in sync with trunk, but I don't want to have to redo it twice (it's a pain since you have to regenerate all the files, etc). If the basics look OK I will resync with trunk and then we can commit soonafter.

          Show
          Todd Lipcon added a comment - Hey Ning. Can you take a look at this change? It's no longer in sync with trunk, but I don't want to have to redo it twice (it's a pain since you have to regenerate all the files, etc). If the basics look OK I will resync with trunk and then we can commit soonafter.
          Hide
          Ning Zhang added a comment -

          I will take a look.

          Show
          Ning Zhang added a comment - I will take a look.
          Hide
          Carl Steinbach added a comment -

          @Ning: I removed the 0.6 tag. Can you please review this change? Thanks.

          Show
          Carl Steinbach added a comment - @Ning: I removed the 0.6 tag. Can you please review this change? Thanks.
          Hide
          Ning Zhang added a comment -

          Carl and Todd, is this a blocking issue for 0.6? If not, we can make it in 0.7 and get 0.6 release ASAP.

          Show
          Ning Zhang added a comment - Carl and Todd, is this a blocking issue for 0.6? If not, we can make it in 0.7 and get 0.6 release ASAP.
          Hide
          Carl Steinbach added a comment -

          HIVE-1526.2.patch.txt:

          • Manage slf4j dependencies with Ivy.
          • Added slf4j dependencies to eclipse classpath.
          • Added "thriftif" macro to $ {hive.root}

            /build.xml which triggers recompilation of all thrift stubs.

          • Modified odbc/Makefile to use Thrift libs and headers in THRIFT_HOME instead of the ones that were checked into service/include.
          • Modified odbc/Makefile to build thrift generated cpp artifacts in ql/src
          • Removed thrift headers/code from service/include (HIVE-1527)
          • Added some missing #includes to the hiveclient source files in odbc/src/cpp.

          Testing:

          • Tested eclipse launch configurations.
          • Built CPP hiveclient lib and tested against HiveServer using HiveClientTestC program.
          Show
          Carl Steinbach added a comment - HIVE-1526 .2.patch.txt: Manage slf4j dependencies with Ivy. Added slf4j dependencies to eclipse classpath. Added "thriftif" macro to $ {hive.root} /build.xml which triggers recompilation of all thrift stubs. Modified odbc/Makefile to use Thrift libs and headers in THRIFT_HOME instead of the ones that were checked into service/include. Modified odbc/Makefile to build thrift generated cpp artifacts in ql/src Removed thrift headers/code from service/include ( HIVE-1527 ) Added some missing #includes to the hiveclient source files in odbc/src/cpp. Testing: Tested eclipse launch configurations. Built CPP hiveclient lib and tested against HiveServer using HiveClientTestC program.
          Hide
          Carl Steinbach added a comment -

          Sorry, that was a false alarm about the patch. Turns out the github Hive mirror lags the main repo by about a week.

          @Todd: This patch introduces unsatisfied dependencies on slf4j-api and slf4j-log4j12. Can you please update the patch to pull these dependencies down with Ivy?

          Show
          Carl Steinbach added a comment - Sorry, that was a false alarm about the patch. Turns out the github Hive mirror lags the main repo by about a week. @Todd: This patch introduces unsatisfied dependencies on slf4j-api and slf4j-log4j12. Can you please update the patch to pull these dependencies down with Ivy?
          Hide
          Todd Lipcon added a comment -

          Hi Carl. It appears to apply fine for me on today's trunk:

          todd@todd-laptop:~/git/hive$ git reset --hard HEAD
          HEAD is now at 08d5e5b HIVE-1661. Default values for parameters (Siying Dong via He Yongqiang)
          todd@todd-laptop:~/git/hive$ curl 'https://issues.apache.org/jira/secure/attachment/12455323/hive-1526.txt' | patch -p0
          ... patch output....
          todd@todd-laptop:~/git/hive$

          Sure you're up to date on trunk?

          Show
          Todd Lipcon added a comment - Hi Carl. It appears to apply fine for me on today's trunk: todd@todd-laptop:~/git/hive$ git reset --hard HEAD HEAD is now at 08d5e5b HIVE-1661 . Default values for parameters (Siying Dong via He Yongqiang) todd@todd-laptop:~/git/hive$ curl 'https://issues.apache.org/jira/secure/attachment/12455323/hive-1526.txt' | patch -p0 ... patch output.... todd@todd-laptop:~/git/hive$ Sure you're up to date on trunk?
          Hide
          Carl Steinbach added a comment -

          @Todd: Can you please regenerate this patch? Both 'patch -p0' and 'git apply -p0' fail. Thanks.

          Show
          Carl Steinbach added a comment - @Todd: Can you please regenerate this patch? Both 'patch -p0' and 'git apply -p0' fail. Thanks.
          Hide
          Ning Zhang added a comment -

          The Hive ODBC code is dependent on Thrift as well. In particular the hive client and unixODBC libraries have to be linked with the new libthrift.so. Can you test if the ODBC code is compatible with the new thrift version?

          Show
          Ning Zhang added a comment - The Hive ODBC code is dependent on Thrift as well. In particular the hive client and unixODBC libraries have to be linked with the new libthrift.so. Can you test if the ODBC code is compatible with the new thrift version?
          Hide
          Todd Lipcon added a comment -

          Here is a patch along with the newly built jars from Thrift 0.4.0.

          I agree that long term we should make codegen part of the build, but I think it's enough of a hassle to require everyone to install the same version of thrift, we should punt for now.

          Show
          Todd Lipcon added a comment - Here is a patch along with the newly built jars from Thrift 0.4.0. I agree that long term we should make codegen part of the build, but I think it's enough of a hassle to require everyone to install the same version of thrift, we should punt for now.
          Hide
          John Sichi added a comment -

          Note HIVE-1527.

          Show
          John Sichi added a comment - Note HIVE-1527 .
          Hide
          Todd Lipcon added a comment -

          Er, sorry, not THRIFT-381, but rather THRIFT-907. Too many browser tabs!

          Show
          Todd Lipcon added a comment - Er, sorry, not THRIFT-381 , but rather THRIFT-907 . Too many browser tabs!
          Hide
          Todd Lipcon added a comment -

          Planning on upgrading to 0.4.0. The main incompatible changes to deal with:

          • Methods that return enums now actually return a java enum type instead of an int (eg fb303's get_status method as well as the operator and stage getType() methods)
          • readBinary and writeBinary now use ByteBuffer instead of byte[]

          Unfortunately the maven situation is still not yet sorted out. I built a libthrift.jar and a libfb303.jar from the 0.4.0 release tree and will upload them here. Had to fix a minor issue to get fb303 to build (see THRIFT-381)

          Aside from that, it seems pretty straightforward. I have a patch, running unit tests now.

          Show
          Todd Lipcon added a comment - Planning on upgrading to 0.4.0. The main incompatible changes to deal with: Methods that return enums now actually return a java enum type instead of an int (eg fb303's get_status method as well as the operator and stage getType() methods) readBinary and writeBinary now use ByteBuffer instead of byte[] Unfortunately the maven situation is still not yet sorted out. I built a libthrift.jar and a libfb303.jar from the 0.4.0 release tree and will upload them here. Had to fix a minor issue to get fb303 to build (see THRIFT-381 ) Aside from that, it seems pretty straightforward. I have a patch, running unit tests now.

            People

            • Assignee:
              Carl Steinbach
              Reporter:
              Carl Steinbach
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development