Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-592

HCatalogStore should supports RCFile and default hive field delimiter.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: Catalog
    • Labels:
      None

      Description

      If hive table used default row format delimiter, hive doesn't return field.delim. As the result, tajo can't scan hive tables which used default row format delimiter. So, in this case, tajo must set field.delim to \001.

      And current HCatalogStore just supports TextFile, it should supports RCFile.

      At last, I found that HCatalogStore doesn't set compression and field.delim when it created tables. So,we should implement these.

      1. TAJO-592_2.patch
        9 kB
        Jaehwa Jung
      2. TAJO-592_3.patch
        10 kB
        Jaehwa Jung
      3. TAJO-592.patch
        8 kB
        Jaehwa Jung

        Activity

        Hide
        blrunner Jaehwa Jung added a comment -

        I created a new RB https://reviews.apache.org/r/17901/.
        Please review it.

        Show
        blrunner Jaehwa Jung added a comment - I created a new RB https://reviews.apache.org/r/17901/ . Please review it.
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12627996/TAJO-592.patch
        against master revision 4179a7c.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-drivers/tajo-hcatalog.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/115//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/115//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-hcatalog.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/115//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12627996/TAJO-592.patch against master revision 4179a7c. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-drivers/tajo-hcatalog. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/115//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/115//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-hcatalog.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/115//console This message is automatically generated.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Thank you for nice work.

        I would like to suggest one thing. I think that the title is too general. Later, could you please describe the title to well summarize the issue itself?

        Thanks!

        Show
        hyunsik Hyunsik Choi added a comment - Thank you for nice work. I would like to suggest one thing. I think that the title is too general. Later, could you please describe the title to well summarize the issue itself? Thanks!
        Hide
        blrunner Jaehwa Jung added a comment -

        Thanks Hyunsik Choi.
        I read your review, and I uploaded second patch.
        Could you review it again?

        Show
        blrunner Jaehwa Jung added a comment - Thanks Hyunsik Choi . I read your review, and I uploaded second patch. Could you review it again?
        Hide
        hyunsik Hyunsik Choi added a comment -

        Jaehwa Jung,

        Thank you for quick response. Could you tell me where did you upload the second patch? I couldn't find the second patch.

        Show
        hyunsik Hyunsik Choi added a comment - Jaehwa Jung , Thank you for quick response. Could you tell me where did you upload the second patch? I couldn't find the second patch.
        Hide
        blrunner Jaehwa Jung added a comment -

        Sorry, Hyunsik Choi.

        It just had been uploaded to review board.
        I uploaded it to JIRA right now.

        Show
        blrunner Jaehwa Jung added a comment - Sorry, Hyunsik Choi . It just had been uploaded to review board. I uploaded it to JIRA right now.
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12628203/TAJO-592_2.patch
        against master revision 8e859fb.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-drivers/tajo-hcatalog.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/117//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/117//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-hcatalog.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/117//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628203/TAJO-592_2.patch against master revision 8e859fb. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-drivers/tajo-hcatalog. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/117//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/117//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-hcatalog.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/117//console This message is automatically generated.
        Hide
        blrunner Jaehwa Jung added a comment -

        I fixed some bugs.
        Please review it again.

        Show
        blrunner Jaehwa Jung added a comment - I fixed some bugs. Please review it again.
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12628223/TAJO-592_3.patch
        against master revision 8e859fb.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-drivers/tajo-hcatalog.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/118//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/118//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-hcatalog.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/118//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628223/TAJO-592_3.patch against master revision 8e859fb. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-drivers/tajo-hcatalog. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/118//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/118//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-hcatalog.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/118//console This message is automatically generated.
        Hide
        blrunner Jaehwa Jung added a comment -

        This issue got one +1 on RB. I just committed it to master branch.

        P.S Thanks Jihoon Kwon

        Show
        blrunner Jaehwa Jung added a comment - This issue got one +1 on RB. I just committed it to master branch. P.S Thanks Jihoon Kwon
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #63 (See https://builds.apache.org/job/Tajo-master-build/63/)
        TAJO-592: HCatalogStore should supports RCFile and default hive field delimiter. (jaehwa) (jhjung: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=519a16a23a4cd7e3eca9959b501297e4be60ca6e)

        • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java
        • CHANGES.txt
        • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java
        • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #63 (See https://builds.apache.org/job/Tajo-master-build/63/ ) TAJO-592 : HCatalogStore should supports RCFile and default hive field delimiter. (jaehwa) (jhjung: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=519a16a23a4cd7e3eca9959b501297e4be60ca6e ) tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java CHANGES.txt tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java

          People

          • Assignee:
            blrunner Jaehwa Jung
            Reporter:
            blrunner Jaehwa Jung
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development