Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4
    • Component/s: None
    • Labels:
      None

      Description

      Currently HCat allow a selected set of DDL statements and delegate them to Hive. We shall allow more DDL statements. The criteria is the DDL statement is nonblocking. Locking is not considered since otherwise most DDL operation will be blocking. Allowed DDL includes:

      • database DDL
      • view DDL
      • index DDL (no actual index building, eg, alter index rebuild is excluded)
      • authorization DDL
      • lock management DDL
      • table management DDL (no create as select)
      • table partition management DDL (no CONCATENATE, archive/unarchive)
      • table schema management DDL
      • Misc: show function, explain, dfs, set

      import/export which are supported before is dropped since they are blocking.

      1. HCATALOG-261-4.patch
        57 kB
        Daniel Dai
      2. HCATALOG-261-3.patch
        60 kB
        Daniel Dai
      3. HCATALOG-261-2.patch
        60 kB
        Daniel Dai
      4. HCATALOG-261-1.patch
        33 kB
        Daniel Dai

        Issue Links

          Activity

          Hide
          alangates Alan Gates added a comment -

          Looks good. A couple of questions.

          In HCatSemanticAnalyzer, did you mean to take out the authorize calls in the DESCTABLE and SHOWPARTITIONS cases?

          For the SEQUENCEFILE_INPUT and OUTPUT constants, there are not already Hive constants for this that we could reuse?

          Show
          alangates Alan Gates added a comment - Looks good. A couple of questions. In HCatSemanticAnalyzer, did you mean to take out the authorize calls in the DESCTABLE and SHOWPARTITIONS cases? For the SEQUENCEFILE_INPUT and OUTPUT constants, there are not already Hive constants for this that we could reuse?
          Hide
          daijy Daniel Dai added a comment -

          I totally ignore authorization part since I assume Enis will rework the authorization on task level. I will resync and verify the authorization is in the right place after enis' patch.

          I didn't find sequencefile related constant in HCatConstants, it might because we don't support sequence file before.

          Show
          daijy Daniel Dai added a comment - I totally ignore authorization part since I assume Enis will rework the authorization on task level. I will resync and verify the authorization is in the right place after enis' patch. I didn't find sequencefile related constant in HCatConstants, it might because we don't support sequence file before.
          Hide
          daijy Daniel Dai added a comment -

          HCATALOG-261-2.patch resync with trunk, and include changes of HCATALOG-242.

          Need to do svn rm manually:
          svn rm src/java/org/apache/hcatalog/cli/SemanticAnalysis/AddPartitionHook.java
          svn rm src/java/org/apache/hcatalog/cli/SemanticAnalysis/AlterTableFileFormatHook.java

          Show
          daijy Daniel Dai added a comment - HCATALOG-261 -2.patch resync with trunk, and include changes of HCATALOG-242 . Need to do svn rm manually: svn rm src/java/org/apache/hcatalog/cli/SemanticAnalysis/AddPartitionHook.java svn rm src/java/org/apache/hcatalog/cli/SemanticAnalysis/AlterTableFileFormatHook.java
          Hide
          alangates Alan Gates added a comment -

          +1, looks good. I'll commit once HCATALOG-239 is in.

          Show
          alangates Alan Gates added a comment - +1, looks good. I'll commit once HCATALOG-239 is in.
          Hide
          alangates Alan Gates added a comment -

          After applying this patch writing to partitioned tables fails. Hadoop_Checkin_5, 6 and Pig_Checkin_6, which passed before, all fail. Also a number of the newly added tests fail, but that looks like because they depend on HCATALOG-264. That issue is marked as depending on this one, but is it the other way around?

          Show
          alangates Alan Gates added a comment - After applying this patch writing to partitioned tables fails. Hadoop_Checkin_5, 6 and Pig_Checkin_6, which passed before, all fail. Also a number of the newly added tests fail, but that looks like because they depend on HCATALOG-264 . That issue is marked as depending on this one, but is it the other way around?
          Hide
          alangates Alan Gates added a comment -

          After applying the patch for HCATALOG-279 on top of this one, the Hadoop_Checkin 5,6 and Pig_Checkin_6 tests pass. But I still see some failures.

          Hive_Checkin_4 aborts with:

          FAILED: Error in metadata: MetaException(message:Got exception: org.apache.hadoop.security.AccessControlException org.apache.hadoop.security.AccessControlException: Permission denied: user=hcat, access=WRITE, inode="hdp1warehouse":hortonal:supergroup:rwxr-xr-x)
          

          This looks like a security error issue, as this operation should be being done by the user running the job (in this case hortonal) rather than as hcat.

          Hive_Write_1, which passes without this patch, also fails.

          The new tests Pig_HCAT_COOP_2, HCat_View_1, HCat_Authorize_3 fail.

          Show
          alangates Alan Gates added a comment - After applying the patch for HCATALOG-279 on top of this one, the Hadoop_Checkin 5,6 and Pig_Checkin_6 tests pass. But I still see some failures. Hive_Checkin_4 aborts with: FAILED: Error in metadata: MetaException(message:Got exception: org.apache.hadoop.security.AccessControlException org.apache.hadoop.security.AccessControlException: Permission denied: user=hcat, access=WRITE, inode= "hdp1warehouse" :hortonal:supergroup:rwxr-xr-x) This looks like a security error issue, as this operation should be being done by the user running the job (in this case hortonal) rather than as hcat. Hive_Write_1, which passes without this patch, also fails. The new tests Pig_HCAT_COOP_2, HCat_View_1, HCat_Authorize_3 fail.
          Hide
          daijy Daniel Dai added a comment -

          Pig_HCAT_COOP_2 is addressed in HCATALOG-281, HCat_View_1 is a typo in test case, I upload HCATALOG-261-3.patch to fix it.

          I am still looking at HCat_Authorize_3 and Hive_Checkin_4.

          Show
          daijy Daniel Dai added a comment - Pig_HCAT_COOP_2 is addressed in HCATALOG-281 , HCat_View_1 is a typo in test case, I upload HCATALOG-261 -3.patch to fix it. I am still looking at HCat_Authorize_3 and Hive_Checkin_4.
          Hide
          daijy Daniel Dai added a comment -

          HCat_Authorize_3 is testing authorization system before HCATALOG-245. After HCATALOG-245, the test case should no longer working. Remove this test. Attach HCATALOG-261-4.path for it.

          For Hive_Checkin_4, I get exception initially, but after I set hive.metastore.execute.setugi to true, the test works for me.

          Show
          daijy Daniel Dai added a comment - HCat_Authorize_3 is testing authorization system before HCATALOG-245 . After HCATALOG-245 , the test case should no longer working. Remove this test. Attach HCATALOG-261 -4.path for it. For Hive_Checkin_4, I get exception initially, but after I set hive.metastore.execute.setugi to true, the test works for me.
          Hide
          alangates Alan Gates added a comment -

          Patch 4 checked into trunk and branch 0.4

          Show
          alangates Alan Gates added a comment - Patch 4 checked into trunk and branch 0.4
          Hide
          alangates Alan Gates added a comment -

          Issue closed with 0.4 release.

          Show
          alangates Alan Gates added a comment - Issue closed with 0.4 release.

            People

            • Assignee:
              daijy Daniel Dai
              Reporter:
              daijy Daniel Dai
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development