Hive
  1. Hive
  2. HIVE-874

add partitions found during metastore check

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0, 0.4.0, 0.4.1, 0.5.0, 0.6.0
    • Fix Version/s: 0.5.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      'msck' just reports the list of partition directories that exist but do not have corresponding metadata. This can happen if a process outside of hive is populating the directories. Hive should support an option to 'msck' that would also add default metadata for these partitions.

      1. HIVE-874.patch
        10 kB
        Cyrus Katrak

        Issue Links

          Activity

          Prasad Chakka created issue -
          Hide
          Prasad Chakka added a comment -

          Cyrus,

          I created a separate JIRA since HIVE-493 is about automatically inferring partitions during query time and not updating metadata at all. I would like to keep that open since this may not be sufficient for some usecases. Could you upload your patch here?

          As for unit tests, check hive/ql/src/test/queries/clientpositive directory that contains the CLI tests. One of the tests could be to create hdfs directories for a table like 'srcpart' and run 'msck repair' on it and do a 'show partitions' on the table again to see if the partition is listed. We need to have some unit tests so that this functionality will not be broken accidentally by some other check-in.

          Show
          Prasad Chakka added a comment - Cyrus, I created a separate JIRA since HIVE-493 is about automatically inferring partitions during query time and not updating metadata at all. I would like to keep that open since this may not be sufficient for some usecases. Could you upload your patch here? As for unit tests, check hive/ql/src/test/queries/clientpositive directory that contains the CLI tests. One of the tests could be to create hdfs directories for a table like 'srcpart' and run 'msck repair' on it and do a 'show partitions' on the table again to see if the partition is listed. We need to have some unit tests so that this functionality will not be broken accidentally by some other check-in.
          Prasad Chakka made changes -
          Field Original Value New Value
          Link This issue is related to HIVE-493 [ HIVE-493 ]
          Hide
          Prasad Chakka added a comment -

          HIVE-874 can be used in certain cases where partition directories are created by non-Hive processes.

          Show
          Prasad Chakka added a comment - HIVE-874 can be used in certain cases where partition directories are created by non-Hive processes.
          Cyrus Katrak made changes -
          Attachment HIVE-874.patch [ 12422235 ]
          Hide
          Cyrus Katrak added a comment -

          Tests added

          Show
          Cyrus Katrak added a comment - Tests added
          Cyrus Katrak made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Prasad Chakka added a comment -

          looks good, will run tests and commit to trunk.

          Show
          Prasad Chakka added a comment - looks good, will run tests and commit to trunk.
          Hide
          Prasad Chakka added a comment -

          committed to trunk. Tahnks Cyrus.

          Show
          Prasad Chakka added a comment - committed to trunk. Tahnks Cyrus.
          Hide
          Namit Jain added a comment -

          @Prasad, feel free to open it back if you think the patch needs to be backported to 0.4 also

          Show
          Namit Jain added a comment - @Prasad, feel free to open it back if you think the patch needs to be backported to 0.4 also
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.5.0 [ 12314156 ]
          Resolution Fixed [ 1 ]
          Zheng Shao made changes -
          Affects Version/s 0.3.2 [ 12314124 ]
          Zheng Shao made changes -
          Affects Version/s 0.6.0 [ 12314524 ]
          Affects Version/s 0.2.0 [ 12313565 ]
          Carl Steinbach made changes -
          Affects Version/s 0.3.1 [ 12313845 ]
          Carl Steinbach made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Ashutosh Chauhan made changes -
          Link This issue is duplicated by HIVE-2173 [ HIVE-2173 ]
          Hide
          Oz Nevo added a comment -

          I know its old - but can the msck also support the reverse operation - meaning create partitions based on directories added to HDFS?

          Show
          Oz Nevo added a comment - I know its old - but can the msck also support the reverse operation - meaning create partitions based on directories added to HDFS?
          Hide
          Oz Nevo added a comment -

          Sorry correction - I mean remove partitions once the partition directory was removed

          Show
          Oz Nevo added a comment - Sorry correction - I mean remove partitions once the partition directory was removed
          Hide
          Oz Nevo added a comment -

          Sorry - correction to previous comment - remove partitions once the partition directory was removed from HDFS. Obviously adding partitions is the current functionality.

          Show
          Oz Nevo added a comment - Sorry - correction to previous comment - remove partitions once the partition directory was removed from HDFS. Obviously adding partitions is the current functionality.

            People

            • Assignee:
              Cyrus Katrak
              Reporter:
              Prasad Chakka
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development