Hive
  1. Hive
  2. HIVE-2502

Add whitelist for hosts used in table/partition locations

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Add a whitelist of host names that can be checked before creating/altering a table/partition to verify that the location is acceptable. The whitelist should be empty by default, and should be configurable. The check should default to pass if there is no host in the location, or the whitelist is empty.

      1. HIVE-2502.4.patch.txt
        11 kB
        Kevin Wilfong
      2. HIVE-2502.3.patch.txt
        11 kB
        Kevin Wilfong
      3. HIVE-2502.2.patch.txt
        11 kB
        Kevin Wilfong
      4. HIVE-2502.1.patch.txt
        10 kB
        Kevin Wilfong

        Activity

        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2371/
        -----------------------------------------------------------

        Review request for hive and Paul Yang.

        Summary
        -------

        I added a whitelist configuration variable, whose value should be a comma separated list of hosts. A set of strings is created based on this list and the host name in a location given to a create/alter table/partition command to the HiveMetastore is compared to the values in that set. If the list is nonempty, the table being altered/created or containing the partition is not external/non-native, and the location has an explicit host, the command is failed if the host is not in that set, otherwise the command continues as normal.

        This addresses bug HIVE-2502.
        https://issues.apache.org/jira/browse/HIVE-2502

        Diffs


        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1182505
        trunk/conf/hive-default.xml 1182505
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1182505
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1182505
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1182505

        Diff: https://reviews.apache.org/r/2371/diff

        Testing
        -------

        I used Thrift calls to test the following for each of the 4 commands create/alter table/partition:

        Issuing the command with a location with a host that is not in the whitelist fails.
        Issuing the command with a location with a host in the whitelist does not fail.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2371/ ----------------------------------------------------------- Review request for hive and Paul Yang. Summary ------- I added a whitelist configuration variable, whose value should be a comma separated list of hosts. A set of strings is created based on this list and the host name in a location given to a create/alter table/partition command to the HiveMetastore is compared to the values in that set. If the list is nonempty, the table being altered/created or containing the partition is not external/non-native, and the location has an explicit host, the command is failed if the host is not in that set, otherwise the command continues as normal. This addresses bug HIVE-2502 . https://issues.apache.org/jira/browse/HIVE-2502 Diffs trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1182505 trunk/conf/hive-default.xml 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1182505 Diff: https://reviews.apache.org/r/2371/diff Testing ------- I used Thrift calls to test the following for each of the 4 commands create/alter table/partition: Issuing the command with a location with a host that is not in the whitelist fails. Issuing the command with a location with a host in the whitelist does not fail. Thanks, Kevin
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2371/#review2575
        -----------------------------------------------------------

        Looks good, but can we do host:port combinations instead of just the host?

        • Paul

        On 2011-10-12 22:48:52, Kevin Wilfong wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2371/

        -----------------------------------------------------------

        (Updated 2011-10-12 22:48:52)

        Review request for hive and Paul Yang.

        Summary

        -------

        I added a whitelist configuration variable, whose value should be a comma separated list of hosts. A set of strings is created based on this list and the host name in a location given to a create/alter table/partition command to the HiveMetastore is compared to the values in that set. If the list is nonempty, the table being altered/created or containing the partition is not external/non-native, and the location has an explicit host, the command is failed if the host is not in that set, otherwise the command continues as normal.

        This addresses bug HIVE-2502.

        https://issues.apache.org/jira/browse/HIVE-2502

        Diffs

        -----

        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1182505

        trunk/conf/hive-default.xml 1182505

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1182505

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1182505

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1182505

        Diff: https://reviews.apache.org/r/2371/diff

        Testing

        -------

        I used Thrift calls to test the following for each of the 4 commands create/alter table/partition:

        Issuing the command with a location with a host that is not in the whitelist fails.

        Issuing the command with a location with a host in the whitelist does not fail.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2371/#review2575 ----------------------------------------------------------- Looks good, but can we do host:port combinations instead of just the host? Paul On 2011-10-12 22:48:52, Kevin Wilfong wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2371/ ----------------------------------------------------------- (Updated 2011-10-12 22:48:52) Review request for hive and Paul Yang. Summary ------- I added a whitelist configuration variable, whose value should be a comma separated list of hosts. A set of strings is created based on this list and the host name in a location given to a create/alter table/partition command to the HiveMetastore is compared to the values in that set. If the list is nonempty, the table being altered/created or containing the partition is not external/non-native, and the location has an explicit host, the command is failed if the host is not in that set, otherwise the command continues as normal. This addresses bug HIVE-2502 . https://issues.apache.org/jira/browse/HIVE-2502 Diffs ----- trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1182505 trunk/conf/hive-default.xml 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1182505 Diff: https://reviews.apache.org/r/2371/diff Testing ------- I used Thrift calls to test the following for each of the 4 commands create/alter table/partition: Issuing the command with a location with a host that is not in the whitelist fails. Issuing the command with a location with a host in the whitelist does not fail. Thanks, Kevin
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2371/
        -----------------------------------------------------------

        (Updated 2011-10-14 01:27:18.448099)

        Review request for hive and Paul Yang.

        Changes
        -------

        Thanks for the feedback Paul. I changed it to use host:port pairs instead of just hosts.

        Summary
        -------

        I added a whitelist configuration variable, whose value should be a comma separated list of hosts. A set of strings is created based on this list and the host name in a location given to a create/alter table/partition command to the HiveMetastore is compared to the values in that set. If the list is nonempty, the table being altered/created or containing the partition is not external/non-native, and the location has an explicit host, the command is failed if the host is not in that set, otherwise the command continues as normal.

        This addresses bug HIVE-2502.
        https://issues.apache.org/jira/browse/HIVE-2502

        Diffs (updated)


        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1182505
        trunk/conf/hive-default.xml 1182505
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1182505
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1182505
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1182505

        Diff: https://reviews.apache.org/r/2371/diff

        Testing
        -------

        I used Thrift calls to test the following for each of the 4 commands create/alter table/partition:

        Issuing the command with a location with a host that is not in the whitelist fails.
        Issuing the command with a location with a host in the whitelist does not fail.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2371/ ----------------------------------------------------------- (Updated 2011-10-14 01:27:18.448099) Review request for hive and Paul Yang. Changes ------- Thanks for the feedback Paul. I changed it to use host:port pairs instead of just hosts. Summary ------- I added a whitelist configuration variable, whose value should be a comma separated list of hosts. A set of strings is created based on this list and the host name in a location given to a create/alter table/partition command to the HiveMetastore is compared to the values in that set. If the list is nonempty, the table being altered/created or containing the partition is not external/non-native, and the location has an explicit host, the command is failed if the host is not in that set, otherwise the command continues as normal. This addresses bug HIVE-2502 . https://issues.apache.org/jira/browse/HIVE-2502 Diffs (updated) trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1182505 trunk/conf/hive-default.xml 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1182505 Diff: https://reviews.apache.org/r/2371/diff Testing ------- I used Thrift calls to test the following for each of the 4 commands create/alter table/partition: Issuing the command with a location with a host that is not in the whitelist fails. Issuing the command with a location with a host in the whitelist does not fail. Thanks, Kevin
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2371/
        -----------------------------------------------------------

        (Updated 2011-10-14 16:56:01.946053)

        Review request for hive and Paul Yang.

        Changes
        -------

        All lines are within the 100 char limit.

        Summary
        -------

        I added a whitelist configuration variable, whose value should be a comma separated list of hosts. A set of strings is created based on this list and the host name in a location given to a create/alter table/partition command to the HiveMetastore is compared to the values in that set. If the list is nonempty, the table being altered/created or containing the partition is not external/non-native, and the location has an explicit host, the command is failed if the host is not in that set, otherwise the command continues as normal.

        This addresses bug HIVE-2502.
        https://issues.apache.org/jira/browse/HIVE-2502

        Diffs (updated)


        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1182505
        trunk/conf/hive-default.xml 1182505
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1182505
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1182505
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1182505

        Diff: https://reviews.apache.org/r/2371/diff

        Testing
        -------

        I used Thrift calls to test the following for each of the 4 commands create/alter table/partition:

        Issuing the command with a location with a host that is not in the whitelist fails.
        Issuing the command with a location with a host in the whitelist does not fail.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2371/ ----------------------------------------------------------- (Updated 2011-10-14 16:56:01.946053) Review request for hive and Paul Yang. Changes ------- All lines are within the 100 char limit. Summary ------- I added a whitelist configuration variable, whose value should be a comma separated list of hosts. A set of strings is created based on this list and the host name in a location given to a create/alter table/partition command to the HiveMetastore is compared to the values in that set. If the list is nonempty, the table being altered/created or containing the partition is not external/non-native, and the location has an explicit host, the command is failed if the host is not in that set, otherwise the command continues as normal. This addresses bug HIVE-2502 . https://issues.apache.org/jira/browse/HIVE-2502 Diffs (updated) trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1182505 trunk/conf/hive-default.xml 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1182505 Diff: https://reviews.apache.org/r/2371/diff Testing ------- I used Thrift calls to test the following for each of the 4 commands create/alter table/partition: Issuing the command with a location with a host that is not in the whitelist fails. Issuing the command with a location with a host in the whitelist does not fail. Thanks, Kevin
        Hide
        Paul Yang added a comment -

        +1 Will test and commit.

        Show
        Paul Yang added a comment - +1 Will test and commit.
        Hide
        Paul Yang added a comment -

        Got failures in a few tests:

        alter_view_rename.q
        create_or_replace_view.q
        create_view.q
        recursive_view.q

        Seems like the views case isn't properly handled.

        Show
        Paul Yang added a comment - Got failures in a few tests: alter_view_rename.q create_or_replace_view.q create_view.q recursive_view.q Seems like the views case isn't properly handled.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2371/
        -----------------------------------------------------------

        (Updated 2011-10-27 17:38:58.120670)

        Review request for hive and Paul Yang.

        Changes
        -------

        I fixed an issue that was causing altering views to fail.

        I ran all the test queries and they succeeded.

        Summary
        -------

        I added a whitelist configuration variable, whose value should be a comma separated list of hosts. A set of strings is created based on this list and the host name in a location given to a create/alter table/partition command to the HiveMetastore is compared to the values in that set. If the list is nonempty, the table being altered/created or containing the partition is not external/non-native, and the location has an explicit host, the command is failed if the host is not in that set, otherwise the command continues as normal.

        This addresses bug HIVE-2502.
        https://issues.apache.org/jira/browse/HIVE-2502

        Diffs (updated)


        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1189406
        trunk/conf/hive-default.xml 1189406
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1189406
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1189406
        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1189406

        Diff: https://reviews.apache.org/r/2371/diff

        Testing
        -------

        I used Thrift calls to test the following for each of the 4 commands create/alter table/partition:

        Issuing the command with a location with a host that is not in the whitelist fails.
        Issuing the command with a location with a host in the whitelist does not fail.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2371/ ----------------------------------------------------------- (Updated 2011-10-27 17:38:58.120670) Review request for hive and Paul Yang. Changes ------- I fixed an issue that was causing altering views to fail. I ran all the test queries and they succeeded. Summary ------- I added a whitelist configuration variable, whose value should be a comma separated list of hosts. A set of strings is created based on this list and the host name in a location given to a create/alter table/partition command to the HiveMetastore is compared to the values in that set. If the list is nonempty, the table being altered/created or containing the partition is not external/non-native, and the location has an explicit host, the command is failed if the host is not in that set, otherwise the command continues as normal. This addresses bug HIVE-2502 . https://issues.apache.org/jira/browse/HIVE-2502 Diffs (updated) trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1189406 trunk/conf/hive-default.xml 1189406 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1189406 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1189406 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1189406 Diff: https://reviews.apache.org/r/2371/diff Testing ------- I used Thrift calls to test the following for each of the 4 commands create/alter table/partition: Issuing the command with a location with a host that is not in the whitelist fails. Issuing the command with a location with a host in the whitelist does not fail. Thanks, Kevin
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2371/#review2881
        -----------------------------------------------------------

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
        <https://reviews.apache.org/r/2371/#comment6452>

        This is the only change I made in r4, it now only checks the location if the table is not a view.

        • Kevin

        On 2011-10-27 17:38:58, Kevin Wilfong wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2371/

        -----------------------------------------------------------

        (Updated 2011-10-27 17:38:58)

        Review request for hive and Paul Yang.

        Summary

        -------

        I added a whitelist configuration variable, whose value should be a comma separated list of hosts. A set of strings is created based on this list and the host name in a location given to a create/alter table/partition command to the HiveMetastore is compared to the values in that set. If the list is nonempty, the table being altered/created or containing the partition is not external/non-native, and the location has an explicit host, the command is failed if the host is not in that set, otherwise the command continues as normal.

        This addresses bug HIVE-2502.

        https://issues.apache.org/jira/browse/HIVE-2502

        Diffs

        -----

        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1189406

        trunk/conf/hive-default.xml 1189406

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1189406

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1189406

        trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1189406

        Diff: https://reviews.apache.org/r/2371/diff

        Testing

        -------

        I used Thrift calls to test the following for each of the 4 commands create/alter table/partition:

        Issuing the command with a location with a host that is not in the whitelist fails.

        Issuing the command with a location with a host in the whitelist does not fail.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2371/#review2881 ----------------------------------------------------------- trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java < https://reviews.apache.org/r/2371/#comment6452 > This is the only change I made in r4, it now only checks the location if the table is not a view. Kevin On 2011-10-27 17:38:58, Kevin Wilfong wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2371/ ----------------------------------------------------------- (Updated 2011-10-27 17:38:58) Review request for hive and Paul Yang. Summary ------- I added a whitelist configuration variable, whose value should be a comma separated list of hosts. A set of strings is created based on this list and the host name in a location given to a create/alter table/partition command to the HiveMetastore is compared to the values in that set. If the list is nonempty, the table being altered/created or containing the partition is not external/non-native, and the location has an explicit host, the command is failed if the host is not in that set, otherwise the command continues as normal. This addresses bug HIVE-2502 . https://issues.apache.org/jira/browse/HIVE-2502 Diffs ----- trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1189406 trunk/conf/hive-default.xml 1189406 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1189406 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1189406 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1189406 Diff: https://reviews.apache.org/r/2371/diff Testing ------- I used Thrift calls to test the following for each of the 4 commands create/alter table/partition: Issuing the command with a location with a host that is not in the whitelist fails. Issuing the command with a location with a host in the whitelist does not fail. Thanks, Kevin

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development