Hive
  1. Hive
  2. HIVE-2032

create database does not honour warehouse.dir in dbproperties

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.7.0, 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: Clients
    • Labels:
      None

      Description

      1. create database db with dbproperties ('hive.metastore.warehouse.dir' = 'loc');

      The above command does not set location of 'db' to 'loc'. It instead creates 'db.db' under the warehouse directory configured in hive-site.xml of CLI. Looks conflicting with HIVE-1820's expectation. If scratch dir is specified here, that is honoured.

      1. DatabaseLocation.patch
        1 kB
        Thiruvel Thirumoolan

        Activity

        Hide
        Thiruvel Thirumoolan added a comment -

        Preliminary patch. Test cases are added.

        Show
        Thiruvel Thirumoolan added a comment - Preliminary patch. Test cases are added.
        Hide
        Amareshwari Sriramadasu added a comment -

        I see changes for alter Database also to change the location URI. What are semantics of alter database for changing location uri? Is it going to create the tables under the new location after altering? If so, what happens to earlier tables. I see that location uri is accessed only in create db and drop db. So, I think we should not allow altering the location uri. Shall we throw an error if it altered?

        Changes for create DB look fine. Can you update TestHiveMetaStore.testDatabase() to test the same?

        Show
        Amareshwari Sriramadasu added a comment - I see changes for alter Database also to change the location URI. What are semantics of alter database for changing location uri? Is it going to create the tables under the new location after altering? If so, what happens to earlier tables. I see that location uri is accessed only in create db and drop db. So, I think we should not allow altering the location uri. Shall we throw an error if it altered? Changes for create DB look fine. Can you update TestHiveMetaStore.testDatabase() to test the same?
        Hide
        Thiruvel Thirumoolan added a comment -

        @Amareshwari

        Post altering, new tables will be created under new location. Old tables' have the fully qualified location in metadata and they should continue to work as before.

        The reasons I went with alter location are:

        1. Allow migration to happen if one would like to reorganize or if quota runs out. Not sure how many folks have this situation.
        2. One can migrate new tables to another DFS cluster (existing cluster becoming full).
        3. One can migrate between file systems if they have sufficient use cases.

        Do you think these are valid use cases?

        Show
        Thiruvel Thirumoolan added a comment - @Amareshwari Post altering, new tables will be created under new location. Old tables' have the fully qualified location in metadata and they should continue to work as before. The reasons I went with alter location are: 1. Allow migration to happen if one would like to reorganize or if quota runs out. Not sure how many folks have this situation. 2. One can migrate new tables to another DFS cluster (existing cluster becoming full). 3. One can migrate between file systems if they have sufficient use cases. Do you think these are valid use cases?
        Hide
        Amareshwari Sriramadasu added a comment -

        Use cases make sense. But, drop database would remove tables only from new location.
        Does anyone have the idea why alter Database currently allows to change only DB properties? Namit/Ning?

        Post altering, new tables will be created under new location. Old tables' have the fully qualified location in metadata and they should continue to work as before.

        Thiruvel, did you get a chance to test this? Because, your changes in patch does not look complete. Changes should be propagated to PersistentManager through ObjectStore.

        Show
        Amareshwari Sriramadasu added a comment - Use cases make sense. But, drop database would remove tables only from new location. Does anyone have the idea why alter Database currently allows to change only DB properties? Namit/Ning? Post altering, new tables will be created under new location. Old tables' have the fully qualified location in metadata and they should continue to work as before. Thiruvel, did you get a chance to test this? Because, your changes in patch does not look complete. Changes should be propagated to PersistentManager through ObjectStore.
        Hide
        Thiruvel Thirumoolan added a comment -

        > Use cases make sense. But, drop database would remove tables only from new location.

        If I am not wrong, drop db succeeds only if all tables under it are dropped.

        > Thiruvel, did you get a chance to test this? Because, your changes in patch does not look complete. Changes should be propagated to PersistentManager through ObjectStore.

        Will take a look.

        Show
        Thiruvel Thirumoolan added a comment - > Use cases make sense. But, drop database would remove tables only from new location. If I am not wrong, drop db succeeds only if all tables under it are dropped. > Thiruvel, did you get a chance to test this? Because, your changes in patch does not look complete. Changes should be propagated to PersistentManager through ObjectStore. Will take a look.
        Hide
        Thiruvel Thirumoolan added a comment -

        HIVE-1537 is supporting LOCATION in create database command
        HIVE-675 - first comment is consistent with HIVE-1537
        HIVE-1820 is about warehouse location through DBPROPERTIES

        My approach on this JIRA so far was based on HIVE-1820.

        What is the suggested way to specify location during create database?

        @Carl, @Ning, @He Yongqiang - suggestions?

        Show
        Thiruvel Thirumoolan added a comment - HIVE-1537 is supporting LOCATION in create database command HIVE-675 - first comment is consistent with HIVE-1537 HIVE-1820 is about warehouse location through DBPROPERTIES My approach on this JIRA so far was based on HIVE-1820 . What is the suggested way to specify location during create database? @Carl, @Ning, @He Yongqiang - suggestions?
        Hide
        Ning Zhang added a comment -

        Thiruvel, HIVE-1820 (actually the subtask of it HIVE-1836) is not intended to define the database LOCATION. The database metastore schema already contains a LOCATION field, but it was not currently exposed through the CREATE DATABASE syntax. That's what HIVE-1537 is about to solve. HIVE-1836 is to introduce dbproperties of generic key-value pairs. The dbproperties can be used/interpreted by application-level pre-execution hooks. We currently do not require the warehouse.dir (if it is one of the keys) must be consistent with the LOCATION field, and I think probably we don't need to keep 2 piece of information with the same semantics.

        If you want to change the database location URL I think probably a better way is to change the CREATE DATABASE syntax to add location (HIVE-1537) and any table created under that database should honor the database location property. What do you think?

        Show
        Ning Zhang added a comment - Thiruvel, HIVE-1820 (actually the subtask of it HIVE-1836 ) is not intended to define the database LOCATION. The database metastore schema already contains a LOCATION field, but it was not currently exposed through the CREATE DATABASE syntax. That's what HIVE-1537 is about to solve. HIVE-1836 is to introduce dbproperties of generic key-value pairs. The dbproperties can be used/interpreted by application-level pre-execution hooks. We currently do not require the warehouse.dir (if it is one of the keys) must be consistent with the LOCATION field, and I think probably we don't need to keep 2 piece of information with the same semantics. If you want to change the database location URL I think probably a better way is to change the CREATE DATABASE syntax to add location ( HIVE-1537 ) and any table created under that database should honor the database location property. What do you think?
        Hide
        Amareshwari Sriramadasu added a comment -

        I agree with Ning. There should not be two piece of information with same semantics. Location should be added to create database and alter database if it can be altered. Also, We should document that dbproperties are interpreted at application level.

        Show
        Amareshwari Sriramadasu added a comment - I agree with Ning. There should not be two piece of information with same semantics. Location should be added to create database and alter database if it can be altered. Also, We should document that dbproperties are interpreted at application level.
        Hide
        Thiruvel Thirumoolan added a comment -

        I prefer HIVE-1537 approach. I got confused by the test queries and found 1537 only later.

        Thanks for the clarification.

        Show
        Thiruvel Thirumoolan added a comment - I prefer HIVE-1537 approach. I got confused by the test queries and found 1537 only later. Thanks for the clarification.
        Hide
        Thiruvel Thirumoolan added a comment -

        HIVE-1537 will address specifying a location for a db.

        Show
        Thiruvel Thirumoolan added a comment - HIVE-1537 will address specifying a location for a db.

          People

          • Assignee:
            Thiruvel Thirumoolan
            Reporter:
            Thiruvel Thirumoolan
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development