Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9815

Metastore column"SERDE_PARAMS"."PARAM_VALUE" limited to 4000 bytes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 2.3.0, 3.0.0
    • Component/s: Metastore
    • Labels:
      None

      Description

      Currently, in the hive metastore schema, the length of the column SERDE_PARAMS.PARAM_VALUE is set to 4000 bytes. This is not enough for users that have a key with a value larger than 4000 bytes. Say something like hbase.columns.mapping.

      I am not a database historian but appears that this limitation may have been put in place because Oracle's varchar2 was restricted to 4k bytes for a long time until recently.
      According to the following documentation, even today Oracle DB's varchar2 only supports a max size of 4000 unless a configuration parameter MAX_STRING_SIZE is set to EXTENDED.
      http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623

      MAX_STRING_SIZE=EXTENDED
      

      Postgres supports a max of 1GB for character datatype according to http://www.postgresql.org/docs/8.3/static/datatype-character.html

      MySQL can support upto 65535 bytes for the entire row. So long as the PARAM_KEY value + PARAM_VALUE is less than 65535, we should be good.
      http://dev.mysql.com/doc/refman/5.0/en/char.html

      SQL Server's varchar max length is 8000 and can go beyond using "varchar(max)".
      http://dev.mysql.com/doc/refman/5.0/en/char.html

      Derby's varchar can be upto 32672 bytes.
      https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html

        Attachments

        1. Hv2.txt
          27 kB
          Naveen Gangam

          Issue Links

            Activity

              People

              • Assignee:
                ngangam Naveen Gangam
                Reporter:
                ngangam Naveen Gangam
              • Votes:
                4 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: