Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4282

Allow Impala to create columns whose type has more than 4000 characters

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.6.0, Impala 2.7.0, Impala 2.8.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Frontend
    • Labels:

      Description

      When creating table with complex datatype having more than 4000 characters in hive we received the following error

      FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:javax.jdo.JDODataStoreException: Add req_V2` (`CD_ID`,`COMMENT`,`COLUMN_NAME`,`TYPE_NAME`,`INTEGER_IDX`) VALUES (?,?,?,?,?)

      As a workaround I increased the width of the TYPE_NAME column in COLUMNS_V2 table of HMS DB which allowed me to create tables in hive.

      But in impala, the table creation fails in the analysis phase
      HiveServer2Error: AnalysisException: Type of column 'xxx' exceeds maximum type length of 4000 characters

      I see that the MAX_TYPE_NAME_LENGTH is configured to 4000 characters which is causing the problem. Could you please remove this hard limit or make it configurable

      https://github.com/cloudera/Impala/blob/cdh5.7.0-release/fe/src/test/java/com/cloudera/impala/analysis/AnalyzeDDLTest.java#L1416-L1418

        Activity

        Hide
        alex.behm Alexander Behm added a comment -
        Show
        alex.behm Alexander Behm added a comment - Greg Rahn FYI
        Hide
        alex.behm Alexander Behm added a comment -

        commit 0c8715292e93769c71f8160ad85ecd57552ca585
        Author: Alex Behm <alex.behm@cloudera.com>
        Date: Wed Feb 15 23:15:18 2017 -0800

        IMPALA-4282: Remove max length check for type strings.

        During analysis, Impala used to enforce the default HMS
        limit on the number of characters of a type string.
        That enforcement has the benefit of a clear error
        message, but has the severe drawback of not having
        any workaround because the limit is baked into the code.
        Going above the 4000 limit is pretty easy with complex
        types, and several users have already run into this
        issue.

        This patch removes the Impala-side enforcement such
        that the 4000 limit can be increased by altering
        the corresponding columns in the Metastore's backend
        database.

        Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
        Reviewed-on: http://gerrit.cloudera.org:8080/6034
        Reviewed-by: Alex Behm <alex.behm@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        alex.behm Alexander Behm added a comment - commit 0c8715292e93769c71f8160ad85ecd57552ca585 Author: Alex Behm <alex.behm@cloudera.com> Date: Wed Feb 15 23:15:18 2017 -0800 IMPALA-4282 : Remove max length check for type strings. During analysis, Impala used to enforce the default HMS limit on the number of characters of a type string. That enforcement has the benefit of a clear error message, but has the severe drawback of not having any workaround because the limit is baked into the code. Going above the 4000 limit is pretty easy with complex types, and several users have already run into this issue. This patch removes the Impala-side enforcement such that the 4000 limit can be increased by altering the corresponding columns in the Metastore's backend database. Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d Reviewed-on: http://gerrit.cloudera.org:8080/6034 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            alex.behm Alexander Behm
            Reporter:
            Venkat Sambath Venkat Sambath
          • Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development