Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4282

Allow Impala to create columns whose type has more than 4000 characters

        Details

        • Type: Improvement
        • Status: Resolved
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: Impala 2.6.0, Impala 2.7.0, Impala 2.8.0
        • Fix Version/s: Impala 2.9.0
        • Component/s: Frontend
        • Labels:

          Description

          When creating table with complex datatype having more than 4000 characters in hive we received the following error

          FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:javax.jdo.JDODataStoreException: Add req_V2` (`CD_ID`,`COMMENT`,`COLUMN_NAME`,`TYPE_NAME`,`INTEGER_IDX`) VALUES (?,?,?,?,?)

          As a workaround I increased the width of the TYPE_NAME column in COLUMNS_V2 table of HMS DB which allowed me to create tables in hive.

          But in impala, the table creation fails in the analysis phase
          HiveServer2Error: AnalysisException: Type of column 'xxx' exceeds maximum type length of 4000 characters

          I see that the MAX_TYPE_NAME_LENGTH is configured to 4000 characters which is causing the problem. Could you please remove this hard limit or make it configurable

          https://github.com/cloudera/Impala/blob/cdh5.7.0-release/fe/src/test/java/com/cloudera/impala/analysis/AnalyzeDDLTest.java#L1416-L1418

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            alex.behm Alexander Behm added a comment -

            Greg Rahn FYI

            Show
            alex.behm Alexander Behm added a comment - Greg Rahn FYI
            Hide
            Permalink
            alex.behm Alexander Behm added a comment -

            commit 0c8715292e93769c71f8160ad85ecd57552ca585
            Author: Alex Behm <alex.behm@cloudera.com>
            Date: Wed Feb 15 23:15:18 2017 -0800

            IMPALA-4282: Remove max length check for type strings.

            During analysis, Impala used to enforce the default HMS
            limit on the number of characters of a type string.
            That enforcement has the benefit of a clear error
            message, but has the severe drawback of not having
            any workaround because the limit is baked into the code.
            Going above the 4000 limit is pretty easy with complex
            types, and several users have already run into this
            issue.

            This patch removes the Impala-side enforcement such
            that the 4000 limit can be increased by altering
            the corresponding columns in the Metastore's backend
            database.

            Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
            Reviewed-on: http://gerrit.cloudera.org:8080/6034
            Reviewed-by: Alex Behm <alex.behm@cloudera.com>
            Tested-by: Impala Public Jenkins

            Show
            alex.behm Alexander Behm added a comment - commit 0c8715292e93769c71f8160ad85ecd57552ca585 Author: Alex Behm <alex.behm@cloudera.com> Date: Wed Feb 15 23:15:18 2017 -0800 IMPALA-4282 : Remove max length check for type strings. During analysis, Impala used to enforce the default HMS limit on the number of characters of a type string. That enforcement has the benefit of a clear error message, but has the severe drawback of not having any workaround because the limit is baked into the code. Going above the 4000 limit is pretty easy with complex types, and several users have already run into this issue. This patch removes the Impala-side enforcement such that the 4000 limit can be increased by altering the corresponding columns in the Metastore's backend database. Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d Reviewed-on: http://gerrit.cloudera.org:8080/6034 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Impala Public Jenkins

              People

              • Assignee:
                alex.behm Alexander Behm
                Reporter:
                Venkat Sambath Venkat Sambath
              • Votes:
                2 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development