Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11837

comments do not support unicode characters well.

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 0.13.1, 1.1.0
    • None
    • Hive
    • None
    • Hadoop 2.7
      Hive 0.13.1 / Hive 1.1.0
      RHEL 6.4 / SLES 11.3

    Description

      the terminal encoding is set to UTF-8, It can display Chinese characters.

      then I create a table with a comment in Chinese, both "show create table" and "desc formatted table" can not display the Chinese characters in the table comments, meanwhile it can display Chinese characters in column comment.. See below:

      0: jdbc:hive2://ha-cluster/default> create table tt(id int comment '列中文测试') comment '表中文测试';
      No rows affected (0.152 seconds)
      0: jdbc:hive2://ha-cluster/default>
      0: jdbc:hive2://ha-cluster/default>
      0: jdbc:hive2://ha-cluster/default> desc formatted tt;
      -------------------------------------------------------------------------------------------------------------------

      col_name data_type comment

      -------------------------------------------------------------------------------------------------------------------

      1. col_name
      data_type comment
        NULL NULL
      id int 列中文测试
        NULL NULL
      1. Detailed Table Information
      NULL NULL
      Database: default NULL
      Owner: admin NULL
      CreateTime: Wed Sep 16 11:13:34 CST 2015 NULL
      LastAccessTime: UNKNOWN NULL
      Protect Mode: None NULL
      Retention: 0 NULL
      Location: hdfs://hacluster/user/hive/warehouse/tt NULL
      Table Type: MANAGED_TABLE NULL
      Table Parameters: NULL NULL
        comment \u8868\u4E2D\u6587\u6D4B\u8BD5
        transient_lastDdlTime 1442373214
        NULL NULL
      1. Storage Information
      NULL NULL
      SerDe Library: org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe NULL
      InputFormat: org.apache.hadoop.hive.ql.io.RCFileInputFormat NULL
      OutputFormat: org.apache.hadoop.hive.ql.io.RCFileOutputFormat NULL
      Compressed: No NULL
      Num Buckets: -1 NULL
      Bucket Columns: [] NULL
      Sort Columns: [] NULL
      Storage Desc Params: NULL NULL
        serialization.format 1

      -------------------------------------------------------------------------------------------------------------------
      27 rows selected (0.172 seconds)
      0: jdbc:hive2://ha-cluster/default> show create table tt;
      ------------------------------------------------------------

      createtab_stmt

      ------------------------------------------------------------

      CREATE TABLE `tt`(
      `id` int COMMENT '-�K�')
      COMMENT 'h-�K�'
      ROW FORMAT SERDE
      'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
      STORED AS INPUTFORMAT
      'org.apache.hadoop.hive.ql.io.RCFileInputFormat'
      OUTPUTFORMAT
      'org.apache.hadoop.hive.ql.io.RCFileOutputFormat'
      LOCATION
      'hdfs://hacluster/user/hive/warehouse/tt'
      TBLPROPERTIES (
      'transient_lastDdlTime'='1442373214')

      ------------------------------------------------------------
      13 rows selected (0.129 seconds)
      0: jdbc:hive2://ha-cluster/default>

      Attachments

        1. HIVE-11837.1.patch
          1 kB
          Yongzhi Chen
        2. HIVE-11837.patch
          1 kB
          Rudd Chen

        Issue Links

          Activity

            I guess I'll resolve the other as dup as this has more info

            sershe Sergey Shelukhin added a comment - I guess I'll resolve the other as dup as this has more info
            LueChen Rudd Chen added a comment -

            o, maybe the unicode has not been supported compeletely.

            but to this issue, the patch above can solve. I think it might be a little carelessness.

            LueChen Rudd Chen added a comment - o, maybe the unicode has not been supported compeletely. but to this issue, the patch above can solve. I think it might be a little carelessness.
            ychena Yongzhi Chen added a comment -

            The original fix is outdated. Attach a new fix.

            ychena Yongzhi Chen added a comment - The original fix is outdated. Attach a new fix.
            szehon Szehon Ho added a comment -

            Looks pretty simple, +1. Guess jira description should be 'show create table does not show unicode characters well'?

            szehon Szehon Ho added a comment - Looks pretty simple, +1. Guess jira description should be 'show create table does not show unicode characters well'?
            hiveqa Hive QA added a comment -

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12791019/HIVE-11837.1.patch

            ERROR: -1 due to no test(s) being added or modified.

            ERROR: -1 due to 9 failed/errored test(s), 9736 tests executed
            Failed tests:

            TestMiniTezCliDriver-vector_decimal_2.q-schema_evol_text_fetchwork_table.q-constprog_semijoin.q-and-12-more - did not produce a TEST-*.xml file
            TestMiniTezCliDriver-vector_decimal_round.q-cbo_windowing.q-tez_schema_evolution.q-and-12-more - did not produce a TEST-*.xml file
            TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not produce a TEST-*.xml file
            TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more - did not produce a TEST-*.xml file
            TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more - did not produce a TEST-*.xml file
            TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not produce a TEST-*.xml file
            TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more - did not produce a TEST-*.xml file
            org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
            org.apache.hive.jdbc.TestSSL.testSSLVersion
            

            Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7161/testReport
            Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7161/console
            Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7161/

            Messages:

            Executing org.apache.hive.ptest.execution.TestCheckPhase
            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 9 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12791019 - PreCommit-HIVE-TRUNK-Build

            hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12791019/HIVE-11837.1.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 9 failed/errored test(s), 9736 tests executed Failed tests: TestMiniTezCliDriver-vector_decimal_2.q-schema_evol_text_fetchwork_table.q-constprog_semijoin.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vector_decimal_round.q-cbo_windowing.q-tez_schema_evolution.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hive.jdbc.TestSSL.testSSLVersion Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7161/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7161/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7161/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed This message is automatically generated. ATTACHMENT ID: 12791019 - PreCommit-HIVE-TRUNK-Build
            ychena Yongzhi Chen added a comment -

            The failures are not related.
            +1

            ychena Yongzhi Chen added a comment - The failures are not related. +1
            ychena Yongzhi Chen added a comment -

            Need more research on how to make desc formatted work.

            ychena Yongzhi Chen added a comment - Need more research on how to make desc formatted work.
            cxgu@sina.com MK added a comment -

            meet this issue in 0.10 / 1.1.0 / 1.2.1

            cxgu@sina.com MK added a comment - meet this issue in 0.10 / 1.1.0 / 1.2.1
            darkz darkz added a comment -

            Why this path not merge into the hive code so long

            darkz darkz added a comment - Why this path not merge into the hive code so long
            hiveqa Hive QA added a comment -

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12791019/HIVE-11837.1.patch

            ERROR: -1 due to build exiting with an error

            Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3991/testReport
            Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3991/console
            Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3991/

            Messages:

            Executing org.apache.hive.ptest.execution.TestCheckPhase
            Executing org.apache.hive.ptest.execution.PrepPhase
            Tests exited with: NonZeroExitCodeException
            Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N'
            2017-03-07 11:23:43.327
            + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
            + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
            + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
            + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
            + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
            + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
            + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
            + export 'MAVEN_OPTS=-Xmx1g '
            + MAVEN_OPTS='-Xmx1g '
            + cd /data/hiveptest/working/
            + tee /data/hiveptest/logs/PreCommit-HIVE-Build-3991/source-prep.txt
            + [[ false == \t\r\u\e ]]
            + mkdir -p maven ivy
            + [[ git = \s\v\n ]]
            + [[ git = \g\i\t ]]
            + [[ -z master ]]
            + [[ -d apache-github-source-source ]]
            + [[ ! -d apache-github-source-source/.git ]]
            + [[ ! -d apache-github-source-source ]]
            + date '+%Y-%m-%d %T.%3N'
            2017-03-07 11:23:43.330
            + cd apache-github-source-source
            + git fetch origin
            + git reset --hard HEAD
            HEAD is now at 79c5092 HIVE-16101: QTest failure BeeLine escape_comments after HIVE-16045(Peter Vary via Zoltan Haindirch)
            + git clean -f -d
            + git checkout master
            Already on 'master'
            Your branch is up-to-date with 'origin/master'.
            + git reset --hard origin/master
            HEAD is now at 79c5092 HIVE-16101: QTest failure BeeLine escape_comments after HIVE-16045(Peter Vary via Zoltan Haindirch)
            + git merge --ff-only origin/master
            Already up-to-date.
            + date '+%Y-%m-%d %T.%3N'
            2017-03-07 11:23:44.378
            + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
            + patchFilePath=/data/hiveptest/working/scratch/build.patch
            + [[ -f /data/hiveptest/working/scratch/build.patch ]]
            + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
            + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
            error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: No such file or directory
            The patch does not appear to apply with p0, p1, or p2
            + exit 1
            '
            

            This message is automatically generated.

            ATTACHMENT ID: 12791019 - PreCommit-HIVE-Build

            hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12791019/HIVE-11837.1.patch ERROR: -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3991/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3991/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3991/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-03-07 11:23:43.327 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-3991/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-03-07 11:23:43.330 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 79c5092 HIVE-16101: QTest failure BeeLine escape_comments after HIVE-16045(Peter Vary via Zoltan Haindirch) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 79c5092 HIVE-16101: QTest failure BeeLine escape_comments after HIVE-16045(Peter Vary via Zoltan Haindirch) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-03-07 11:23:44.378 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: No such file or directory The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12791019 - PreCommit-HIVE-Build
            ferhui Hui Fei added a comment -

            HIVE-14013 has fixed the same problem. we can resove it as duplicate

            ferhui Hui Fei added a comment - HIVE-14013 has fixed the same problem. we can resove it as duplicate
            ferhui Hui Fei added a comment -
            ferhui Hui Fei added a comment - CC aihuaxu
            aihuaxu Aihua Xu added a comment -

            Yeah. It should have been fixed by HIVE-14013. Resolve as dup now.

            aihuaxu Aihua Xu added a comment - Yeah. It should have been fixed by HIVE-14013 . Resolve as dup now.

            People

              LueChen Rudd Chen
              LueChen Rudd Chen
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: