Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8744

hbase_stats3.q test fails when paths stored at JDBCStatsUtils.getIdColumnName() are too large

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.15.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      This test is related to the bug HIVE-8065 where I am trying to support HDFS encryption. One of the enhancements to support it is to create a .hive-staging directory on the same table directory location where the query is executed.

      Now, when running the hbase_stats3.q test from a temporary directory that has a large path, then the new path, a combination of table location + .hive-staging + random temporary subdirectories, is too large to fit into the statistics table, so the path is truncated.

      This causes the following error:

      2014-11-04 08:57:36,680 ERROR [LocalJobRunner Map Task Executor #0]: jdbc.JDBCStatsPublisher (JDBCStatsPublisher.java:publishStat(199)) - Error during publishing statistics. 
      java.sql.SQLDataException: A truncation error was encountered trying to shrink VARCHAR 'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.&' to length 255.
      	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
      	at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
      	at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
      	at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
      	at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeLargeUpdate(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source)
      	at org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:148)
      	at org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher$2.run(JDBCStatsPublisher.java:145)
      	at org.apache.hadoop.hive.ql.exec.Utilities.executeWithRetry(Utilities.java:2667)
      	at org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.publishStat(JDBCStatsPublisher.java:161)
      	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1031)
      	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:870)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
      	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
      	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
      	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      Caused by: java.sql.SQLException: A truncation error was encountered trying to shrink VARCHAR 'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.&' to length 255.
      	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
      	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
      	... 30 more
      Caused by: ERROR 22001: A truncation error was encountered trying to shrink VARCHAR 'pfile:/home/hiveptest/hive-ptest-cloudera-slaves-ee9-24.vpc.&' to length 255.
      	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
      	at org.apache.derby.iapi.types.SQLChar.hasNonBlankChars(Unknown Source)
      	at org.apache.derby.iapi.types.SQLVarchar.normalize(Unknown Source)
      	at org.apache.derby.iapi.types.SQLVarchar.normalize(Unknown Source)
      	at org.apache.derby.iapi.types.DataTypeDescriptor.normalize(Unknown Source)
      	at org.apache.derby.impl.sql.execute.NormalizeResultSet.normalizeColumn(Unknown Source)
      	at org.apache.derby.impl.sql.execute.NormalizeResultSet.normalizeRow(Unknown Source)
      	at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown Source)
      	at org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown Source)
      	at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source)
      	at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
      	at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
      	... 24 more
      

      We should increment the size of the VARCHAR datatype in order to fix this.

        Attachments

        1. HIVE-8744.1.patch
          1.0 kB
          Sergio Peña
        2. HIVE-8744.2.patch
          2 kB
          Sergio Peña

          Issue Links

            Activity

              People

              • Assignee:
                spena Sergio Peña
                Reporter:
                spena Sergio Peña
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: