Hive
  1. Hive
  2. HIVE-4220

TimestampWritable.toString throws array index exception sometimes

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 45
              at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:215)
              at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:170)
              at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:288)
              at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:348)
              at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
              at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
              at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
              at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
              at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      Caused by: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 45
              at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:194)
              at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1449)
              at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:193)
              ... 11 more
      Caused by: java.lang.ArrayIndexOutOfBoundsException: 45
              at sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:436)
              at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2081)
              at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:1996)
              at java.util.Calendar.setTimeInMillis(Calendar.java:1110)
              at java.util.Calendar.setTime(Calendar.java:1076)
              at java.text.SimpleDateFormat.format(SimpleDateFormat.java:875)
              at java.text.SimpleDateFormat.format(SimpleDateFormat.java:868)
              at java.text.DateFormat.format(DateFormat.java:316)
              at org.apache.hadoop.hive.serde2.io.TimestampWritable.toString(TimestampWritable.java:327)
              at org.apache.hadoop.hive.serde2.lazy.LazyTimestamp.writeUTF8(LazyTimestamp.java:95)
              at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:234)
              at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:427)
              at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:381)
              at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:365)
              at org.apache.hadoop.hive.ql.exec.ListSinkOperator.processOp(ListSinkOperator.java:96)
              at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:487)
              at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:821)
              at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
              at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:487)
              at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:821)
              at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
              at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:487)
              at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:474)
              at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:468)
              at org.apache.hadoop.hive.ql.exec.FetchTask.fetchAndPush(FetchTask.java:222)
              at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:188)
              ... 13 more
      

      data formatter in TimestampWritable is declared static and shared but it's not thread-safe.

        Issue Links

          Activity

          Hide
          Navis added a comment -

          Mikhail Bautin The default value of max-worker of HiveServer is Integer.MAX and I've thought it could make too many formatters in some (erroneous) situation. But admittedly, it's safer and cleaner. +1 and running test.

          Show
          Navis added a comment - Mikhail Bautin The default value of max-worker of HiveServer is Integer.MAX and I've thought it could make too many formatters in some (erroneous) situation. But admittedly, it's safer and cleaner. +1 and running test.
          Hide
          Mikhail Bautin added a comment -

          Navis: I think HIVE-4516 solves the same problem but in somewhat simpler way (using ThreadLocal). Could you please take a look at the patch there and let us know what you think?

          Ashutosh Chauhan: I think it is reasonable to assume at this point that Hive primitives, especially as low-level as TimestampWritable, have to be thread-safe. This is not only required by third-party low-latency query processing systems such as AmpLab's Shark, but also by the effort in the Hive community itself to speed up query processing (e.g. http://hortonworks.com/blog/introducing-tez-faster-hadoop-processing/) that I believe will inevitably require keeping pre-existing multi-threaded "executor" JVMs around.

          Show
          Mikhail Bautin added a comment - Navis : I think HIVE-4516 solves the same problem but in somewhat simpler way (using ThreadLocal). Could you please take a look at the patch there and let us know what you think? Ashutosh Chauhan : I think it is reasonable to assume at this point that Hive primitives, especially as low-level as TimestampWritable, have to be thread-safe. This is not only required by third-party low-latency query processing systems such as AmpLab's Shark, but also by the effort in the Hive community itself to speed up query processing (e.g. http://hortonworks.com/blog/introducing-tez-faster-hadoop-processing/ ) that I believe will inevitably require keeping pre-existing multi-threaded "executor" JVMs around.
          Hide
          Navis added a comment -

          Yes it is, and it's very hard to reproduce. It happened once on hive server when multiple clients are fetching rows containing timestamp and after applying this, not happened till now.

          Show
          Navis added a comment - Yes it is, and it's very hard to reproduce. It happened once on hive server when multiple clients are fetching rows containing timestamp and after applying this, not happened till now.
          Hide
          Ashutosh Chauhan added a comment -

          Since there is no test-case, I am having hard time envisioning the scenario where this might occur. Is it that there is a concurrent access of this non-thread-safe formatter where this will occur? Explaining the scenario where this bug will showup will be useful.

          Show
          Ashutosh Chauhan added a comment - Since there is no test-case, I am having hard time envisioning the scenario where this might occur. Is it that there is a concurrent access of this non-thread-safe formatter where this will occur? Explaining the scenario where this bug will showup will be useful.
          Hide
          Phabricator added a comment -

          navis requested code review of "HIVE-4220 [jira] TimestampWritable.toString throws array index exception sometimes".

          Reviewers: JIRA

          HIVE-4220 TimestampWritable.toString throws array index exception sometimes

          org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 45
          at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:215)
          at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:170)
          at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:288)
          at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:348)
          at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
          at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
          at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
          at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
          at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:662)
          Caused by: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 45
          at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:194)
          at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1449)
          at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:193)
          ... 11 more
          Caused by: java.lang.ArrayIndexOutOfBoundsException: 45
          at sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:436)
          at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2081)
          at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:1996)
          at java.util.Calendar.setTimeInMillis(Calendar.java:1110)
          at java.util.Calendar.setTime(Calendar.java:1076)
          at java.text.SimpleDateFormat.format(SimpleDateFormat.java:875)
          at java.text.SimpleDateFormat.format(SimpleDateFormat.java:868)
          at java.text.DateFormat.format(DateFormat.java:316)
          at org.apache.hadoop.hive.serde2.io.TimestampWritable.toString(TimestampWritable.java:327)
          at org.apache.hadoop.hive.serde2.lazy.LazyTimestamp.writeUTF8(LazyTimestamp.java:95)
          at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:234)
          at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:427)
          at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:381)
          at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:365)
          at org.apache.hadoop.hive.ql.exec.ListSinkOperator.processOp(ListSinkOperator.java:96)
          at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:487)
          at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:821)
          at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
          at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:487)
          at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:821)
          at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
          at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:487)
          at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:474)
          at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:468)
          at org.apache.hadoop.hive.ql.exec.FetchTask.fetchAndPush(FetchTask.java:222)
          at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:188)
          ... 13 more

          data formatter in TimestampWritable is declared static and shared but it's not thread-safe.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D9669

          AFFECTED FILES
          serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java

          MANAGE HERALD RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/23067/

          To: JIRA, navis

          Show
          Phabricator added a comment - navis requested code review of " HIVE-4220 [jira] TimestampWritable.toString throws array index exception sometimes". Reviewers: JIRA HIVE-4220 TimestampWritable.toString throws array index exception sometimes org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 45 at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:215) at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:170) at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:288) at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:348) at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553) at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 45 at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:194) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1449) at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:193) ... 11 more Caused by: java.lang.ArrayIndexOutOfBoundsException: 45 at sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:436) at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2081) at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:1996) at java.util.Calendar.setTimeInMillis(Calendar.java:1110) at java.util.Calendar.setTime(Calendar.java:1076) at java.text.SimpleDateFormat.format(SimpleDateFormat.java:875) at java.text.SimpleDateFormat.format(SimpleDateFormat.java:868) at java.text.DateFormat.format(DateFormat.java:316) at org.apache.hadoop.hive.serde2.io.TimestampWritable.toString(TimestampWritable.java:327) at org.apache.hadoop.hive.serde2.lazy.LazyTimestamp.writeUTF8(LazyTimestamp.java:95) at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:234) at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:427) at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:381) at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:365) at org.apache.hadoop.hive.ql.exec.ListSinkOperator.processOp(ListSinkOperator.java:96) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:487) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:821) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:487) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:821) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:487) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:474) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:468) at org.apache.hadoop.hive.ql.exec.FetchTask.fetchAndPush(FetchTask.java:222) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:188) ... 13 more data formatter in TimestampWritable is declared static and shared but it's not thread-safe. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D9669 AFFECTED FILES serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/23067/ To: JIRA, navis

            People

            • Assignee:
              Navis
              Reporter:
              Navis
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development