Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-2012

OOM while writing history

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      I met OOM when querying on 10TB data set.
      Here is the log.

      2015-12-08 03:39:13,591 INFO org.apache.tajo.util.history.HistoryWriter: Saving query unit: hdfs://appliance007:9010/tmp/tajo-hadoop/staging/history/20151208/query-detail/q_1449490267507_0003/eb_1449490267507_0003_000001.hist
      
      2015-12-08 03:39:22,520 ERROR org.apache.tajo.util.history.HistoryWriter: Error while saving query history: q_1449490267507_0003:Requested array size exceeds VM limit
      
      java.lang.OutOfMemoryError: Requested array size exceeds VM limit
      
              at java.util.Arrays.copyOf(Arrays.java:3332)
      
              at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
      
              at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
      
              at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
      
              at java.lang.StringBuffer.append(StringBuffer.java:272)
      
              at java.io.StringWriter.write(StringWriter.java:112)
      
              at com.google.gson.stream.JsonWriter.string(JsonWriter.java:538)
      
              at com.google.gson.stream.JsonWriter.value(JsonWriter.java:404)
      
              at com.google.gson.internal.bind.TypeAdapters$13.write(TypeAdapters.java:353)
      
              at com.google.gson.internal.bind.TypeAdapters$13.write(TypeAdapters.java:337)
      
              at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
      
              at com.google.gson.internal.bind.ArrayTypeAdapter.write(ArrayTypeAdapter.java:93)
      
              at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
      
              at com.google.gson.internal.bind.ArrayTypeAdapter.write(ArrayTypeAdapter.java:93)
      
              at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
      
              at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:89)
      
              at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:195)
      
              at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
      
              at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:96)
      
              at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:60)
      
              at com.google.gson.Gson.toJson(Gson.java:586)
      
              at com.google.gson.Gson.toJson(Gson.java:565)
      
              at com.google.gson.Gson.toJson(Gson.java:520)
      
              at org.apache.tajo.util.history.StageHistory.toTasksJson(StageHistory.java:233)
      
              at org.apache.tajo.util.history.HistoryWriter$WriterThread.writeQueryHistory(HistoryWriter.java:378)
      
              at org.apache.tajo.util.history.HistoryWriter$WriterThread.writeHistory(HistoryWriter.java:318)
      
              at org.apache.tajo.util.history.HistoryWriter$WriterThread.run(HistoryWriter.java:244)
      
      2015-12-08 09:11:08,538 INFO org.apache.tajo.util.history.HistoryCleaner: Cleaning task history dir: file:/tmp/tajo-hadoop/history/20151201
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jihoonson Jihoon Son
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: