[TAJO-2012] OOM while writing history

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 0.13.0
• Assignee: Unassigned
• Reporter: Jihoon Son
• Created: 08/Dec/15 03:00
• Updated: 02/Dec/17 17:33

Description

I met OOM when querying on 10TB data set.
Here is the log.

2015-12-08 03:39:13,591 INFO org.apache.tajo.util.history.HistoryWriter: Saving query unit: hdfs://appliance007:9010/tmp/tajo-hadoop/staging/history/20151208/query-detail/q_1449490267507_0003/eb_1449490267507_0003_000001.hist

2015-12-08 03:39:22,520 ERROR org.apache.tajo.util.history.HistoryWriter: Error while saving query history: q_1449490267507_0003:Requested array size exceeds VM limit

java.lang.OutOfMemoryError: Requested array size exceeds VM limit

        at java.util.Arrays.copyOf(Arrays.java:3332)

        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)

        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)

        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)

        at java.lang.StringBuffer.append(StringBuffer.java:272)

        at java.io.StringWriter.write(StringWriter.java:112)

        at com.google.gson.stream.JsonWriter.string(JsonWriter.java:538)

        at com.google.gson.stream.JsonWriter.value(JsonWriter.java:404)

        at com.google.gson.internal.bind.TypeAdapters$13.write(TypeAdapters.java:353)

        at com.google.gson.internal.bind.TypeAdapters$13.write(TypeAdapters.java:337)

        at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)

        at com.google.gson.internal.bind.ArrayTypeAdapter.write(ArrayTypeAdapter.java:93)

        at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)

        at com.google.gson.internal.bind.ArrayTypeAdapter.write(ArrayTypeAdapter.java:93)

        at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)

        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:89)

        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:195)

        at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)

        at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:96)

        at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:60)

        at com.google.gson.Gson.toJson(Gson.java:586)

        at com.google.gson.Gson.toJson(Gson.java:565)

        at com.google.gson.Gson.toJson(Gson.java:520)

        at org.apache.tajo.util.history.StageHistory.toTasksJson(StageHistory.java:233)

        at org.apache.tajo.util.history.HistoryWriter$WriterThread.writeQueryHistory(HistoryWriter.java:378)

        at org.apache.tajo.util.history.HistoryWriter$WriterThread.writeHistory(HistoryWriter.java:318)

        at org.apache.tajo.util.history.HistoryWriter$WriterThread.run(HistoryWriter.java:244)

2015-12-08 09:11:08,538 INFO org.apache.tajo.util.history.HistoryCleaner: Cleaning task history dir: file:/tmp/tajo-hadoop/history/20151201