Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9612

Turn off DEBUG logging for Lazy Objects for tests

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.2.0
    • None
    • None

    Description

      Our tests are collecting a tremendous amounts of logs:

      [root@ip-10-152-185-204 TestRCFile]# pwd
/data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-2709/succeeded/TestRCFile
[root@ip-10-152-185-204 TestRCFile]# ls -lh hive.log 
-rw-r--r-- 1 hiveptest hiveptest 143M Feb  8 03:54 hive.log

      Much of this logging is due to stack traces printed at DEBUG. 

      2015-02-08 00:54:07,942 DEBUG [main]: lazy.LazyDouble (LazyDouble.java:init(55)) - Data not in the Double data type range so converted to null. Given data is :
java.lang.NumberFormatException: empty String
        at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1011)
        at java.lang.Double.parseDouble(Double.java:540)
        at org.apache.hadoop.hive.serde2.lazy.LazyDouble.init(LazyDouble.java:51)
        at org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase$FieldInfo.uncheckedGetField(ColumnarStructBase.java:111)
        at org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase.getFieldsAsList(ColumnarStructBase.java:224)
        at org.apache.hadoop.hive.serde2.objectinspector.ColumnarStructObjectInspector.getStructFieldsDataAsList(ColumnarStructObjectInspector.java:76)
        at org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe.serialize(ColumnarSerDe.java:144)
        at org.apache.hadoop.hive.ql.io.TestRCFile.partialReadTest(TestRCFile.java:598)
        at org.apache.hadoop.hive.ql.io.TestRCFile.testWriteAndPartialRead(TestRCFile.java:417)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

2015-02-08 00:54:17,992 DEBUG [main]: lazy.LazyPrimitive (LazyPrimitive.java:logExceptionMessage(81)) - Data not in the INT data type range so converted to null. Given data is :
java.lang.Exception: For debugging purposes
	at org.apache.hadoop.hive.serde2.lazy.LazyPrimitive.logExceptionMessage(LazyPrimitive.java:81)
	at org.apache.hadoop.hive.serde2.lazy.LazyInteger.init(LazyInteger.java:59)
	at org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase$FieldInfo.uncheckedGetField(ColumnarStructBase.java:111)
	at org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase.getField(ColumnarStructBase.java:172)
	at org.apache.hadoop.hive.serde2.objectinspector.ColumnarStructObjectInspector.getStructFieldData(ColumnarStructObjectInspector.java:67)
	at org.apache.hadoop.hive.ql.io.TestRCFile.testSimpleReadAndWrite(TestRCFile.java:232)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

      I think we should disable logging for LazyPrimitive, LazyDouble, and friends for tests. I think we'd do that here:

      https://github.com/apache/hive/blob/trunk/ql/src/main/resources/hive-exec-log4j.properties
      and
      https://github.com/apache/hive/blob/trunk/data/conf/hive-log4j.properties

      Attachments

        1. HIVE-9612.1.patch
          0.6 kB
          Sergio Peña

        Activity

          People

            spena Sergio Peña
            brocknoland Brock Noland
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: