HBase
  1. HBase
  2. HBASE-10470

Import generates huge log file while importing large amounts of data

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.0, 0.96.2, 0.99.0, 0.94.17
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Committed to 0.94 and 0.96 as well.

      Description

      Import mapper has System.out.println statements for each key value if there is filter associated with the import. This is generating huge log file while importing large amounts of data. These statements must be changed to trace and log4j must be used for logging.

      1. 0.94-HBASE-10470.patch
        1.0 kB
        Vasu Mariyala
      2. trunk-HBASE-10470.patch
        2 kB
        Vasu Mariyala
      3. 10470.txt
        1.0 kB
        Lars Hofhansl

        Activity

        Hide
        stack added a comment -

        Thanks Lars Hofhansl Odd that builds failed on commit of this though seem unrelated. I started them all going tonight to see if we go blue again....

        Show
        stack added a comment - Thanks Lars Hofhansl Odd that builds failed on commit of this though seem unrelated. I started them all going tonight to see if we go blue again....
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in hbase-0.96 #282 (See https://builds.apache.org/job/hbase-0.96/282/)
        HBASE-10470 Import generates huge log file while importing large amounts of data. (Vasu Mariyala) (larsh: rev 1564958)

        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Show
        Hudson added a comment - SUCCESS: Integrated in hbase-0.96 #282 (See https://builds.apache.org/job/hbase-0.96/282/ ) HBASE-10470 Import generates huge log file while importing large amounts of data. (Vasu Mariyala) (larsh: rev 1564958) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in hbase-0.96-hadoop2 #194 (See https://builds.apache.org/job/hbase-0.96-hadoop2/194/)
        HBASE-10470 Import generates huge log file while importing large amounts of data. (Vasu Mariyala) (larsh: rev 1564958)

        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Show
        Hudson added a comment - FAILURE: Integrated in hbase-0.96-hadoop2 #194 (See https://builds.apache.org/job/hbase-0.96-hadoop2/194/ ) HBASE-10470 Import generates huge log file while importing large amounts of data. (Vasu Mariyala) (larsh: rev 1564958) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #125 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/125/)
        HBASE-10470 Import generates huge log file while importing large amounts of data (Vasu Mariyala) (tedyu: rev 1564932)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #125 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/125/ ) HBASE-10470 Import generates huge log file while importing large amounts of data (Vasu Mariyala) (tedyu: rev 1564932) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #80 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/80/)
        HBASE-10470 Import generates huge log file while importing large amounts of data (Vasu Mariyala) (tedyu: rev 1564933)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #80 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/80/ ) HBASE-10470 Import generates huge log file while importing large amounts of data (Vasu Mariyala) (tedyu: rev 1564933) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4890 (See https://builds.apache.org/job/HBase-TRUNK/4890/)
        HBASE-10470 Import generates huge log file while importing large amounts of data (Vasu Mariyala) (tedyu: rev 1564933)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4890 (See https://builds.apache.org/job/HBase-TRUNK/4890/ ) HBASE-10470 Import generates huge log file while importing large amounts of data (Vasu Mariyala) (tedyu: rev 1564933) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.94 #1273 (See https://builds.apache.org/job/HBase-0.94/1273/)
        HBASE-10470 Import generates huge log file while importing large amounts of data. (Vasu Mariyala) (larsh: rev 1564950)

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.94 #1273 (See https://builds.apache.org/job/HBase-0.94/1273/ ) HBASE-10470 Import generates huge log file while importing large amounts of data. (Vasu Mariyala) (larsh: rev 1564950) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.94-JDK7 #39 (See https://builds.apache.org/job/HBase-0.94-JDK7/39/)
        HBASE-10470 Import generates huge log file while importing large amounts of data. (Vasu Mariyala) (larsh: rev 1564950)

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.94-JDK7 #39 (See https://builds.apache.org/job/HBase-0.94-JDK7/39/ ) HBASE-10470 Import generates huge log file while importing large amounts of data. (Vasu Mariyala) (larsh: rev 1564950) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98 #135 (See https://builds.apache.org/job/HBase-0.98/135/)
        HBASE-10470 Import generates huge log file while importing large amounts of data (Vasu Mariyala) (tedyu: rev 1564932)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #135 (See https://builds.apache.org/job/HBase-0.98/135/ ) HBASE-10470 Import generates huge log file while importing large amounts of data (Vasu Mariyala) (tedyu: rev 1564932) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #10 (See https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/10/)
        HBASE-10470 Import generates huge log file while importing large amounts of data. (Vasu Mariyala) (larsh: rev 1564950)

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #10 (See https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/10/ ) HBASE-10470 Import generates huge log file while importing large amounts of data. (Vasu Mariyala) (larsh: rev 1564950) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.94-security #400 (See https://builds.apache.org/job/HBase-0.94-security/400/)
        HBASE-10470 Import generates huge log file while importing large amounts of data. (Vasu Mariyala) (larsh: rev 1564950)

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.94-security #400 (See https://builds.apache.org/job/HBase-0.94-security/400/ ) HBASE-10470 Import generates huge log file while importing large amounts of data. (Vasu Mariyala) (larsh: rev 1564950) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Hide
        Lars Hofhansl added a comment -

        0.96 patch.
        Makes it the same as 0.94.

        Show
        Lars Hofhansl added a comment - 0.96 patch. Makes it the same as 0.94.
        Hide
        Lars Hofhansl added a comment - - edited

        I'll commit this to 0.94 and 0.96 (stack assuming you want this)
        So that Andy can close this for the RC.

        Show
        Lars Hofhansl added a comment - - edited I'll commit this to 0.94 and 0.96 ( stack assuming you want this) So that Andy can close this for the RC.
        Hide
        Ted Yu added a comment -

        Integrated to 0.98 and trunk.

        Thanks for the patch, Vasu.

        Show
        Ted Yu added a comment - Integrated to 0.98 and trunk. Thanks for the patch, Vasu.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12627193/trunk-HBASE-10470.patch
        against trunk revision .
        ATTACHMENT ID: 12627193

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12627193/trunk-HBASE-10470.patch against trunk revision . ATTACHMENT ID: 12627193 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. -1 javadoc . The javadoc tool appears to have generated 2 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8605//console This message is automatically generated.
        Hide
        Andrew Purtell added a comment -

        +1

        I was working on a RC earlier but am now waiting for a security fix to show up later today.

        Show
        Andrew Purtell added a comment - +1 I was working on a RC earlier but am now waiting for a security fix to show up later today.
        Hide
        Lars Hofhansl added a comment -

        Andrew Purtell, if you haven't rolled the RC, yet, might as well pull it in. Agreed?

        Show
        Lars Hofhansl added a comment - Andrew Purtell , if you haven't rolled the RC, yet, might as well pull it in. Agreed?
        Hide
        Lars Hofhansl added a comment -

        +1
        (as discussed offline, this is terrible)

        Show
        Lars Hofhansl added a comment - +1 (as discussed offline, this is terrible)
        Hide
        Ted Yu added a comment -

        +1

        Show
        Ted Yu added a comment - +1

          People

          • Assignee:
            Vasu Mariyala
            Reporter:
            Vasu Mariyala
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development