Pig
  1. Pig
  2. PIG-188

There seems to be some mismatches between the actual stderr log and what I expected

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None

      Description

      With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0181, there were a total of 21 tasks (1 map and 20 reduces).

      From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.

      However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.

      In addition, the Input-split *** values in the logs do not make much sense to me:

      Input-split file: null
      Input-split start-offset: -1
      Input-split length: -1

      Here is Pig script:

      define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
      A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
      B = stream A through X;
      store B into 'logging_test_1';
      C = load 'logging_test_1/_logs/logging_test_1';
      store C into 'results_26';
      

      Here are the logs:

      ===== Task Information Header =====
      Command: PigLoggingTest 10 t 
      Start time: Fri Apr 04 19:18:44 PDT 2008
      Input-split file: null
      Input-split start-offset: -1
      Input-split length: -1
      =====          * * *          =====
      This is stderr message number 1
      This is stderr message number 2
      This is stderr message number 3
      This is stderr message number 4
      This is stderr message number 5
      This is stderr message number 6
      This is stderr message number 7
      This is stderr message number 8
      This is stderr message number 9
      This is stderr message number 10
      ===== Task Information Footer =====
      End time: Fri Apr 04 19:18:45 PDT 2008
      Exit code: 0
      Input records: 10000
      Input bytes: 1898380 bytes 
      Output records: 4
      Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
      =====          * * *          =====
      ===== Task Information Header =====
      Command: PigLoggingTest 10 t 
      Start time: Fri Apr 04 19:31:34 PDT 2008
      Input-split file: null
      Input-split start-offset: -1
      Input-split length: -1
      =====          * * *          =====
      This is stderr message number 1
      This is stderr message number 2
      This is stderr message number 3
      This is stderr message number 4
      This is stderr message number 5
      This is stderr message number 6
      This is stderr message number 7
      This is stderr message number 8
      This is stderr message number 9
      This is stderr message number 10
      ===== Task Information Footer =====
      End time: Fri Apr 04 19:31:36 PDT 2008
      Exit code: 0
      Input records: 10000
      Input bytes: 1898380 bytes 
      Output records: 4
      Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
      =====          * * *          =====
      ===== Task Information Header =====
      Command: ./cplusplus/PigLoggingTest 10 t 
      Start time: Fri Apr 04 10:11:22 PDT 2008
      Input-split file: null
      Input-split start-offset: -1
      Input-split length: -1
      =====          * * *          =====
      ===== Task Information Footer =====
      End time: Fri Apr 04 10:11:22 PDT 2008
      Exit code: -127
      Input records: 747
      Input bytes: 141796 bytes 
      Output records: 0
      Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
      =====          * * *          =====
      ===== Task Information Header =====
      Command: ./cplusplus/PigLoggingTest 10 t 
      Start time: Fri Apr 04 10:11:28 PDT 2008
      Input-split file: null
      Input-split start-offset: -1
      Input-split length: -1
      =====          * * *          =====
      ===== Task Information Footer =====
      End time: Fri Apr 04 10:11:28 PDT 2008
      Exit code: -127
      Input records: 747
      Input bytes: 141796 bytes 
      Output records: 0
      Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
      =====          * * *          =====
      ===== Task Information Header =====
      Command: ./cplusplus/PigLoggingTest 10 t 
      Start time: Fri Apr 04 10:11:32 PDT 2008
      Input-split file: null
      Input-split start-offset: -1
      Input-split length: -1
      =====          * * *          =====
      ===== Task Information Footer =====
      End time: Fri Apr 04 10:11:33 PDT 2008
      Exit code: -127
      Input records: 747
      Input bytes: 141796 bytes 
      Output records: 0
      Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
      =====          * * *          =====
      ===== Task Information Header =====
      Command: ./cplusplus/PigLoggingTest 10 t 
      Start time: Fri Apr 04 10:11:37 PDT 2008
      Input-split file: null
      Input-split start-offset: -1
      Input-split length: -1
      =====          * * *          =====
      ===== Task Information Footer =====
      End time: Fri Apr 04 10:11:37 PDT 2008
      Exit code: -127
      Input records: 747
      Input bytes: 141796 bytes 
      Output records: 0
      Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
      =====          * * *          =====
      
      1. PIG-188_0_20080407.patch
        2 kB
        Arun C Murthy
      2. PIG-188_1_20080407.patch
        4 kB
        Arun C Murthy
      3. PIG-188_2_20080408.patch
        4 kB
        Arun C Murthy
      4. PigLoggingTest.cpp
        2 kB
        Xu Zhang

        Activity

        Xu Zhang created issue -
        Xu Zhang made changes -
        Field Original Value New Value
        Attachment PigLoggingTest.cpp [ 12379461 ]
        Xu Zhang made changes -
        Description With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0182, there were a total of 26 tasks (6 maps and 20 reduces).

        From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.

        However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.

        In addition, the Input-split *** values in the logs do not make much sense to me:

        {quote}
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        {quote}

        Here is Pig script:

        {code}
        define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
        A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
        B = stream A through X;
        store B into 'logging_test_1';
        C = load 'logging_test_1/_logs/logging_test_1';
        store C into 'results_26';
        {code}

        Here are the logs:

        {noformat}
        ===== Task Information Header =====
        Command: PigLoggingTest 10 t
        Start time: Fri Apr 04 19:18:44 PDT 2008
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        ===== * * * =====
        This is stderr message number 1
        This is stderr message number 2
        This is stderr message number 3
        This is stderr message number 4
        This is stderr message number 5
        This is stderr message number 6
        This is stderr message number 7
        This is stderr message number 8
        This is stderr message number 9
        This is stderr message number 10
        ===== Task Information Footer =====
        End time: Fri Apr 04 19:18:45 PDT 2008
        Exit code: 0
        Input records: 10000
        Input bytes: 1898380 bytes
        Output records: 4
        Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
        ===== * * * =====
        ===== Task Information Header =====
        Command: PigLoggingTest 10 t
        Start time: Fri Apr 04 19:31:34 PDT 2008
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        ===== * * * =====
        This is stderr message number 1
        This is stderr message number 2
        This is stderr message number 3
        This is stderr message number 4
        This is stderr message number 5
        This is stderr message number 6
        This is stderr message number 7
        This is stderr message number 8
        This is stderr message number 9
        This is stderr message number 10
        ===== Task Information Footer =====
        End time: Fri Apr 04 19:31:36 PDT 2008
        Exit code: 0
        Input records: 10000
        Input bytes: 1898380 bytes
        Output records: 4
        Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
        ===== * * * =====
        ===== Task Information Header =====
        Command: ./cplusplus/PigLoggingTest 10 t
        Start time: Fri Apr 04 10:11:22 PDT 2008
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        ===== * * * =====
        ===== Task Information Footer =====
        End time: Fri Apr 04 10:11:22 PDT 2008
        Exit code: -127
        Input records: 747
        Input bytes: 141796 bytes
        Output records: 0
        Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
        ===== * * * =====
        ===== Task Information Header =====
        Command: ./cplusplus/PigLoggingTest 10 t
        Start time: Fri Apr 04 10:11:28 PDT 2008
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        ===== * * * =====
        ===== Task Information Footer =====
        End time: Fri Apr 04 10:11:28 PDT 2008
        Exit code: -127
        Input records: 747
        Input bytes: 141796 bytes
        Output records: 0
        Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
        ===== * * * =====
        ===== Task Information Header =====
        Command: ./cplusplus/PigLoggingTest 10 t
        Start time: Fri Apr 04 10:11:32 PDT 2008
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        ===== * * * =====
        ===== Task Information Footer =====
        End time: Fri Apr 04 10:11:33 PDT 2008
        Exit code: -127
        Input records: 747
        Input bytes: 141796 bytes
        Output records: 0
        Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
        ===== * * * =====
        ===== Task Information Header =====
        Command: ./cplusplus/PigLoggingTest 10 t
        Start time: Fri Apr 04 10:11:37 PDT 2008
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        ===== * * * =====
        ===== Task Information Footer =====
        End time: Fri Apr 04 10:11:37 PDT 2008
        Exit code: -127
        Input records: 747
        Input bytes: 141796 bytes
        Output records: 0
        Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
        ===== * * * =====
        {noformat}
        With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0181, there were a total of 21 tasks (1 map and 20 reduces).

        From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.

        However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.

        In addition, the Input-split *** values in the logs do not make much sense to me:

        {quote}
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        {quote}

        Here is Pig script:

        {code}
        define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
        A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
        B = stream A through X;
        store B into 'logging_test_1';
        C = load 'logging_test_1/_logs/logging_test_1';
        store C into 'results_26';
        {code}

        Here are the logs:

        {noformat}
        ===== Task Information Header =====
        Command: PigLoggingTest 10 t
        Start time: Fri Apr 04 19:18:44 PDT 2008
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        ===== * * * =====
        This is stderr message number 1
        This is stderr message number 2
        This is stderr message number 3
        This is stderr message number 4
        This is stderr message number 5
        This is stderr message number 6
        This is stderr message number 7
        This is stderr message number 8
        This is stderr message number 9
        This is stderr message number 10
        ===== Task Information Footer =====
        End time: Fri Apr 04 19:18:45 PDT 2008
        Exit code: 0
        Input records: 10000
        Input bytes: 1898380 bytes
        Output records: 4
        Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
        ===== * * * =====
        ===== Task Information Header =====
        Command: PigLoggingTest 10 t
        Start time: Fri Apr 04 19:31:34 PDT 2008
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        ===== * * * =====
        This is stderr message number 1
        This is stderr message number 2
        This is stderr message number 3
        This is stderr message number 4
        This is stderr message number 5
        This is stderr message number 6
        This is stderr message number 7
        This is stderr message number 8
        This is stderr message number 9
        This is stderr message number 10
        ===== Task Information Footer =====
        End time: Fri Apr 04 19:31:36 PDT 2008
        Exit code: 0
        Input records: 10000
        Input bytes: 1898380 bytes
        Output records: 4
        Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
        ===== * * * =====
        ===== Task Information Header =====
        Command: ./cplusplus/PigLoggingTest 10 t
        Start time: Fri Apr 04 10:11:22 PDT 2008
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        ===== * * * =====
        ===== Task Information Footer =====
        End time: Fri Apr 04 10:11:22 PDT 2008
        Exit code: -127
        Input records: 747
        Input bytes: 141796 bytes
        Output records: 0
        Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
        ===== * * * =====
        ===== Task Information Header =====
        Command: ./cplusplus/PigLoggingTest 10 t
        Start time: Fri Apr 04 10:11:28 PDT 2008
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        ===== * * * =====
        ===== Task Information Footer =====
        End time: Fri Apr 04 10:11:28 PDT 2008
        Exit code: -127
        Input records: 747
        Input bytes: 141796 bytes
        Output records: 0
        Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
        ===== * * * =====
        ===== Task Information Header =====
        Command: ./cplusplus/PigLoggingTest 10 t
        Start time: Fri Apr 04 10:11:32 PDT 2008
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        ===== * * * =====
        ===== Task Information Footer =====
        End time: Fri Apr 04 10:11:33 PDT 2008
        Exit code: -127
        Input records: 747
        Input bytes: 141796 bytes
        Output records: 0
        Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
        ===== * * * =====
        ===== Task Information Header =====
        Command: ./cplusplus/PigLoggingTest 10 t
        Start time: Fri Apr 04 10:11:37 PDT 2008
        Input-split file: null
        Input-split start-offset: -1
        Input-split length: -1
        ===== * * * =====
        ===== Task Information Footer =====
        End time: Fri Apr 04 10:11:37 PDT 2008
        Exit code: -127
        Input records: 747
        Input bytes: 141796 bytes
        Output records: 0
        Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
        ===== * * * =====
        {noformat}
        Arun C Murthy made changes -
        Attachment PIG-188_0_20080407.patch [ 12379619 ]
        Arun C Murthy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Attachment PIG-188_1_20080407.patch [ 12379631 ]
        Arun C Murthy made changes -
        Attachment PIG-188_2_20080408.patch [ 12379667 ]
        Alan Gates made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.1.0 [ 12312848 ]
        Resolution Fixed [ 1 ]
        Alan Gates made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Arun C Murthy
            Reporter:
            Xu Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development