Pig
  1. Pig
  2. PIG-3671

CONCAT operation bleeds into ToDate, making it ERROR

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: 0.12.0
    • Fix Version/s: None
    • Component/s: internal-udfs
    • Labels:
      None
    • Environment:

      Moonlight on the ocean last night was pretty.

      Description

      date_and_time = LOAD 'date_and_time.txt' AS (date:chararray, time:chararray);
      date_time_concat = FOREACH date_and_time GENERATE CONCAT(CONCAT(date, ' '), time) AS date_time;
      date_time_problem = FOREACH date_time_concat GENERATE ToDate(date_time) AS date_time:datetime;
      dump date_time_problem

      g.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing [POUserFunc (Name: POUserFunc(org.apache.pig.builtin.ToDateISO)[datetime] - scope-12 Operator Key: scope-12) children: null at []]: java.lang.IllegalArgumentException: Invalid format: "#date time"
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:338)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:378)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:298)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:282)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
      at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
      Caused by: java.lang.IllegalArgumentException: Invalid format: "#date time"
      at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:683)
      at org.apache.pig.builtin.ToDate.extractDateTime(ToDate.java:124)
      at org.apache.pig.builtin.ToDateISO.exec(ToDateISO.java:38)
      at org.apache.pig.builtin.ToDateISO.exec(ToDateISO.java:31)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:330)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNextDateTime(POUserFunc.java:422)
      at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:329)

      1. date_and_time.txt
        2 kB
        Russell Jurney
      2. date_time_bug.pig
        0.3 kB
        Russell Jurney

        Activity

        Russell Jurney made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Invalid [ 6 ]
        Russell Jurney made changes -
        Assignee Daniel Dai [ daijy ]
        Russell Jurney made changes -
        Attachment date_time_bug.pig [ 12623460 ]
        Russell Jurney made changes -
        Attachment date_time_bug.pig [ 12623459 ]
        Russell Jurney made changes -
        Description date_and_time = LOAD 'date_and_time.txt' AS (date:chararray, time:chararray);
        date_time_concat = FOREACH date_and_time GENERATE CONCAT(CONCAT(date, ' '), time) AS date_time;
        date_time_problem = FOREACH date_time_concat GENERATE ToDate(date_time) AS date_time:datetime;
        dump date_time_problem

        g.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing [POUserFunc (Name: POUserFunc(org.apache.pig.builtin.ToDateISO)[datetime] - scope-12 Operator Key: scope-12) children: null at []]: java.lang.IllegalArgumentException: Invalid format: "#date time"
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:338)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:378)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:298)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:282)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
        Caused by: java.lang.IllegalArgumentException: Invalid format: "#date time"
        at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:683)
        at org.apache.pig.builtin.ToDate.extractDateTime(ToDate.java:124)
        at org.apache.pig.builtin.ToDateISO.exec(ToDateISO.java:38)
        at org.apache.pig.builtin.ToDateISO.exec(ToDateISO.java:31)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:330)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNextDateTime(POUserFunc.java:422)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:329)
        Russell Jurney made changes -
        Attachment date_time_bug.pig [ 12623459 ]
        Russell Jurney made changes -
        Field Original Value New Value
        Attachment date_and_time.txt [ 12623456 ]
        Russell Jurney created issue -

          People

          • Assignee:
            Daniel Dai
            Reporter:
            Russell Jurney
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development