Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-502

OutputFormat has inconsistent context state in interface functions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.12.0
    • Component/s: IO
    • Labels:
      None

      Description

      I created example project to demonstrate problematic behaviour:
      https://github.com/sisidra/crunch-ofb

      1. FormatBundle config is not populated to Configuration in checkOutputSpecs:
      https://github.com/sisidra/crunch-ofb/blob/master/src/main/java/com/spotify/crunch/bugreport/MyOutputFormat.java#L39

      15/03/02 15:40:24 INFO bugreport.MyOutputFormat: my.config.key (checkOutputSpecs): null
      15/03/02 15:40:24 ERROR bugreport.MyOutputFormat: Wrong my.config.key value in checkOutputSpecs!
      

      2. TaskAttemptContext. getTaskAttemptID().toString() is different in getRecordWriter and getOutputCommitter:

      2015-03-02 15:40:38,960 INFO [main] com.spotify.crunch.bugreport.MyOutputFormat: TaskAttemptID (getOutputCommitter): attempt_1422406067005_0121_m_000000_0
      ...
      2015-03-02 15:40:39,789 INFO [main] com.spotify.crunch.bugreport.MyOutputFormat: TaskAttemptID (getRecordWriter): attempt_1422406067005_out0_0121_m_000000_0
      

        Attachments

        1. CRUNCH-502.patch
          4 kB
          Josh Wills
        2. CRUNCH-502b.patch
          4 kB
          Josh Wills
        3. CRUNCH-502c.patch
          4 kB
          Josh Wills

          Activity

            People

            • Assignee:
              jwills Josh Wills
              Reporter:
              martins.kalvans Mārtiņš Kalvāns
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: