Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12105

Beam Schema RowCoder can't roundtrip DateTime

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: P3
    • Resolution: Unresolved
    • Affects Version/s: 2.21.0, 2.22.0, 2.23.0, 2.24.0, 2.25.0, 2.26.0, 2.27.0, 2.28.0
    • Fix Version/s: None
    • Component/s: sdk-java-core
    • Labels:
      None

      Description

      Beam 2.21.0+ won't roundtrip DateTime. The test was changed in the PR that introduced the breakage: https://github.com/apache/beam/commit/fc6cef9972780ca6b7525d4aadd65a8344221f1b#diff-6dc06eab59c695aaa6c73cf608b392a6a6ab0809e72f1fc3b11805752b8eb69eL68

      This is what doesn't work:

      Row.withSchema(...).addValues(inputRow.getDateTime()).build();
      

      Workaround:

      Row.withSchema(...).addValues(inputRow.getDateTime().toInstant()).build();
      

      Stack trace:

      Caused by: java.lang.ClassCastException: org.joda.time.DateTime cannot be cast to org.joda.time.Instant                                    
              at org.apache.beam.sdk.coders.InstantCoder.encode(InstantCoder.java:34)                                                            
              at org.apache.beam.sdk.coders.RowCoderGenerator$EncodeInstruction.encodeDelegate(RowCoderGenerator.java:274)                       
              at org.apache.beam.sdk.coders.Coder$ByteBuddy$qZItcXc5.encode(Unknown Source)                                                      
              at org.apache.beam.sdk.coders.Coder$ByteBuddy$qZItcXc5.encode(Unknown Source)                                                      
              at org.apache.beam.sdk.schemas.SchemaCoder.encode(SchemaCoder.java:118)                                                            
              at org.apache.beam.sdk.coders.Coder.encode(Coder.java:136)                                                                         
              at org.apache.beam.sdk.util.CoderUtils.encodeToSafeStream(CoderUtils.java:85)                                                      
              at org.apache.beam.sdk.util.CoderUtils.encodeToByteArray(CoderUtils.java:69)                                                       
              at org.apache.beam.sdk.util.CoderUtils.encodeToByteArray(CoderUtils.java:54)                                                       
              at org.apache.beam.sdk.util.CoderUtils.clone(CoderUtils.java:144)                                                                  
              at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.<init>(MutationDetectors.java:118)                        
              at org.apache.beam.sdk.util.MutationDetectors.forValueWithCoder(MutationDetectors.java:49)                                         
              at org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add(ImmutabilityCheckingBundleFacto
      ry.java:115)                                                                                                                               
              at org.apache.beam.runners.direct.ParDoEvaluator$BundleOutputManager.output(ParDoEvaluator.java:305)                               
              at org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:272)             
              at org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:84)                       
              at org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:418)       
              at org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:406)  
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              apilloud Andrew Pilloud
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: