Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-13593

PutIceberg issue with decimal scale

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.28.0, 2.0.0
    • None
    • None

    Description

      A problem was observed when PutIceberg processor was used with a table that contained a column accepting decimal values and the datatype was decimal(7,2).

      When the following value was present in the incoming record, 
      "cost": 351.00,
      the JsonTreeReader trimmed the second zero and returned just "351.0".

      That value was not accepted by PutIceberg processor, it reported an error:

      java.lang.IllegalArgumentException: Cannot write value as decimal(7,2), wrong scale: 351.0

       

      Full trace:

      2024-07-23 07:23:54,883 ERROR org.apache.nifi.processors.iceberg.PutIceberg: PutIceberg[id=ebfa9858-ce4c-3b0a-b9dc-b347b5b7031a] Exception occurred while writing iceberg records. Removing uncommitted data files
      java.lang.IllegalArgumentException: Cannot write value as decimal(7,2), wrong scale: 351.0
          at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:463)
          at org.apache.iceberg.parquet.ParquetValueWriters$IntegerDecimalWriter.write(ParquetValueWriters.java:247)
          at org.apache.iceberg.parquet.ParquetValueWriters$IntegerDecimalWriter.write(ParquetValueWriters.java:235)
          at org.apache.iceberg.parquet.ParquetValueWriters$OptionWriter.write(ParquetValueWriters.java:356)
          at org.apache.iceberg.parquet.ParquetValueWriters$StructWriter.write(ParquetValueWriters.java:589)
          at org.apache.iceberg.parquet.ParquetWriter.add(ParquetWriter.java:138)
          at org.apache.iceberg.io.DataWriter.write(DataWriter.java:71)
          at org.apache.iceberg.io.BaseTaskWriter$RollingFileWriter.write(BaseTaskWriter.java:362)
          at org.apache.iceberg.io.BaseTaskWriter$RollingFileWriter.write(BaseTaskWriter.java:345)
          at org.apache.iceberg.io.BaseTaskWriter$BaseRollingWriter.write(BaseTaskWriter.java:277)
          at org.apache.iceberg.io.UnpartitionedWriter.write(UnpartitionedWriter.java:42)
          at org.apache.nifi.processors.iceberg.PutIceberg.doOnTrigger(PutIceberg.java:305)
       

       

      Attachments

        Issue Links

          Activity

            People

              kzsihovszki Zsihovszki Krisztina
              kzsihovszki Zsihovszki Krisztina
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m