Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
This is the verbatim response from the user posting on nifi-dev:
Here’s a really stripped back example.
I have an input as below
{
"value": "123456.00"
}
And I want to write the value into MongoDB as a numeric value, but critically not as a double (these will be currency values)
In order to try to enforce the type used to write to Mongo we created a simple Avro schema, for the example above this would be as follows;
{
"type": "record",
"name": "TransactionEvent",
"namespace": "com.example.demo",
"fields": [
{
"name": "value",
"type": [
"null",
{
"type": "bytes",
"logicalType": "decimal",
"precision": 10,
"scale": 2
}
]
}
]
}
Hoping that this would map to a Decimal128 in Mongo, however we consistently see double as the type in Mongo regardless of any variations of Avro schema we have tried.
On having a quick look into the code I’ve identified 2 possible problem areas.
1. The conversion of the Avro schema into the internal representation which seems to treat Avro logical decimal types as double (ref org.apache.nifi.avro.AvroTypeUtil – line 343)
2. The Mongo processor which uses this type information to decide what Mongo types to persist data as.
For a quick win, which would hopefully have a smaller impact, I was hoping that I could fork the Mongo processor and keep the changes local to that but since the information about the Avro logical type is lost before the schema information gets to MongoDB i’m not sure that will be possible now.
When we reached this point and the changes we were looking at seemed like they could be a little more complex than hoped we wanted to reach out to see if
1. We’re doing something wrong
2. Anybody else had encountered a similar situation
3. If we did look to introduce changes either to the Mongo processor or more widely for support of BigDecimal would this be of wider use?
It would appear to be a distinctly different type than Double:
https://mongodb.github.io/mongo-java-driver/3.5/javadoc/?org/bson/types/Decimal128.html
Attachments
Issue Links
- links to
