I'm loading a CSV file into a dataframe using Spark. I have defined a Schema and specified one of the fields as non-nullable.
When setting the mode to DROPMALFORMED, I expect any rows in the CSV with missing (null) values for those columns to result in the whole row being dropped. At the moment, the CSV loader correctly drops rows that do not conform to the field type, but the nullable property is seemingly ignored.
Example CSV input:
Example Spark job:
The actual output will be:
Note that the row containing non-integer values has been dropped, as expected, but rows containing null values persist, despite the nullable property being set to false in the schema definition.
My expected output is: