Make DBOutputFormat us column names from setOutput(...)

• Key: HADOOP-4955
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Kevin Peterson
• Reporter: Kevin Peterson
• Affects Version/s: 0.19.0
• Fix Version/s: 0.19.1

Description

In org.apache.hadoop.mapred.lib.db.DBOutputFormat, the supplied names of the columns are not used for inserting values. The column names supplied to DBOutputFormat.setOutput(JobConf, String, String...) are used to determine the number of values to insert, but the order is dictated by the table definition of the underlying database. This affects the correct indices for DBWritable.write(PreparedStatement).

I will attach a patch that correctly maps these values.

I am characterizing this as a bug rather than an improvement because there may be existing code which implicitly relied on DBOutputFormat ignoring the supplied table names.

Comments

[Kevin Peterson]

I'm not able to connect to svn.apache.org to get latest source. This patch is relative to the 0.19.0 release.

[Kevin Peterson]

Let me know if I need to regenerate the patch in a different format.

[Enis Soztutar]

Somwtimes the names of the fields are not available, in which case we send a new String[] with null values in it. Could you change the patch to first check this :

if(fieldNames == null || fieldNames.length < 0 )
  throw new IllegalArgumentException("Fields names cannot be empty");

if(fieldNames[0] != null) {
    query.append(" (");
     .....
}

[Chris Douglas]

Canceling patch while Enis's comments are addressed.

The patch should apply with patch -p0 < patch

[Kevin Peterson]

I will incorporate Enis's comments into patch this weekend.

[Kevin Peterson]

Updated patch to allow supplying an array of nulls to indicate the number of fields without knowing their names. Added unit test covering this behavior.

I allowed zero length array. I'm not sure if there's a valid use for this, or if most databases would even allow it. But it's harmless to allow it.

[Enis Soztutar]

[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

I will commit this one once the tests finish.

[Enis Soztutar]

I committed this to trunk, 0.19 and 0.20 branches. Thanks Kevin!

[Hudson]

Integrated in Hadoop-trunk #756 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/756/)