Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: contrib/vertica
    • Labels:
      None
    • Environment:

      Hadoop 0.21.0 pre-release and Vertica 3.5

    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Adds support for Vertica 3.5 truncate table, deploy_design and numeric types.

      Description

      Vertica 3.5 includes three changes that the formatters should handle:

      1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
      2) truncate table instead of recreating the table
      3) numeric, decimal, money, number types (all the same path)

      1. MAPREDUCE-1097-3.patch
        13 kB
        Omer Trajman
      2. MAPREDUCE-1097-2.patch
        19 kB
        Omer Trajman
      3. MAPREDUCE-1097.patch
        17 kB
        Omer Trajman

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #216 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/216/)
        . Add support for Vertica 3.5 to its contrib module. Contributed by Omer Trajman

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #216 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/216/ ) . Add support for Vertica 3.5 to its contrib module. Contributed by Omer Trajman
        Hide
        Chris Douglas added a comment -

        Thought about this some more - since 0.21 hasn't been released and this is the first version that has this contrib seems like its ok to not handle the case of a persisted Writable with a double instead of a decimal. Changed to incompatible to make explicit.

        Since you're the sole contributor to this module, I view this as your decision. FWIW, as it hasn't yet been released, an incompatible change is exactly the risk early adopters take, in my opinion.

        I committed this. Thanks, Omer!

        Show
        Chris Douglas added a comment - Thought about this some more - since 0.21 hasn't been released and this is the first version that has this contrib seems like its ok to not handle the case of a persisted Writable with a double instead of a decimal. Changed to incompatible to make explicit. Since you're the sole contributor to this module, I view this as your decision. FWIW, as it hasn't yet been released, an incompatible change is exactly the risk early adopters take, in my opinion. I committed this. Thanks, Omer!
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12429841/MAPREDUCE-1097-3.patch
        against trunk revision 897118.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/371/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/371/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/371/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/371/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429841/MAPREDUCE-1097-3.patch against trunk revision 897118. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/371/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/371/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/371/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/371/console This message is automatically generated.
        Hide
        Omer Trajman added a comment -

        Thought about this some more - since 0.21 hasn't been released and this is the first version that has this contrib seems like its ok to not handle the case of a persisted Writable with a double instead of a decimal. Changed to incompatible to make explicit.

        Show
        Omer Trajman added a comment - Thought about this some more - since 0.21 hasn't been released and this is the first version that has this contrib seems like its ok to not handle the case of a persisted Writable with a double instead of a decimal. Changed to incompatible to make explicit.
        Hide
        Omer Trajman added a comment -

        Updated patch against latest changes.

        Show
        Omer Trajman added a comment - Updated patch against latest changes.
        Hide
        Aaron Kimball added a comment -

        VerticaRecord does implement Writable, so it's conceivable that users may have stored VerticaRecord instances in SequenceFiles on HDFS; regardless of whether you intended this to be an internal-only class, users do funny things.. but hopefully not too many will have done so. I don't think it's a big problem, especially since this is a fairly young contrib module. Some volatility should probably be expected here.

        At the very least, though, you should mark this issue as an "incompatible change" (see the "Edit this issue" link on the left).

        Thanks for making the other changes. I'm +0.75 on this patch – would appreciate it if someone else could weigh in on the issue of whether they think persistent VerticaRecord incompatibility is problematic or if they've got a creative solution. (Some day we'll all just use Avro and schema evolution will take care of all of this... but until then..)

        Show
        Aaron Kimball added a comment - VerticaRecord does implement Writable, so it's conceivable that users may have stored VerticaRecord instances in SequenceFiles on HDFS; regardless of whether you intended this to be an internal-only class, users do funny things.. but hopefully not too many will have done so. I don't think it's a big problem, especially since this is a fairly young contrib module. Some volatility should probably be expected here. At the very least, though, you should mark this issue as an "incompatible change" (see the "Edit this issue" link on the left). Thanks for making the other changes. I'm +0.75 on this patch – would appreciate it if someone else could weigh in on the issue of whether they think persistent VerticaRecord incompatibility is problematic or if they've got a creative solution. (Some day we'll all just use Avro and schema evolution will take care of all of this... but until then..)
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12427297/MAPREDUCE-1097-2.patch
        against trunk revision 888269.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/173/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/173/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/173/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/173/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427297/MAPREDUCE-1097-2.patch against trunk revision 888269. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/173/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/173/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/173/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/173/console This message is automatically generated.
        Hide
        Omer Trajman added a comment -

        fixes per Aaron's comments.

        Show
        Omer Trajman added a comment - fixes per Aaron's comments.
        Hide
        Omer Trajman added a comment -

        Thanks Aaron - I'll tackle these and post a new patch.

        On VerticaRecord - it's not supposed to be used as a persistent format so I didn't put in any handling for these changes. Not sure if there's any way to indicate that.

        Show
        Omer Trajman added a comment - Thanks Aaron - I'll tackle these and post a new patch. On VerticaRecord - it's not supposed to be used as a persistent format so I didn't put in any handling for these changes. Not sure if there's any way to indicate that.
        Hide
        Aaron Kimball added a comment -

        Omer,

        The gridmix failure is a known bug elsewhere in Hadoop; that's unrelated to this patch.

        As for the rest of the code you posted...

        VerticaOutputFormat:

        • make a constant VERSION_3_5 = 305 in VerticaUtil?
        • "pool for refresh complete" -> "poll..."
        • You select table_name, projection_name, and status from vt_projection_refresh, but you never seem to read projection_name from the ResultSet. Is this intentional?
        • Also, I think you should call ResultSet.close() before calling the next stmt.executeQuery()
        • Related, at the end of the function, I think you may want to call stmt.close() to ensure that all those resources are freed before continuing on.
        • What's an "ahm" ? (Add to source comment?)

        VerticaRecord:

        • line 378: no need to cast to BigDecimal before calling toString
        • It looks like REAL, DECIMAL, and NUMERIC used to be treated as doubles in Java, and are now treated as BigDecimal. Is this correct? If so, what happens to existing data (e.g., serialized into SequenceFiles) that contains data of this type? Is this an incompatible change?
        • It also looks as though you are changing how nulls are handled. Is this backwards compatible with existing VerticaRecord deployments?
        • line 575: unnecessary cast to BigDecimal.
        Show
        Aaron Kimball added a comment - Omer, The gridmix failure is a known bug elsewhere in Hadoop; that's unrelated to this patch. As for the rest of the code you posted... VerticaOutputFormat: make a constant VERSION_3_5 = 305 in VerticaUtil? "pool for refresh complete" -> "poll..." You select table_name, projection_name, and status from vt_projection_refresh, but you never seem to read projection_name from the ResultSet. Is this intentional? Also, I think you should call ResultSet.close() before calling the next stmt.executeQuery() Related, at the end of the function, I think you may want to call stmt.close() to ensure that all those resources are freed before continuing on. What's an "ahm" ? (Add to source comment?) VerticaRecord: line 378: no need to cast to BigDecimal before calling toString It looks like REAL, DECIMAL, and NUMERIC used to be treated as doubles in Java, and are now treated as BigDecimal. Is this correct? If so, what happens to existing data (e.g., serialized into SequenceFiles) that contains data of this type? Is this an incompatible change? It also looks as though you are changing how nulls are handled. Is this backwards compatible with existing VerticaRecord deployments? line 575: unnecessary cast to BigDecimal.
        Hide
        Omer Trajman added a comment -

        Test failure was in gridmix. Any suggestions? Should I just resubmit?

        Show
        Omer Trajman added a comment - Test failure was in gridmix. Any suggestions? Should I just resubmit?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12425767/MAPREDUCE-1097.patch
        against trunk revision 887135.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/293/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/293/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/293/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/293/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12425767/MAPREDUCE-1097.patch against trunk revision 887135. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/293/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/293/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/293/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/293/console This message is automatically generated.
        Hide
        Omer Trajman added a comment -

        wrong target

        Show
        Omer Trajman added a comment - wrong target
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12425767/MAPREDUCE-1097.patch
        against trunk revision 882790.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/259/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/259/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/259/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/259/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12425767/MAPREDUCE-1097.patch against trunk revision 882790. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/259/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/259/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/259/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/259/console This message is automatically generated.

          People

          • Assignee:
            Omer Trajman
            Reporter:
            Omer Trajman
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development