Hive
  1. Hive
  2. HIVE-4516

Fix concurrency bug in serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      A patch for concurrent use of TimestampWritable which occurs in a multithreaded scenario (as found in AmpLab Shark). A static SimpleDateFormat (not ThreadSafe) is used by TimestampWritable in CTAS DDL statements where it manifests as data corruption when used in a concurrent environment.

      1. HIVE-4516.D10929.1.patch
        2 kB
        Phabricator
      2. TimestampWritable.java.patch
        2 kB
        Jon Hartlaub

        Issue Links

          Activity

          Hide
          Ashutosh Chauhan added a comment -

          This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          Show
          Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #226 (See https://builds.apache.org/job/Hive-trunk-hadoop2/226/)
          HIVE-4516 : Fix concurrency bug in serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java (Jon Hartlaub and Navis via Ashutosh Chauhan) (Revision 1489673)

          Result = ABORTED
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489673
          Files :

          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #226 (See https://builds.apache.org/job/Hive-trunk-hadoop2/226/ ) HIVE-4516 : Fix concurrency bug in serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java (Jon Hartlaub and Navis via Ashutosh Chauhan) (Revision 1489673) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489673 Files : /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #2129 (See https://builds.apache.org/job/Hive-trunk-h0.21/2129/)
          HIVE-4516 : Fix concurrency bug in serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java (Jon Hartlaub and Navis via Ashutosh Chauhan) (Revision 1489673)

          Result = FAILURE
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489673
          Files :

          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #2129 (See https://builds.apache.org/job/Hive-trunk-h0.21/2129/ ) HIVE-4516 : Fix concurrency bug in serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java (Jon Hartlaub and Navis via Ashutosh Chauhan) (Revision 1489673) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489673 Files : /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Jon and Navis!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Jon and Navis!
          Hide
          Brock Noland added a comment -

          I hit this bug and this patch resolved the issue.

          Show
          Brock Noland added a comment - I hit this bug and this patch resolved the issue.
          Hide
          Phabricator added a comment -

          ashutoshc has accepted the revision "HIVE-4516 [jira] Fix concurrency bug in serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java".

          +1

          REVISION DETAIL
          https://reviews.facebook.net/D10929

          BRANCH
          HIVE-4516

          ARCANIST PROJECT
          hive

          To: JIRA, ashutoshc, navis

          Show
          Phabricator added a comment - ashutoshc has accepted the revision " HIVE-4516 [jira] Fix concurrency bug in serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java". +1 REVISION DETAIL https://reviews.facebook.net/D10929 BRANCH HIVE-4516 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis
          Hide
          Phabricator added a comment -

          navis requested code review of "HIVE-4516 [jira] Fix concurrency bug in serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java".

          Reviewers: JIRA

          HIVE-4516 Fix concurrency bug in serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java

          A patch for concurrent use of TimestampWritable which occurs in a multithreaded scenario (as found in AmpLab Shark). A static SimpleDateFormat (not ThreadSafe) is used by TimestampWritable in CTAS DDL statements where it manifests as data corruption when used in a concurrent environment.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D10929

          AFFECTED FILES
          serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java

          MANAGE HERALD RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/26121/

          To: JIRA, navis

          Show
          Phabricator added a comment - navis requested code review of " HIVE-4516 [jira] Fix concurrency bug in serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java". Reviewers: JIRA HIVE-4516 Fix concurrency bug in serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java A patch for concurrent use of TimestampWritable which occurs in a multithreaded scenario (as found in AmpLab Shark). A static SimpleDateFormat (not ThreadSafe) is used by TimestampWritable in CTAS DDL statements where it manifests as data corruption when used in a concurrent environment. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D10929 AFFECTED FILES serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/26121/ To: JIRA, navis
          Hide
          Navis added a comment -

          passed all tests. But HIVE-4220 has some minor optimization which removes unnecessary date.toString() call. It would be good to be merged with that.

          Show
          Navis added a comment - passed all tests. But HIVE-4220 has some minor optimization which removes unnecessary date.toString() call. It would be good to be merged with that.
          Hide
          Navis added a comment -

          +1, running test

          Show
          Navis added a comment - +1, running test
          Hide
          Jon Hartlaub added a comment -

          I agree - HIVE-4220 may be a manifestation, although I think this ThreadLocal based patch or one using commons.lang FastDateFormat (which is thread-safe) would be preferable over the patch submitted in HIVE-4220.

          Show
          Jon Hartlaub added a comment - I agree - HIVE-4220 may be a manifestation, although I think this ThreadLocal based patch or one using commons.lang FastDateFormat (which is thread-safe) would be preferable over the patch submitted in HIVE-4220 .
          Hide
          Teddy Choi added a comment -

          It seems like HIVE-4220.

          Show
          Teddy Choi added a comment - It seems like HIVE-4220 .

            People

            • Assignee:
              Jon Hartlaub
              Reporter:
              Jon Hartlaub
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development