[HCATALOG-160] HBaseDirectOutputStorageDriver outputVersion isn't consitent within the same MR job - ASF JIRA

Francis Liu created issue - 10/Nov/11 22:56

Francis Liu made changes - 11/Nov/11 18:20

Field	Original Value	New Value
Attachment		HCATALOG-160.patch [ 12503395 ]

Francis Liu made changes - 11/Nov/11 18:21

Status

Open [ 1 ]

Patch Available [ 10002 ]

Hide

Permalink

jiraposter@reviews.apache.org added a comment - 11/Nov/11 18:25

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2809/
-----------------------------------------------------------

Review request for hcatalog, Sushanth Sowmyan, Vandana Ayyalasomayajula, and David Capwell.

Summary
-------

HBaseDirectOutputStorageDriver missed serializing the updated OutputJobInfo, fixed that.

This addresses bug hcatalog-160.
https://issues.apache.org/jira/browse/hcatalog-160

Diffs

storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java 65dfccb
storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseBulkOutputStorageDriver.java c25e56d
storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseDirectOutputStorageDriver.java d612584

Diff: https://reviews.apache.org/r/2809/diff

Testing
-------

Updated unit tests to such a scenario and it passes now.

Thanks,

Francis

Show

jiraposter@reviews.apache.org added a comment - 11/Nov/11 18:25 ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2809/ ----------------------------------------------------------- Review request for hcatalog, Sushanth Sowmyan, Vandana Ayyalasomayajula, and David Capwell. Summary ------- HBaseDirectOutputStorageDriver missed serializing the updated OutputJobInfo, fixed that. This addresses bug hcatalog-160. https://issues.apache.org/jira/browse/hcatalog-160 Diffs storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java 65dfccb storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseBulkOutputStorageDriver.java c25e56d storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseDirectOutputStorageDriver.java d612584 Diff: https://reviews.apache.org/r/2809/diff Testing ------- Updated unit tests to such a scenario and it passes now. Thanks, Francis

Hide

Permalink

Francis Liu added a comment - 16/Nov/11 19:58

svn friendly patch.

Show

Francis Liu added a comment - 16/Nov/11 19:58 svn friendly patch.

Francis Liu made changes - 16/Nov/11 19:58

Attachment

HCATALOG-160_svn.patch [ 12503937 ]

Hide

Permalink

jiraposter@reviews.apache.org added a comment - 23/Nov/11 00:16

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2809/#review3449
-----------------------------------------------------------

storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java
<https://reviews.apache.org/r/2809/#comment7705>

Shouldn't this be done within super.initialize() Once the outputJobInfo is updated with revision String in HBaseBaseOutputStorageDriver, it needs to be overwritten in configuration there only.

Ashutosh

On 2011-11-11 18:24:59, Francis Liu wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/2809/

-----------------------------------------------------------

(Updated 2011-11-11 18:24:59)

Review request for hcatalog, Sushanth Sowmyan, Vandana Ayyalasomayajula, and David Capwell.

Summary

-------

HBaseDirectOutputStorageDriver missed serializing the updated OutputJobInfo, fixed that.

This addresses bug hcatalog-160.

https://issues.apache.org/jira/browse/hcatalog-160

Diffs

-----

storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java 65dfccb

storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseBulkOutputStorageDriver.java c25e56d

storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseDirectOutputStorageDriver.java d612584

Diff: https://reviews.apache.org/r/2809/diff

Testing

-------

Updated unit tests to such a scenario and it passes now.

Thanks,

Francis

Show

jiraposter@reviews.apache.org added a comment - 23/Nov/11 00:16 ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2809/#review3449 ----------------------------------------------------------- storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java < https://reviews.apache.org/r/2809/#comment7705 > Shouldn't this be done within super.initialize() Once the outputJobInfo is updated with revision String in HBaseBaseOutputStorageDriver, it needs to be overwritten in configuration there only. Ashutosh On 2011-11-11 18:24:59, Francis Liu wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2809/ ----------------------------------------------------------- (Updated 2011-11-11 18:24:59) Review request for hcatalog, Sushanth Sowmyan, Vandana Ayyalasomayajula, and David Capwell. Summary ------- HBaseDirectOutputStorageDriver missed serializing the updated OutputJobInfo, fixed that. This addresses bug hcatalog-160. https://issues.apache.org/jira/browse/hcatalog-160 Diffs ----- storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java 65dfccb storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseBulkOutputStorageDriver.java c25e56d storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseDirectOutputStorageDriver.java d612584 Diff: https://reviews.apache.org/r/2809/diff Testing ------- Updated unit tests to such a scenario and it passes now. Thanks, Francis

Hide

Permalink

jiraposter@reviews.apache.org added a comment - 23/Nov/11 00:24

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2809/#review3450
-----------------------------------------------------------

storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java
<https://reviews.apache.org/r/2809/#comment7706>

I left it to the subclass to do the serialization since the subclass may need to add more information to outputJobInfo before it gets serialized. Such as in HBaseBulkOutputStorageDriver where the intermediate location has to be stored in OutputJobInfo.

Francis

On 2011-11-11 18:24:59, Francis Liu wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/2809/

-----------------------------------------------------------

(Updated 2011-11-11 18:24:59)

Review request for hcatalog, Sushanth Sowmyan, Vandana Ayyalasomayajula, and David Capwell.

Summary

-------

HBaseDirectOutputStorageDriver missed serializing the updated OutputJobInfo, fixed that.

This addresses bug hcatalog-160.

https://issues.apache.org/jira/browse/hcatalog-160

Diffs

-----

storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java 65dfccb

storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseBulkOutputStorageDriver.java c25e56d

storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseDirectOutputStorageDriver.java d612584

Diff: https://reviews.apache.org/r/2809/diff

Testing

-------

Updated unit tests to such a scenario and it passes now.

Thanks,

Francis

Show

jiraposter@reviews.apache.org added a comment - 23/Nov/11 00:24 ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2809/#review3450 ----------------------------------------------------------- storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java < https://reviews.apache.org/r/2809/#comment7706 > I left it to the subclass to do the serialization since the subclass may need to add more information to outputJobInfo before it gets serialized. Such as in HBaseBulkOutputStorageDriver where the intermediate location has to be stored in OutputJobInfo. Francis On 2011-11-11 18:24:59, Francis Liu wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2809/ ----------------------------------------------------------- (Updated 2011-11-11 18:24:59) Review request for hcatalog, Sushanth Sowmyan, Vandana Ayyalasomayajula, and David Capwell. Summary ------- HBaseDirectOutputStorageDriver missed serializing the updated OutputJobInfo, fixed that. This addresses bug hcatalog-160. https://issues.apache.org/jira/browse/hcatalog-160 Diffs ----- storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java 65dfccb storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseBulkOutputStorageDriver.java c25e56d storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseDirectOutputStorageDriver.java d612584 Diff: https://reviews.apache.org/r/2809/diff Testing ------- Updated unit tests to such a scenario and it passes now. Thanks, Francis

Hide

Permalink

jiraposter@reviews.apache.org added a comment - 23/Nov/11 00:30

On 2011-11-23 00:23:02, Francis Liu wrote:

> storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java, line 41

> <https://reviews.apache.org/r/2809/diff/1/?file=57620#file57620line41>

>

> I left it to the subclass to do the serialization since the subclass may need to add more information to outputJobInfo before it gets serialized. Such as in HBaseBulkOutputStorageDriver where the intermediate location has to be stored in OutputJobInfo.

I see. In that case, it will be better to add a comment in HBaseBaseOutputStorageDriver::initialize() that anyone extending this class must necessarily override initialize() and then overwrite the outputJobInfo in it. Also, patch doesn't apply cleanly, can you refresh it?

Ashutosh

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2809/#review3450
-----------------------------------------------------------

On 2011-11-11 18:24:59, Francis Liu wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/2809/

-----------------------------------------------------------

(Updated 2011-11-11 18:24:59)

Review request for hcatalog, Sushanth Sowmyan, Vandana Ayyalasomayajula, and David Capwell.

Summary

-------

HBaseDirectOutputStorageDriver missed serializing the updated OutputJobInfo, fixed that.

This addresses bug hcatalog-160.

https://issues.apache.org/jira/browse/hcatalog-160

Diffs

-----

storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java 65dfccb

storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseBulkOutputStorageDriver.java c25e56d

storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseDirectOutputStorageDriver.java d612584

Diff: https://reviews.apache.org/r/2809/diff

Testing

-------

Updated unit tests to such a scenario and it passes now.

Thanks,

Francis

Show

jiraposter@reviews.apache.org added a comment - 23/Nov/11 00:30 On 2011-11-23 00:23:02, Francis Liu wrote: > storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java, line 41 > < https://reviews.apache.org/r/2809/diff/1/?file=57620#file57620line41 > > > I left it to the subclass to do the serialization since the subclass may need to add more information to outputJobInfo before it gets serialized. Such as in HBaseBulkOutputStorageDriver where the intermediate location has to be stored in OutputJobInfo. I see. In that case, it will be better to add a comment in HBaseBaseOutputStorageDriver::initialize() that anyone extending this class must necessarily override initialize() and then overwrite the outputJobInfo in it. Also, patch doesn't apply cleanly, can you refresh it? Ashutosh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2809/#review3450 ----------------------------------------------------------- On 2011-11-11 18:24:59, Francis Liu wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2809/ ----------------------------------------------------------- (Updated 2011-11-11 18:24:59) Review request for hcatalog, Sushanth Sowmyan, Vandana Ayyalasomayajula, and David Capwell. Summary ------- HBaseDirectOutputStorageDriver missed serializing the updated OutputJobInfo, fixed that. This addresses bug hcatalog-160. https://issues.apache.org/jira/browse/hcatalog-160 Diffs ----- storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java 65dfccb storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseBulkOutputStorageDriver.java c25e56d storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseDirectOutputStorageDriver.java d612584 Diff: https://reviews.apache.org/r/2809/diff Testing ------- Updated unit tests to such a scenario and it passes now. Thanks, Francis

Francis Liu made changes - 23/Nov/11 07:03

Attachment

HCATALOG-160_rev2.patch [ 12504854 ]

Hide

Permalink

jiraposter@reviews.apache.org added a comment - 23/Nov/11 07:03

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2809/
-----------------------------------------------------------

(Updated 2011-11-23 07:03:08.841754)

Review request for hcatalog, Sushanth Sowmyan, Vandana Ayyalasomayajula, and David Capwell.

Changes
-------

updated patch with changes in trunk

Summary
-------

HBaseDirectOutputStorageDriver missed serializing the updated OutputJobInfo, fixed that.

This addresses bug hcatalog-160.
https://issues.apache.org/jira/browse/hcatalog-160

Diffs (updated)

storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseBaseOutputStorageDriver.java 596f942
storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java 65dfccb
storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseDirectOutputStorageDriver.java 1c5c60d

Diff: https://reviews.apache.org/r/2809/diff

Testing
-------

Updated unit tests to such a scenario and it passes now.

Thanks,

Francis

Show

jiraposter@reviews.apache.org added a comment - 23/Nov/11 07:03 ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2809/ ----------------------------------------------------------- (Updated 2011-11-23 07:03:08.841754) Review request for hcatalog, Sushanth Sowmyan, Vandana Ayyalasomayajula, and David Capwell. Changes ------- updated patch with changes in trunk Summary ------- HBaseDirectOutputStorageDriver missed serializing the updated OutputJobInfo, fixed that. This addresses bug hcatalog-160. https://issues.apache.org/jira/browse/hcatalog-160 Diffs (updated) storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseBaseOutputStorageDriver.java 596f942 storage-drivers/hbase/src/java/org/apache/hcatalog/hbase/HBaseDirectOutputStorageDriver.java 65dfccb storage-drivers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseDirectOutputStorageDriver.java 1c5c60d Diff: https://reviews.apache.org/r/2809/diff Testing ------- Updated unit tests to such a scenario and it passes now. Thanks, Francis

Hide

Permalink

Ashutosh Chauhan added a comment - 23/Nov/11 18:24

Patch committed to trunk. Thanks, Francis!

Show

Ashutosh Chauhan added a comment - 23/Nov/11 18:24 Patch committed to trunk. Thanks, Francis!

Ashutosh Chauhan made changes - 23/Nov/11 18:24

Status	Patch Available [ 10002 ]	Resolved [ 5 ]
Resolution		Fixed [ 1 ]

Hide

Permalink

Alan Gates added a comment - 17/May/12 02:20

Issue closed with 0.4 release.

Show

Alan Gates added a comment - 17/May/12 02:20 Issue closed with 0.4 release.

Alan Gates made changes - 17/May/12 02:20

Status

Resolved [ 5 ]

Closed [ 6 ]

HBaseDirectOutputStorageDriver outputVersion isn't consitent within the same MR job

Details

Description

Attachments

Activity

People

Dates

Development

Agile