Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-2667

Rewrite ColumnFamilyOutputFormat to use thrift (not arvo)

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      From Jonathan Ellis in http://article.gmane.org/gmane.comp.db.cassandra.user/16478

      > Using "native" Thrift mutations makes the most sense to me now, which
      > would keep it similar in structure but avoid the avroToThrift copy.

      The itch here is that i'd like to use CounterColumn in ColumnFamilyOutputFormat

        Activity

        Hide
        michaelsembwever mck added a comment -

        initial attempt...

        (i'm getting problems with:
        java.io.IOException: InvalidRequestException(why:Column timestamp is required)

        despite setting always setting column.timestamp),

        Show
        michaelsembwever mck added a comment - initial attempt... (i'm getting problems with: java.io.IOException: InvalidRequestException(why:Column timestamp is required) despite setting always setting column.timestamp),
        Hide
        michaelsembwever mck added a comment -

        Fixed in my application (thrift classes can't have direct field access , xxxIsSet need to be true).

        The patch works, as simple as it is.
        And client code only needs to change packages in import statements (no class names change afaik).

        Show
        michaelsembwever mck added a comment - Fixed in my application (thrift classes can't have direct field access , xxxIsSet need to be true). The patch works, as simple as it is. And client code only needs to change packages in import statements (no class names change afaik).
        Hide
        michaelsembwever mck added a comment -

        i hope not too late for a 0.8.x release...?

        Show
        michaelsembwever mck added a comment - i hope not too late for a 0.8.x release...?
        Hide
        jbellis Jonathan Ellis added a comment -

        Why doesn't this break AvroOutputReader?

        Show
        jbellis Jonathan Ellis added a comment - Why doesn't this break AvroOutputReader?
        Hide
        jbellis Jonathan Ellis added a comment -

        Looking at it, I think it does break AOR, but the compiler doesn't know it because there's nothing telling it that the class provided by OutputReader, should be the same expected by OutputFormat/RecordWriter.

        Given that Avro is almost certainly the wrong interface to provide to Streaming (see CASSANDRA-1497), I'd be okay with removing AOR for now and starting over there.

        Show
        jbellis Jonathan Ellis added a comment - Looking at it, I think it does break AOR, but the compiler doesn't know it because there's nothing telling it that the class provided by OutputReader, should be the same expected by OutputFormat/RecordWriter. Given that Avro is almost certainly the wrong interface to provide to Streaming (see CASSANDRA-1497 ), I'd be okay with removing AOR for now and starting over there.
        Hide
        michaelsembwever mck added a comment -

        Same patch but with ..hadoop.streaming package completely removed.

        Show
        michaelsembwever mck added a comment - Same patch but with ..hadoop.streaming package completely removed.
        Hide
        tjake T Jake Luciani added a comment -

        +1

        Show
        tjake T Jake Luciani added a comment - +1
        Hide
        jbellis Jonathan Ellis added a comment -

        committed, with attempt to fix contrib/pig too

        Show
        jbellis Jonathan Ellis added a comment - committed, with attempt to fix contrib/pig too

          People

          • Assignee:
            michaelsembwever mck
            Reporter:
            michaelsembwever mck
            Reviewer:
            T Jake Luciani
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development