This is not a bug - CQL3 tables are intentionally not included in thrift describe_keyspace(s) (CASSANDRA-4377).

See CASSANDRA-4421

It means that CQL3 column families are not accessible through thrift and for me it's an issue (I do not agree with your Resolution label). That's why Pig 0.11 cannot use it. Is there a way to solve it ?
I can help you if necessary

It should be fixed after CASSANDRA-4421

I will work on it once I am done with other assignments soon.

thanks Alex !

To fix it, we need modify CassandraStorage to get CF meta data from system table instead of thrift describe_keyspace because of the CQL3 table doesn't show up in thrift describe_keyspace call.

Patch is attached. It's on top of the 4421 patch

pull @ https://github.com/alexliu68/cassandra/pull/3

Use CassandraStorage for any cql3 tables, you will have composite columns in "columns" bag

Use CqlStorage for any cql3 table.

cassandra://[username:password@]<keyspace>/<columnfamily>[?[page_size=<size>]
[&columns=<col1,col2>][&output_query=<prepared_statement>]
[&where_clause=<clause>][&split_size=<size>][&partitioner=<partitioner>]]

where
page_size is the number of cql3 rows per page (the default is 1000, it's optional)

columns is the column names for the cql3 select query, it's optional

where_clause is the user defined where clause on the indexed column, it's optional

split_size is the number of C* rows per split which can be used to tune the number of mappers

output_query is the prepared query for inserting data to cql3 table (replace the = by @ and ? by #,
because Pig can't take = and ? as url parameter values)

Output row are in the following format

(((name, value), (name, value)), (value ... value), (value...value))

where the name and value tuples are key name and value pairs.

The input schema: ((name, value), (name, value), (name, value)) where keys are in the front.

I attach 5234-1.2-patch.txt to patch 1.2 branch. It update the last patch with the latest 4421 changes.

The patch resolves the following issues.

1. allow access to cql3 type table through CassandraStorage.

2. create new CqlStorage to easy access cql3 tables.

Do you think I can test them now ?

yes, I have done some testing.

Okay, I'll give it a try
thanks

My Pig script doesn't work anymore. I suppose you changed the input format ?
I get :
Projected field [filtre] does not exist in schema: key:chararray,columns:bag{:tuple(name:tuple(),value:byte array)}
when column family is created without COMPACT STORAGE

Something weird is that I get no values when I should get some :
cqlsh:k1> SELECT * FROM cf1 WHERE ISE='XXXX';

ise | filtre | value_1
-------------------
XXXX | 1 | 81056

2013-06-17 13:13:01,372 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(XXXX,

) <-- value not printable for filtre ?

Can you post your schema for the table and pig script, so I can test it.

http://pastebin.com/Fub9t6j9 <-- my column family
http://pastebin.com/HwKxsC4f <-- my pig script

Can you just dump(data) to check whether you have all the data? Then do the following filter scripts.

You definitely broke something as I get different results when dumping data for cql2 and cql3 tables (same structure).
CQL2 :

* format is 
x: {key: chararray,value_1: (name: chararray,value: int),filtre: (name: chararray,value: int),columns: {(name: chararray,value: bytearray)}}
* rows are
(XXXX,(value_1,18584),(filtre,0),{})
(YYYY,(value_1,49926),(filtre,2),{})

CQL3 :

* format is 
x: {key: chararray,columns: {(name: (),value: bytearray)}}
* rows are
(XXXX,{((),),((filtre),),((value_1),??)})
(YYYY,{((),),((filtre),),((value_1),??)})

CQL2 has different structure from CQL3, CQL2 is more to the legacy thrift type CF(Check the developer blog from datastax). You should follow the cql3 format to change your pig script. or use CqlStorage for easy data mapping for cql3 type table.

Use the following script to find the structure of cql3 table, then change your pig script

   rows = LOAD 'cassandra://MyKeyspace/MyColumnFamily' USING CassandraStorage();
   dump(rows);

Great, CqlStorage() helps a lot to get same input.

Thanks

5234-1-1.2-patch.txt is attached to allow CassandraStorage to pass partitioner as parameter.

5234-2-1.2branch.txt is attached to use "cql://" instead of "cassandra://" for CqlStorage

Hmm, I'm seeing errors when running the examples/pig/test tests that don't use cql3.

I will fix it today.

The failed test is where there is filter for COUNT(columns)

-- filter to fully visible rows (no uuid columns) and dump
visible = FILTER rows BY COUNT(columns) == 0;
dump visible;

It turns out to wide row schema issue which could be an error from previous version.

5234-3-1.2branch.txt is attached to fixd failing to run examples/pig/test

@Alex I pulled the branch CASSANDRA-5234 and tried building it. But the build fails when compiling.

It works for me.

try the commands

git checkout cassandra-1.2
patch -p1 < 5234-3-1.2branch.txt 

@Alex: I did check out cassandra-1.2 and tried to apply the patch using "patch -p1 < 5234-3-1.2branch.txt" But it didnt work it threw and exception < is reserveed for future use.
So, I downloaded the patch 5234-3-1.2branch.txt and did a git apply and then committed the unstaged changes. Now when I browsed the file location cassandra "/ src / java / org / apache / cassandra / hadoop / pig /" the patch is not applied. Sorry for my naive questions, I am very new to git and cassandra.

The patch is applied. But do I have to worry about the

5234-3-1.2branch.txt:22: trailing whitespace.
partitioner = FBUtilities.newPartitioner(client.describe_partitioner());
5234-3-1.2branch.txt:735: trailing whitespace.

5234-3-1.2branch.txt:925: trailing whitespace.

5234-3-1.2branch.txt:1297: trailing whitespace.
}
5234-3-1.2branch.txt:1321: trailing whitespace.

Committed, and created CASSANDRA-5709 for example follow-up. Thanks!

Deepak Rosario Pancras it's in the 1.2 branch now, just do a git pull.

Thanks Brandon

5234-3-trunk.txt patch for trunk branch is attached.

@Brandon. Thanks for your patience. I did a git pull on 1.2 and ran the build. It fails again.

I recommend sending an email to user list with the details, as the 1.2 branch is obviously compiling for everyone and JIRA isn't a support forum.

Hello Alex!
I have got error when trying to send where_clause in cqlStorage URL as follows:
rows = LOAD 'cql://keyspace1/test?page_size=1&columns=title,age&split_size=4&where_clause=age=41' USING CqlStorage();
and haven't find any way to escape the equal operator.
It will be better to send the where_clause with URL encoding, for example where_clause=age%3D41.
However for a quick fix i have slightly modify the method getQueryMap as follows:
/** decompose the query to store the parameters in a map*/
public static Map<String, String> getQueryMap(String query) throws Exception
{
String[] params = query.split("&");
Map<String, String> map = new HashMap<String, String>();
for (String param : params)

return map;
}
and now i can send the query with URL-Encoding character.

Thx, I will update the patch for it.

add the patch as a temporary fix

I was trying it locally and I'm not sure whether it fully works. My stacktrace and cassandra table schema is here: http://pastebin.com/uPUAs9T2 (i tried using old cassandra:// ... - it worked for other table) and output when I used cql://.. is here: http://pastebin.com/b0bKd7G3 . I have worked on this with Pig in local mode. It might be important or not: this table contained counters.

I was working on cassandra-1.2 with latest commit 27efded38d855b24f41e5332ffb29cd13d98f8da

Just adding that I'm getting the same result as Konrad Kurdej. I'm using DSE 3.1, but I think this is the same bug. I was able to isolate the problem specifically to counter fields. Here's a simple set up and test case:

cqlsh> create keyspace pigtest with REPLICATION =

;
cqlsh> use pigtest;
cqlsh:pigtest> create table foo ( key_1 text primary key, value_1 bigint );
cqlsh:pigtest> create table foo2 ( key_1 text primary key, value_1 counter );
cqlsh:pigtest> update foo set value_1 = 1 where key_1 = 'foo';
cqlsh:pigtest> update foo2 set value_1 = value_1 + 2 where key_1 = 'foo2';
cqlsh:pigtest> select * from foo;

key_1 | value_1
------+--------
foo | 1

cqlsh:pigtest> select * from foo2;

key_1 | value_1
------+--------
foo2 | 2

Now, the following grunt commands:

counts = LOAD 'cassandra://pigtest/foo' USING CassandraStorage();
dump counts;

Will work, but:

counts = LOAD 'cassandra://pigtest/foo2' USING CassandraStorage();
dump counts;

Will fail with the same stack trace that Konrad mentioned.

This patch against the head appears to have fixed the problem for me. I applied it to DSE 3.1 and it also worked for me.

Basic deal is to use LongType for validation too.

Use CqlStorage for your use case. CassandraStorage has some drawbacks.

CassandraStorage is legacy for any none-CQL3 tables. Use CqlStorage for CQL3 tables.

Cassandra 1.2.8 was released (after regression in 1.2.7), but this last patch for counters appears to be left. Anyone can confirm this?

I downloaded the .deb from http://people.apache.org/~eevans/ but when in grunt i dump a table with counter column, get the error:

java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:537)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:410)
at org.apache.cassandra.db.context.CounterContext.total(CounterContext.java:477)
at org.apache.cassandra.db.marshal.AbstractCommutativeType.compose(AbstractCommutativeType.java:34)
at org.apache.cassandra.db.marshal.AbstractCommutativeType.compose(AbstractCommutativeType.java:25)
at org.apache.cassandra.hadoop.pig.AbstractCassandraStorage.columnToTuple(AbstractCassandraStorage.java:137)
at org.apache.cassandra.hadoop.pig.CqlStorage.getNext(CqlStorage.java:110)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:211)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

My table:
cqlsh:test> desc table votes_count_period3;

CREATE TABLE votes_count_period3 (
period text,
poll timeuuid,
votes counter,
PRIMARY KEY (period, poll)
) WITH
bloom_filter_fp_chance=0.010000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction=

AND
compression=

;

I confirmed this issue. I downloaded the src, applied the patch and after build the cassandra jar, pig works with counter. This will be patched in 1.2.9?

+1

Alex Liu The patch is to the base class shared by CqlStorage and CassandraStorage. They were both broken and they are now both fixed.

The last patch for counters to AbstractCassandraStorage.java has not been applied either to cassandra-1.2 or to the trunk. Consequently, the problem still exists in 1.2.9, as I could verify myself today. I found another recent bug report on SO for the same problem: http://stackoverflow.com/questions/18553230/error-with-cassandra-pig-cql-counter-column.

Should I open a new bug report for the counters bug even though we have a working patch or will you reopen the current issue?