Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Duplicate
    • Fix Version/s: None
    • Component/s: Core
    • Labels:
      None
    • Environment:

      Ubuntu AWS Cassandra 2.0.1 SINGLE NODE on EBS RAID storage
      OSX Cassandra 1.2.8 on SSD storage

      Description

      Possible Resolution:

      What seems to be key is to run a nodetool compact (possibly a nodetool flush) after schema drops / schema creations / schema truncates and invalidate the caches. This seems to align the data for new inserts/updates. From my reproduction tests, I have been unable to generate the database corruption if nodetool flush, nodetool compact, nodetool keycacheinvalidate (we have turned off rowcache due to other bugs). Then, even after running a more stressful test with 10x the inserts and five separate concurrent update threads the corruption did not appear.

      So I believe this is a tentative "fix" to this issue... in general, after any manipulation to the schema, you should run nodetool compact and keycacheinvalidate. I have not tested if a general compact on all keyspaces and tables versus a more specific compact on the affected keyspace and/or keyspace tables is all that is necessary (compact can be a very expensive operation).

      ------------------------------------------------------------------

      Problem Encountered:

      We are encountering inconsistent results from CQL3 queries with column keys using IN clause in WHERE. This has been reproduced in cqlsh and the jdbc driver. Specifically, we are doing queries to pull a subset of column keys for a specific row key.

      We detect this corruption by selecting all the column keys for a row, and then trying different subsets of column keys in WHERE <columnkey> IN (<column key subset list>). We see some of these column key subset queries not return all the column keys, even though the select-all-column-keys query finds them.

      It seems to appear when there is a large amount of raw insertion work (non-updates / new ingested data) combined with simultaneous updates to existing data. EDIT: this also seems to only happen with mass insert+updates after schema changes / drops / table creation / table truncation. See the Possible Resolution section above.

      ------------------------------------------------------------------

      Details:

      Rowkey is e_entid
      Column key is p_prop

      This returns roughly 21 rows for 21 column keys that match p_prop.

      cqlsh> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB';

      These three queries each return one row for the requested single column key in the IN clause:

      SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB' AND p_prop in ('urn:bby:pcm:job:ingest:content:complete:count');
      SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB' AND p_prop in ('urn:bby:pcm:job:ingest:content:all:count');
      SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB' AND p_prop in ('urn:bby:pcm:job:ingest:content:fail:count');

      This query returns ONLY ONE ROW (one column key), not three as I would expect from the three-column-key IN clause:

      cqlsh> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB' AND p_prop in ('urn:bby:pcm:job:ingest:content:complete:count','urn:bby:pcm:job:ingest:content:all:count','urn:bby:pcm:job:ingest:content:fail:count');

      This query does return two rows however for the requested two column keys:

      cqlsh> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB' AND p_prop in ( 'urn:bby:pcm:job:ingest:content:all:count','urn:bby:pcm:job:ingest:content:fail:count');

      cqlsh> describe table internal_submission.entity_job;

      CREATE TABLE entity_job (
      e_entid text,
      p_prop text,
      describes text,
      dndcondition text,
      e_entlinks text,
      e_entname text,
      e_enttype text,
      ingeststatus text,
      ingeststatusdetail text,
      p_flags text,
      p_propid text,
      p_proplinks text,
      p_storage text,
      p_subents text,
      p_val text,
      p_vallang text,
      p_vallinks text,
      p_valtype text,
      p_valunit text,
      p_vars text,
      partnerid text,
      referenceid text,
      size int,
      sourceip text,
      submitdate bigint,
      submitevent text,
      userid text,
      version text,
      PRIMARY KEY (e_entid, p_prop)
      ) WITH
      bloom_filter_fp_chance=0.010000 AND
      caching='KEYS_ONLY' AND
      comment='' AND
      dclocal_read_repair_chance=0.000000 AND
      gc_grace_seconds=864000 AND
      index_interval=128 AND
      read_repair_chance=0.100000 AND
      replicate_on_write='true' AND
      populate_io_cache_on_flush='false' AND
      default_time_to_live=0 AND
      speculative_retry='NONE' AND
      memtable_flush_period_in_ms=0 AND
      compaction=

      {'class': 'SizeTieredCompactionStrategy'}

      AND
      compression=

      {'sstable_compression': 'LZ4Compressor'}

      ;

      CREATE INDEX internal_submission__JobDescribesIDX ON entity_job (describes);

      CREATE INDEX internal_submission__JobDNDConditionIDX ON entity_job (dndcondition);

      CREATE INDEX internal_submission__JobIngestStatusIDX ON entity_job (ingeststatus);

      CREATE INDEX internal_submission__JobIngestStatusDetailIDX ON entity_job (ingeststatusdetail);

      CREATE INDEX internal_submission__JobReferenceIDIDX ON entity_job (referenceid);

      CREATE INDEX internal_submission__JobUserIDX ON entity_job (userid);

      CREATE INDEX internal_submission__JobVersionIDX ON entity_job (version);

      -------------------------------

      My suspicion is that the three-column-key IN Clause is translated (improperly or not) to a two-column key range with the assumption that the third column key is present in that range, but it isn't...

      -----------------------------------

      We have tried: nodetool cache invalidations, start/stop of cassandra. Those did NOT fix the problem. A table dump (COPY TO) and then reload (COPY FROM) does fix the rows, but then more corruption creeps in.

      We are using the cassandra-jdbc driver, but I don't see anything wrong with the issued statements inside the cassandra source code when I step through the code.

      With additional writes, it may be possible that some rows get fixed. Compaction or other jobs may repair this, but on the timescale of hours done debugging, the failures are consistent.

        Issue Links

          Activity

          Hide
          Constance Eustace added a comment -

          This is inconsistent/varies across rows in the table.

          Show
          Constance Eustace added a comment - This is inconsistent/varies across rows in the table.
          Hide
          Constance Eustace added a comment -

          This may be related to us possibly inserting a null or empty column key. We'll try to investigate the data at a lower level

          Show
          Constance Eustace added a comment - This may be related to us possibly inserting a null or empty column key. We'll try to investigate the data at a lower level
          Hide
          Constance Eustace added a comment -

          Or possibly we crammed in multiple column key updates at once..

          Show
          Constance Eustace added a comment - Or possibly we crammed in multiple column key updates at once..
          Hide
          Constance Eustace added a comment -

          We did an export of sorts of the data from the 2.0.1 cassandra down to 1.2.9, imported/loaded. The reproduction test did not find anything.

          Currently we are testing a set of about 1000 rows by:

          For each row key:

          • get the list of column keys by SELECT columnkey FROM table WHERE rowkey = ?
          • generate several 2- 3- and 4- column key WHERE IN clauses of random column keys in random order
          • execute SELECT columnkey, columnkey_othercol FROM table WHERE rowkey = ? and columnkey IN [$columnkeyset]
          • check for missing columns

          On our "corrupted" database the program locates frequent failures.
          Export of that data using SELECT * FROM table ==> JSON and then JSON ==> same schema in 1.2.9 then re-run found no problems.

          Show
          Constance Eustace added a comment - We did an export of sorts of the data from the 2.0.1 cassandra down to 1.2.9, imported/loaded. The reproduction test did not find anything. Currently we are testing a set of about 1000 rows by: For each row key: get the list of column keys by SELECT columnkey FROM table WHERE rowkey = ? generate several 2- 3- and 4- column key WHERE IN clauses of random column keys in random order execute SELECT columnkey, columnkey_othercol FROM table WHERE rowkey = ? and columnkey IN [$columnkeyset] check for missing columns On our "corrupted" database the program locates frequent failures. Export of that data using SELECT * FROM table ==> JSON and then JSON ==> same schema in 1.2.9 then re-run found no problems.
          Hide
          Constance Eustace added a comment -

          THe queries in the "corrupted" instance at least consistently report the wrong query results...

          Show
          Constance Eustace added a comment - THe queries in the "corrupted" instance at least consistently report the wrong query results...
          Hide
          Constance Eustace added a comment -

          SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB' AND p_prop in ('urn:bby:pcm:job:ingest:content:complete:count','urn:bby:pcm:job:ingest:content:all:count','urn:bby:pcm:job:ingest:content:fail:count');

          This query would only return one p_prop row. Any time any of the p_props with values ending in :count were in the in clause, only one would be returned.

          Once a COPY TO / TRUNCATE / COPY FROM reload was done, all worked well.

          There are no counter columns on this table, but there are ones on another table in the keyspace.

          Show
          Constance Eustace added a comment - SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB' AND p_prop in ('urn:bby:pcm:job:ingest:content:complete:count','urn:bby:pcm:job:ingest:content:all:count','urn:bby:pcm:job:ingest:content:fail:count'); This query would only return one p_prop row. Any time any of the p_props with values ending in :count were in the in clause, only one would be returned. Once a COPY TO / TRUNCATE / COPY FROM reload was done, all worked well. There are no counter columns on this table, but there are ones on another table in the keyspace.
          Hide
          Constance Eustace added a comment -

          The error is reappearing.

          Nodetool repairs/compactions/etc triggering didn't do anything.

          Perhaps this is yet another caching error, this time in the key caches. We're going to try some restarts of the node or schemes to clean the caches to see if this is an actual internal data representation problem.

          Show
          Constance Eustace added a comment - The error is reappearing. Nodetool repairs/compactions/etc triggering didn't do anything. Perhaps this is yet another caching error, this time in the key caches. We're going to try some restarts of the node or schemes to clean the caches to see if this is an actual internal data representation problem.
          Hide
          Constance Eustace added a comment -

          nodetool invalidatekeycache did nothing, and we have already turned off rowcaching for another bug...

          A wider audit of 15,000 rows in other tables that have quite complex column key names produced no errors.

          It must be the :count at the end of these column keys...

          Show
          Constance Eustace added a comment - nodetool invalidatekeycache did nothing, and we have already turned off rowcaching for another bug... A wider audit of 15,000 rows in other tables that have quite complex column key names produced no errors. It must be the :count at the end of these column keys...
          Hide
          Constance Eustace added a comment -

          We are currently simply selecting all the column keys rather than doing a subset of the columns using a where <columnkey> in (<columnkeylist>).

          The row we are doing this on isn't horribly worse than just getting the desired ones, but it's still not ideal.

          Show
          Constance Eustace added a comment - We are currently simply selecting all the column keys rather than doing a subset of the columns using a where <columnkey> in (<columnkeylist>). The row we are doing this on isn't horribly worse than just getting the desired ones, but it's still not ideal.
          Hide
          Constance Eustace added a comment -

          We did see this error at some point, it may be a trigger exception where it goes from working to not working...

          https://issues.apache.org/jira/browse/CASSANDRA-4687

          Show
          Constance Eustace added a comment - We did see this error at some point, it may be a trigger exception where it goes from working to not working... https://issues.apache.org/jira/browse/CASSANDRA-4687
          Hide
          Constance Eustace added a comment - - edited

          It is now occurring in prod for other columns. There appears to be some hash key impacts here...

          [10/10/13 12:13:19 AM] AGaal: wasn't a regression; DB corruption on a product entity
          [10/10/13 12:13:20 AM] AGaal: cqlsh> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Product WHERE e_entid = '0d5acd67-3131-11e3-85d7-126aad0075d4-PROD' AND p_prop IN ('_CPSYS_type','_CPSYS_name','urn:bby:pcm:job:id');

          (0 rows)

          cqlsh> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Product WHERE e_entid = '0d5acd67-3131-11e3-85d7-126aad0075d4-PROD' AND p_prop IN ('_CPSYS_type','_CPSYS_name');

          e_entid | e_entname | e_enttype | p_prop | p_flags | p_propid | p_val | p_vallinks | p_vars
          --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
          0d5acd67-3131-11e3-85d7-126aad0075d4-PROD | 1 ft Cat5e Non Booted UTP Unshielded Network Patch Cable :::: 757120254621|NEW | null | __CPSYS_name | null | null | null | null | null
          0d5acd67-3131-11e3-85d7-126aad0075d4-PROD | null | urn:bby:pcm:product | __CPSYS_type | null | null | null | null | null

          (2 rows)
          [10/10/13 12:20:12 AM] AGaal: cqlsh> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Product WHERE e_entid = '0d5acd67-3131-11e3-85d7-126aad0075d4-PROD' AND p_prop IN ('urn:bby:pcm:job:id');

          (0 rows)
          [10/10/13 12:20:12 AM] AGaal: note that in this example 'urn:bby:cpm:job:id' does not exist yet, so asking just for that correctly returns 0 rows:
          [10/10/13 12:20:42 AM] AGaal: but if it's included in a where in() with 2 other properties that do exist, then 0 rows are also returned there too, which is bad
          [10/10/13 12:26:50 AM] AGaal: another work-around for where in() might be to do a select for each desired property, so in this case there would have been 3 selects; could this be faster / more efficient than selecting all?
          [10/10/13 12:37:51 AM] AGaal: we might be able to get some traction here by enabling some cassandra logging and playing with the query
          [10/10/13 12:38:29 AM] AGaal: like if the property name is shortened to 'urn:bby:pcm:', it returns the expected 2 rows
          [10/10/13 12:39:08 AM] AGaal: but if it's 'urn:bby:pcm:j' or ''urn:bby:pcm:d ' it finds 0
          [10/10/13 12:42:41 AM] AGaal: and if the last letter after urn:bby:cpm: is an 'a' or 'b' or 'c' it also returns 2…. and it's consistent with this. So it's finding some sort of match in certain strings… like via a hash or startsWith or something

          Show
          Constance Eustace added a comment - - edited It is now occurring in prod for other columns. There appears to be some hash key impacts here... [10/10/13 12:13:19 AM] AGaal: wasn't a regression; DB corruption on a product entity [10/10/13 12:13:20 AM] AGaal: cqlsh> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Product WHERE e_entid = '0d5acd67-3131-11e3-85d7-126aad0075d4-PROD' AND p_prop IN ('_ CPSYS_type',' _CPSYS_name','urn:bby:pcm:job:id'); (0 rows) cqlsh> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Product WHERE e_entid = '0d5acd67-3131-11e3-85d7-126aad0075d4-PROD' AND p_prop IN ('_ CPSYS_type',' _CPSYS_name'); e_entid | e_entname | e_enttype | p_prop | p_flags | p_propid | p_val | p_vallinks | p_vars ------------------------------------------ ------------------------------------------------------------------------------ ------------------- ------------ ------- -------- ----- ---------- ------- 0d5acd67-3131-11e3-85d7-126aad0075d4-PROD | 1 ft Cat5e Non Booted UTP Unshielded Network Patch Cable :::: 757120254621|NEW | null | __CPSYS_name | null | null | null | null | null 0d5acd67-3131-11e3-85d7-126aad0075d4-PROD | null | urn:bby:pcm:product | __CPSYS_type | null | null | null | null | null (2 rows) [10/10/13 12:20:12 AM] AGaal: cqlsh> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Product WHERE e_entid = '0d5acd67-3131-11e3-85d7-126aad0075d4-PROD' AND p_prop IN ('urn:bby:pcm:job:id'); (0 rows) [10/10/13 12:20:12 AM] AGaal: note that in this example 'urn:bby:cpm:job:id' does not exist yet, so asking just for that correctly returns 0 rows: [10/10/13 12:20:42 AM] AGaal: but if it's included in a where in() with 2 other properties that do exist, then 0 rows are also returned there too, which is bad [10/10/13 12:26:50 AM] AGaal: another work-around for where in() might be to do a select for each desired property, so in this case there would have been 3 selects; could this be faster / more efficient than selecting all? [10/10/13 12:37:51 AM] AGaal: we might be able to get some traction here by enabling some cassandra logging and playing with the query [10/10/13 12:38:29 AM] AGaal: like if the property name is shortened to 'urn:bby:pcm:', it returns the expected 2 rows [10/10/13 12:39:08 AM] AGaal: but if it's 'urn:bby:pcm:j' or ''urn:bby:pcm:d ' it finds 0 [10/10/13 12:42:41 AM] AGaal: and if the last letter after urn:bby:cpm: is an 'a' or 'b' or 'c' it also returns 2…. and it's consistent with this. So it's finding some sort of match in certain strings… like via a hash or startsWith or something
          Hide
          Constance Eustace added a comment - - edited

          I believe that compound/composite keys are encoded by CQL3 with a colon (':') as a separator. Our column keys have a lot of colons in them per RDF style of naming...

          I believe this is the issue...

          Show
          Constance Eustace added a comment - - edited I believe that compound/composite keys are encoded by CQL3 with a colon (':') as a separator. Our column keys have a lot of colons in them per RDF style of naming... I believe this is the issue...
          Hide
          Constance Eustace added a comment -

          This is clearly happening in columns that we update. Possibly the results are consistent for a while, but it may repair itself.

          I just want to note that this is being done against a single-node cassandra server.

          Show
          Constance Eustace added a comment - This is clearly happening in columns that we update. Possibly the results are consistent for a while, but it may repair itself. I just want to note that this is being done against a single-node cassandra server.
          Hide
          Constance Eustace added a comment - - edited

          We have row caching off, but key caching is on.

          So in general I believe:

          • There are niggling character escaping bugs in the the key caching is not working correctly with keys that have ':' characters in them.
          • Those bugs seem to lead to cache inconsistency because of compaction or another underlying housekeeping process changes the key locations, or the additional colons in the composite keys mess up the code that would keep the keycache valid.

          So queries that request partial keysets from a row try to utilize the cache (SELECT columnlist FROM table WHERE rowkey = ? and columnkey in (?,?,?,?) become inconsistent when one of the column key's composite columns is updated...

          But queries that request all keys will just read the data and bypass the cache I'd guess, so they work.

          It appears that eventually the inconsistent results correct themselves, so again, perhaps some cache coherency process makes it consistent...wait for it, wait for it... eventually.

          I'd guess that compaction is the most likely target.

          We now have time to examine source code, would anyone from DSE or mainline committers recommend anyplace to look given what is described? Anyone? Bueller? Bueller?

          Show
          Constance Eustace added a comment - - edited We have row caching off, but key caching is on. So in general I believe: There are niggling character escaping bugs in the the key caching is not working correctly with keys that have ':' characters in them. Those bugs seem to lead to cache inconsistency because of compaction or another underlying housekeeping process changes the key locations, or the additional colons in the composite keys mess up the code that would keep the keycache valid. So queries that request partial keysets from a row try to utilize the cache (SELECT columnlist FROM table WHERE rowkey = ? and columnkey in (?,?,?,?) become inconsistent when one of the column key's composite columns is updated... But queries that request all keys will just read the data and bypass the cache I'd guess, so they work. It appears that eventually the inconsistent results correct themselves, so again, perhaps some cache coherency process makes it consistent...wait for it, wait for it... eventually. I'd guess that compaction is the most likely target. We now have time to examine source code, would anyone from DSE or mainline committers recommend anyplace to look given what is described? Anyone? Bueller? Bueller?
          Hide
          Constance Eustace added a comment - - edited

          This has been reproduced in Cassandra 1.2.8 locally on SSD single-node.

          • we have data that is both separated by @s and separated by :s. We have failures for both separators. We have failures that don't SEEM to involve the word "count" in the columns, but most seem to involve queries with the string "count" in them.

          ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
          ID: 924d6742-31fd-11e3-97f7-001c42000009-CJOB !Unable to render embedded object: File (NOMATCH) not found.!:

          [__CPSYS_name, subPropA:filttest:sdf, urn:bby:pcm:ingest:status, urn@bby@pcm@job@ingest@content@complete@count] to
          [__CPSYS_name, subPropA:filttest:sdf, urn:bby:pcm:ingest:status]

          ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
          select * from wyfr_submission.entity_job where e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' ;

          RESULT:

          e_entid | p_prop | describes | dndcondition | e_entlinks | e_entname | e_enttype | ingeststatus | ingeststatusdetail | p_flags | p_propid | p_proplinks | p_storage | p_subents | p_val | p_vallang | p_vallinks | p_valtype | p_valunit | p_vars | partnerid | referenceid | size | sourceip | submitdate | submitevent | userid | version
          924d6742-31fd-11e3-97f7-001c42000009-CJOB | __CPSYS_links | null | null | [{"content":null,"href":"http://cupcake/wyfr/entities/id/9222fbe1-31fd-11e3-97f7-001c42000009-CJOB","rel":"parent-entity","type":"application/vnd.BUB.content.entity","hreflang":null,"title":"Up to Parent Entity","length":null,"otherAttributes":{}},{"content":null,"href":"http://cupcake/wyfr/entities/id/9222fbe1-31fd-11e3-97f7-001c42000009-CJOB/properties/type/DYNApropAA%3Asdjlk%3Asdfklj%3Afiltertest","rel":"up","type":"application/vnd.BUB.content.property","hreflang":null,"title":"Up to Parent Entity Property","length":null,"otherAttributes":{}}] | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null
          924d6742-31fd-11e3-97f7-001c42000009-CJOB | __CPSYS_name | null | null | null | SUBENT-100 | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null
          924d6742-31fd-11e3-97f7-001c42000009-CJOB | __CPSYS_type | null | null | null | null | urn:bby:pcm:job | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null
          924d6742-31fd-11e3-97f7-001c42000009-CJOB | subPropA:filttest:sdf | null | null | null | null | null | null | null | DVM | 924d6743-31fd-11e3-97f7-001c42000009 | [] | null | [] | ["subE","subI","subH"] | en_us | [] | text | m | [

          {"type":"text","lang":"en_us","uom":"m","datas":["subE","subI","subH"],"links":[]}

          ] | null | null | null | null | null | null | null | null
          924d6742-31fd-11e3-97f7-001c42000009-CJOB | urn:bby:pcm:ingest:status | null | null | null | null | null | subC | null | S00 | 924d6744-31fd-11e3-97f7-001c42000009 | [] | null | [] | null | null | [] | null | null | [

          {"type":"text","lang":"en_us","uom":"m","datas":["subC"],"links":[]}

          ] | null | null | null | null | null | null | null | null
          924d6742-31fd-11e3-97f7-001c42000009-CJOB | urn:bby:pcm:job:sourceparty:reference:id | null | null | null | null | null | null | null | S00 | 924d6745-31fd-11e3-97f7-001c42000009 | [] | null | [] | null | null | [] | null | null | [

          {"type":"text","lang":"en_us","uom":"m","datas":["cdc8911c-7092-4c71-a7a6-935331ff2c5c"],"links":[]}

          ] | null | cdc8911c-7092-4c71-a7a6-935331ff2c5c | null | null | null | null | null | null
          924d6742-31fd-11e3-97f7-001c42000009-CJOB | urn@bby@pcm@job@ingest@content@complete@count | null | null | null | null | null | null | null | null | null | [] | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null

          TRACE:
          DEBUG [Thrift:1] 2013-10-15 14:45:26,841 CassandraServer.java (line 1699) execute_cql3_query
          TRACE [Thrift:1] 2013-10-15 14:45:26,842 QueryProcessor.java (line 100) Process org.apache.cassandra.cql3.statements.SelectStatement@7aacaaae @CL.ONE
          TRACE [Thrift:1] 2013-10-15 14:45:26,843 ReadCallback.java (line 67) Blockfor is 1; setting up requests to /127.0.0.1
          TRACE [ReadStage:3] 2013-10-15 14:45:26,851 SSTableReader.java (line 756) Adding cache entry for KeyCacheKey(/Users/a999166/aaa-projects/apache-cassandra-1.2.8/z-serverfiles/data/wyfr_submission/entity_job/wyfr_submission-entity_job-ic-2, 39323464363734322d333166642d313165332d393766372d3030316334323030303030392d434a4f42) -> org.apache.cassandra.db.RowIndexEntry@4017415d
          TRACE [ReadStage:3] 2013-10-15 14:45:26,853 SliceQueryFilter.java (line 161) collecting 0 of 10000: __CPSYS_links::false:0@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,853 SliceQueryFilter.java (line 161) collecting 1 of 10000: __CPSYS_links:e_entlinks:false:566@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,853 SliceQueryFilter.java (line 161) collecting 1 of 10000: __CPSYS_name::false:0@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,853 SliceQueryFilter.java (line 161) collecting 2 of 10000: __CPSYS_name:e_entname:false:10@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 2 of 10000: __CPSYS_type::false:0@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 3 of 10000: __CPSYS_type:e_enttype:false:15@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 3 of 10000: subPropA\:filttest\:sdf::false:0@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf_flags:false:3@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf_propid:false:36@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf_proplinks:false:2@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf_subents:false:2@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf_val:false:22@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf_vallang:false:5@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf_vallinks:false:2@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf_valtype:false:4@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf_valunit:false:1@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf_vars:false:84@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 4 of 10000: urn\:bby\:pcm\:ingest\:status::false:0@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status:ingeststatus:false:4@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status_flags:false:3@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status_propid:false:36@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status_proplinks:false:2@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status_subents:false:2@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status_vallinks:false:2@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status_vars:false:70@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id::false:0@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id_flags:false:3@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id_propid:false:36@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id_proplinks:false:2@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,857 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id_subents:false:2@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,857 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id_vallinks:false:2@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,857 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id_vars:false:102@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,857 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id:referenceid:false:36@1381445090517000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,857 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn@bby@pcm@job@ingest@content@complete@count::false:0@1381860857189000
          TRACE [ReadStage:3] 2013-10-15 14:45:26,857 SliceQueryFilter.java (line 161) collecting 7 of 10000: urn@bby@pcm@job@ingest@content@complete@count_proplinks:false:2@1381860857189000

          ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
          SELECT * FROM wyfr_submission.entity_job WHERE e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','urn:bby:pcm:ingest:status','subPropA:filttest:sdf','urn@bby@pcm@job@ingest@content@complete@count')

          RESULT:
          e_entid | p_prop | describes | dndcondition | e_entlinks | e_entname | e_enttype | ingeststatus | ingeststatusdetail | p_flags | p_propid | p_proplinks | p_storage | p_subents | p_val | p_vallang | p_vallinks | p_valtype | p_valunit | p_vars | partnerid | referenceid | size | sourceip | submitdate | submitevent | userid | version
          924d6742-31fd-11e3-97f7-001c42000009-CJOB | __CPSYS_name | null | null | null | SUBENT-100 | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null
          924d6742-31fd-11e3-97f7-001c42000009-CJOB | subPropA:filttest:sdf | null | null | null | null | null | null | null | DVM | 924d6743-31fd-11e3-97f7-001c42000009 | [] | null | [] | ["subE","subI","subH"] | en_us | [] | text | m | [

          {"type":"text","lang":"en_us","uom":"m","datas":["subE","subI","subH"],"links":[]}

          ] | null | null | null | null | null | null | null | null
          924d6742-31fd-11e3-97f7-001c42000009-CJOB | urn:bby:pcm:ingest:status | null | null | null | null | null | subC | null | S00 | 924d6744-31fd-11e3-97f7-001c42000009 | [] | null | [] | null | null | [] | null | null | [

          {"type":"text","lang":"en_us","uom":"m","datas":["subC"],"links":[]}

          ] | null | null | null | null | null | null | null | null

          TRACE:
          DEBUG [Thrift:1] 2013-10-15 14:48:39,115 CassandraServer.java (line 1699) execute_cql3_query
          TRACE [Thrift:1] 2013-10-15 14:48:39,116 QueryProcessor.java (line 100) Process org.apache.cassandra.cql3.statements.SelectStatement@58053ef7 @CL.ONE
          TRACE [Thrift:1] 2013-10-15 14:48:39,116 ReadCallback.java (line 67) Blockfor is 1; setting up requests to /127.0.0.1
          TRACE [ReadStage:33] 2013-10-15 14:48:39,117 SliceQueryFilter.java (line 161) collecting 0 of 10000: __CPSYS_name::false:0@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,117 SliceQueryFilter.java (line 161) collecting 1 of 10000: __CPSYS_name:e_entname:false:10@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,117 SliceQueryFilter.java (line 161) collecting 1 of 10000: subPropA\:filttest\:sdf::false:0@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf_flags:false:3@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf_propid:false:36@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf_proplinks:false:2@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf_subents:false:2@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf_val:false:22@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf_vallang:false:5@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf_vallinks:false:2@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf_valtype:false:4@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf_valunit:false:1@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf_vars:false:84@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 2 of 10000: urn\:bby\:pcm\:ingest\:status:ingeststatus:false:4@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 3 of 10000: urn\:bby\:pcm\:ingest\:status_flags:false:3@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 3 of 10000: urn\:bby\:pcm\:ingest\:status_propid:false:36@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 3 of 10000: urn\:bby\:pcm\:ingest\:status_proplinks:false:2@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 3 of 10000: urn\:bby\:pcm\:ingest\:status_subents:false:2@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 3 of 10000: urn\:bby\:pcm\:ingest\:status_vallinks:false:2@1381445090517000
          TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 3 of 10000: urn\:bby\:pcm\:ingest\:status_vars:false:70@1381445090517000
          DEBUG [Thrift:1] 2013-10-15 14:48:39,120 Tracing.java (line 167) request complete

          Show
          Constance Eustace added a comment - - edited This has been reproduced in Cassandra 1.2.8 locally on SSD single-node. we have data that is both separated by @s and separated by :s. We have failures for both separators. We have failures that don't SEEM to involve the word "count" in the columns, but most seem to involve queries with the string "count" in them. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ID: 924d6742-31fd-11e3-97f7-001c42000009-CJOB ! Unable to render embedded object: File (NOMATCH) not found. !: [__CPSYS_name, subPropA:filttest:sdf, urn:bby:pcm:ingest:status, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_name, subPropA:filttest:sdf, urn:bby:pcm:ingest:status] ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ select * from wyfr_submission.entity_job where e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' ; RESULT: e_entid | p_prop | describes | dndcondition | e_entlinks | e_entname | e_enttype | ingeststatus | ingeststatusdetail | p_flags | p_propid | p_proplinks | p_storage | p_subents | p_val | p_vallang | p_vallinks | p_valtype | p_valunit | p_vars | partnerid | referenceid | size | sourceip | submitdate | submitevent | userid | version 924d6742-31fd-11e3-97f7-001c42000009-CJOB | __CPSYS_links | null | null | [{"content":null,"href":"http://cupcake/wyfr/entities/id/9222fbe1-31fd-11e3-97f7-001c42000009-CJOB","rel":"parent-entity","type":"application/vnd.BUB.content.entity","hreflang":null,"title":"Up to Parent Entity","length":null,"otherAttributes":{}},{"content":null,"href":"http://cupcake/wyfr/entities/id/9222fbe1-31fd-11e3-97f7-001c42000009-CJOB/properties/type/DYNApropAA%3Asdjlk%3Asdfklj%3Afiltertest","rel":"up","type":"application/vnd.BUB.content.property","hreflang":null,"title":"Up to Parent Entity Property","length":null,"otherAttributes":{}}] | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null 924d6742-31fd-11e3-97f7-001c42000009-CJOB | __CPSYS_name | null | null | null | SUBENT-100 | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null 924d6742-31fd-11e3-97f7-001c42000009-CJOB | __CPSYS_type | null | null | null | null | urn:bby:pcm:job | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null 924d6742-31fd-11e3-97f7-001c42000009-CJOB | subPropA:filttest:sdf | null | null | null | null | null | null | null | DVM | 924d6743-31fd-11e3-97f7-001c42000009 | [] | null | [] | ["subE","subI","subH"] | en_us | [] | text | m | [ {"type":"text","lang":"en_us","uom":"m","datas":["subE","subI","subH"],"links":[]} ] | null | null | null | null | null | null | null | null 924d6742-31fd-11e3-97f7-001c42000009-CJOB | urn:bby:pcm:ingest:status | null | null | null | null | null | subC | null | S00 | 924d6744-31fd-11e3-97f7-001c42000009 | [] | null | [] | null | null | [] | null | null | [ {"type":"text","lang":"en_us","uom":"m","datas":["subC"],"links":[]} ] | null | null | null | null | null | null | null | null 924d6742-31fd-11e3-97f7-001c42000009-CJOB | urn:bby:pcm:job:sourceparty:reference:id | null | null | null | null | null | null | null | S00 | 924d6745-31fd-11e3-97f7-001c42000009 | [] | null | [] | null | null | [] | null | null | [ {"type":"text","lang":"en_us","uom":"m","datas":["cdc8911c-7092-4c71-a7a6-935331ff2c5c"],"links":[]} ] | null | cdc8911c-7092-4c71-a7a6-935331ff2c5c | null | null | null | null | null | null 924d6742-31fd-11e3-97f7-001c42000009-CJOB | urn@bby@pcm@job@ingest@content@complete@count | null | null | null | null | null | null | null | null | null | [] | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null TRACE: DEBUG [Thrift:1] 2013-10-15 14:45:26,841 CassandraServer.java (line 1699) execute_cql3_query TRACE [Thrift:1] 2013-10-15 14:45:26,842 QueryProcessor.java (line 100) Process org.apache.cassandra.cql3.statements.SelectStatement@7aacaaae @CL.ONE TRACE [Thrift:1] 2013-10-15 14:45:26,843 ReadCallback.java (line 67) Blockfor is 1; setting up requests to /127.0.0.1 TRACE [ReadStage:3] 2013-10-15 14:45:26,851 SSTableReader.java (line 756) Adding cache entry for KeyCacheKey(/Users/a999166/aaa-projects/apache-cassandra-1.2.8/z-serverfiles/data/wyfr_submission/entity_job/wyfr_submission-entity_job-ic-2, 39323464363734322d333166642d313165332d393766372d3030316334323030303030392d434a4f42) -> org.apache.cassandra.db.RowIndexEntry@4017415d TRACE [ReadStage:3] 2013-10-15 14:45:26,853 SliceQueryFilter.java (line 161) collecting 0 of 10000: __CPSYS_links::false:0@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,853 SliceQueryFilter.java (line 161) collecting 1 of 10000: __CPSYS_links:e_entlinks:false:566@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,853 SliceQueryFilter.java (line 161) collecting 1 of 10000: __CPSYS_name::false:0@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,853 SliceQueryFilter.java (line 161) collecting 2 of 10000: __CPSYS_name:e_entname:false:10@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 2 of 10000: __CPSYS_type::false:0@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 3 of 10000: __CPSYS_type:e_enttype:false:15@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 3 of 10000: subPropA\:filttest\:sdf::false:0@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf _flags:false:3@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf _propid:false:36@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf _proplinks:false:2@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf _subents:false:2@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf _val:false:22@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,854 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf _vallang:false:5@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf _vallinks:false:2@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf _valtype:false:4@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf _valunit:false:1@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 4 of 10000: subPropA\:filttest\:sdf _vars:false:84@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 4 of 10000: urn\:bby\:pcm\:ingest\:status::false:0@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status:ingeststatus:false:4@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status _flags:false:3@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,855 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status _propid:false:36@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status _proplinks:false:2@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status _subents:false:2@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status _vallinks:false:2@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:ingest\:status _vars:false:70@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 5 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id::false:0@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id _flags:false:3@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id _propid:false:36@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,856 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id _proplinks:false:2@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,857 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id _subents:false:2@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,857 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id _vallinks:false:2@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,857 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id _vars:false:102@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,857 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn\:bby\:pcm\:job\:sourceparty\:reference\:id:referenceid:false:36@1381445090517000 TRACE [ReadStage:3] 2013-10-15 14:45:26,857 SliceQueryFilter.java (line 161) collecting 6 of 10000: urn@bby@pcm@job@ingest@content@complete@count::false:0@1381860857189000 TRACE [ReadStage:3] 2013-10-15 14:45:26,857 SliceQueryFilter.java (line 161) collecting 7 of 10000: urn@bby@pcm@job@ingest@content@complete@count _proplinks:false:2@1381860857189000 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ SELECT * FROM wyfr_submission.entity_job WHERE e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','urn:bby:pcm:ingest:status','subPropA:filttest:sdf','urn@bby@pcm@job@ingest@content@complete@count') RESULT: e_entid | p_prop | describes | dndcondition | e_entlinks | e_entname | e_enttype | ingeststatus | ingeststatusdetail | p_flags | p_propid | p_proplinks | p_storage | p_subents | p_val | p_vallang | p_vallinks | p_valtype | p_valunit | p_vars | partnerid | referenceid | size | sourceip | submitdate | submitevent | userid | version 924d6742-31fd-11e3-97f7-001c42000009-CJOB | __CPSYS_name | null | null | null | SUBENT-100 | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null | null 924d6742-31fd-11e3-97f7-001c42000009-CJOB | subPropA:filttest:sdf | null | null | null | null | null | null | null | DVM | 924d6743-31fd-11e3-97f7-001c42000009 | [] | null | [] | ["subE","subI","subH"] | en_us | [] | text | m | [ {"type":"text","lang":"en_us","uom":"m","datas":["subE","subI","subH"],"links":[]} ] | null | null | null | null | null | null | null | null 924d6742-31fd-11e3-97f7-001c42000009-CJOB | urn:bby:pcm:ingest:status | null | null | null | null | null | subC | null | S00 | 924d6744-31fd-11e3-97f7-001c42000009 | [] | null | [] | null | null | [] | null | null | [ {"type":"text","lang":"en_us","uom":"m","datas":["subC"],"links":[]} ] | null | null | null | null | null | null | null | null TRACE: DEBUG [Thrift:1] 2013-10-15 14:48:39,115 CassandraServer.java (line 1699) execute_cql3_query TRACE [Thrift:1] 2013-10-15 14:48:39,116 QueryProcessor.java (line 100) Process org.apache.cassandra.cql3.statements.SelectStatement@58053ef7 @CL.ONE TRACE [Thrift:1] 2013-10-15 14:48:39,116 ReadCallback.java (line 67) Blockfor is 1; setting up requests to /127.0.0.1 TRACE [ReadStage:33] 2013-10-15 14:48:39,117 SliceQueryFilter.java (line 161) collecting 0 of 10000: __CPSYS_name::false:0@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,117 SliceQueryFilter.java (line 161) collecting 1 of 10000: __CPSYS_name:e_entname:false:10@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,117 SliceQueryFilter.java (line 161) collecting 1 of 10000: subPropA\:filttest\:sdf::false:0@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf _flags:false:3@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf _propid:false:36@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf _proplinks:false:2@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf _subents:false:2@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf _val:false:22@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf _vallang:false:5@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf _vallinks:false:2@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf _valtype:false:4@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,118 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf _valunit:false:1@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 2 of 10000: subPropA\:filttest\:sdf _vars:false:84@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 2 of 10000: urn\:bby\:pcm\:ingest\:status:ingeststatus:false:4@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 3 of 10000: urn\:bby\:pcm\:ingest\:status _flags:false:3@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 3 of 10000: urn\:bby\:pcm\:ingest\:status _propid:false:36@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 3 of 10000: urn\:bby\:pcm\:ingest\:status _proplinks:false:2@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 3 of 10000: urn\:bby\:pcm\:ingest\:status _subents:false:2@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 3 of 10000: urn\:bby\:pcm\:ingest\:status _vallinks:false:2@1381445090517000 TRACE [ReadStage:33] 2013-10-15 14:48:39,119 SliceQueryFilter.java (line 161) collecting 3 of 10000: urn\:bby\:pcm\:ingest\:status _vars:false:70@1381445090517000 DEBUG [Thrift:1] 2013-10-15 14:48:39,120 Tracing.java (line 167) request complete
          Hide
          Constance Eustace added a comment -

          Some debugging of

          SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','urn:bby:pcm:ingest:status','subPropA:filttest:sdf','urn@bby@pcm@job@ingest@content@complete@count')

          SelectRawStatement[name=wayfair_submission.entity_job, selectClause=[], whereClause=[e_entid EQ '924d6742-31fd-11e3-97f7-001c42000009-CJOB', p_prop IN ['__CPSYS_name', 'urn:bby:pcm:ingest:status', 'subPropA:filttest:sdf', 'urn@bby@pcm@job@ingest@content@complete@count']], isCount=false,
          --> is the CF metadata properly returned for all the columns in the parsed statement?
          --> is a Range/Slice of Columns (SelectStatement:217)
          --> Range must have a high and a low, are the right ones being selected
          [SliceFromReadCommand(table='wayfair_submission', key='39323464363734322d333166642d313165332d393766372d3030316334323030303030392d434a4f42', column_parent='QueryPath(columnFamilyName='entity_job', superColumnName='null', columnName='null')', filter='SliceQueryFilter [reversed=false, slices=[[000c5f5f43505359535f6e616d6500, 000c5f5f43505359535f6e616d6501], [001573756250726f70413a66696c74746573743a73646600, 001573756250726f70413a66696c74746573743a73646601], [001975726e3a6262793a70636d3a696e676573743a73746174757300, 001975726e3a6262793a70636d3a696e676573743a73746174757301], [002d75726e406262794070636d406a6f6240696e6765737440636f6e74656e7440636f6d706c65746540636f756e7400, 002d75726e406262794070636d406a6f6240696e6765737440636f6e74656e7440636f6d706c65746540636f756e7401]], count=10000, toGroup = 1]')]
          SliceFromReadCommand
          (table='wayfair_submission',
          key='39323464363734322d333166642d313165332d393766372d3030316334323030303030392d434a4f42',
          column_parent='QueryPath(columnFamilyName='entity_job', superColumnName='null', columnName='null')',
          filter='
          SliceQueryFilter [
          reversed=false,
          slices=[
          [000c5f5f43505359535f6e616d6500, 000c5f5f43505359535f6e616d6501],
          [001573756250726f70413a66696c74746573743a73646600, 001573756250726f70413a66696c74746573743a73646601],
          [001975726e3a6262793a70636d3a696e676573743a73746174757300, 001975726e3a6262793a70636d3a696e676573743a73746174757301],
          [002d75726e406262794070636d406a6f6240696e6765737440636f6e74656e7440636f6d706c65746540636f756e7400, 002d75726e406262794070636d406a6f6240696e6765737440636f6e74656e7440636f6d706c65746540636f756e7401]
          ],
          count=10000,
          toGroup = 1]'
          )

          Guessing from the length of the four number pairs there, you can see that the four numbers are probably the four column names in terms of length....

          Row(
          key=
          DecoratedKey(8705314879532960628, 39323464363734322d333166642d313165332d393766372d3030316334323030303030392d434a4f42),
          cf=ColumnFamily(
          entity_job [
          __CPSYS_name::false:0@1381445090517000,
          __CPSYS_name:e_entname:false:10@1381445090517000,
          subPropA\:filttest\:sdf::false:0@1381445090517000,
          subPropA\:filttest\:sdf_flags:false:3@1381445090517000,
          subPropA\:filttest\:sdf_propid:false:36@1381445090517000,
          subPropA\:filttest\:sdf_proplinks:false:2@1381445090517000,
          subPropA\:filttest\:sdf_subents:false:2@1381445090517000,
          subPropA\:filttest\:sdf_val:false:22@1381445090517000,
          subPropA\:filttest\:sdf_vallang:false:5@1381445090517000,
          subPropA\:filttest\:sdf_vallinks:false:2@1381445090517000,
          subPropA\:filttest\:sdf_valtype:false:4@1381445090517000,
          subPropA\:filttest\:sdf_valunit:false:1@1381445090517000,
          subPropA\:filttest\:sdf_vars:false:84@1381445090517000,
          urn\:bby\:pcm\:ingest\:status:ingeststatus:false:4@1381445090517000,
          urn\:bby\:pcm\:ingest\:status_flags:false:3@1381445090517000,
          urn\:bby\:pcm\:ingest\:status_propid:false:36@1381445090517000,
          urn\:bby\:pcm\:ingest\:status_proplinks:false:2@1381445090517000,
          urn\:bby\:pcm\:ingest\:status_subents:false:2@1381445090517000,
          urn\:bby\:pcm\:ingest\:status_vallinks:false:2@1381445090517000,
          urn\:bby\:pcm\:ingest\:status_vars:false:70@1381445090517000,
          ]
          )
          )

          ... does that mean the CF metadata doesn't have the columns for the last one? Or is that just the data value?

          Show
          Constance Eustace added a comment - Some debugging of SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','urn:bby:pcm:ingest:status','subPropA:filttest:sdf','urn@bby@pcm@job@ingest@content@complete@count') SelectRawStatement[name=wayfair_submission.entity_job, selectClause=[], whereClause=[e_entid EQ '924d6742-31fd-11e3-97f7-001c42000009-CJOB', p_prop IN ['__CPSYS_name', 'urn:bby:pcm:ingest:status', 'subPropA:filttest:sdf', 'urn@bby@pcm@job@ingest@content@complete@count'] ], isCount=false, --> is the CF metadata properly returned for all the columns in the parsed statement? --> is a Range/Slice of Columns (SelectStatement:217) --> Range must have a high and a low, are the right ones being selected [SliceFromReadCommand(table='wayfair_submission', key='39323464363734322d333166642d313165332d393766372d3030316334323030303030392d434a4f42', column_parent='QueryPath(columnFamilyName='entity_job', superColumnName='null', columnName='null')', filter='SliceQueryFilter [reversed=false, slices=[ [000c5f5f43505359535f6e616d6500, 000c5f5f43505359535f6e616d6501] , [001573756250726f70413a66696c74746573743a73646600, 001573756250726f70413a66696c74746573743a73646601] , [001975726e3a6262793a70636d3a696e676573743a73746174757300, 001975726e3a6262793a70636d3a696e676573743a73746174757301] , [002d75726e406262794070636d406a6f6240696e6765737440636f6e74656e7440636f6d706c65746540636f756e7400, 002d75726e406262794070636d406a6f6240696e6765737440636f6e74656e7440636f6d706c65746540636f756e7401] ], count=10000, toGroup = 1]')] SliceFromReadCommand (table='wayfair_submission', key='39323464363734322d333166642d313165332d393766372d3030316334323030303030392d434a4f42', column_parent='QueryPath(columnFamilyName='entity_job', superColumnName='null', columnName='null')', filter=' SliceQueryFilter [ reversed=false, slices=[ [000c5f5f43505359535f6e616d6500, 000c5f5f43505359535f6e616d6501] , [001573756250726f70413a66696c74746573743a73646600, 001573756250726f70413a66696c74746573743a73646601] , [001975726e3a6262793a70636d3a696e676573743a73746174757300, 001975726e3a6262793a70636d3a696e676573743a73746174757301] , [002d75726e406262794070636d406a6f6240696e6765737440636f6e74656e7440636f6d706c65746540636f756e7400, 002d75726e406262794070636d406a6f6240696e6765737440636f6e74656e7440636f6d706c65746540636f756e7401] ], count=10000, toGroup = 1]' ) Guessing from the length of the four number pairs there, you can see that the four numbers are probably the four column names in terms of length.... Row( key= DecoratedKey(8705314879532960628, 39323464363734322d333166642d313165332d393766372d3030316334323030303030392d434a4f42), cf=ColumnFamily( entity_job [ __CPSYS_name::false:0@1381445090517000, __CPSYS_name:e_entname:false:10@1381445090517000, subPropA\:filttest\:sdf::false:0@1381445090517000, subPropA\:filttest\:sdf _flags:false:3@1381445090517000, subPropA\:filttest\:sdf _propid:false:36@1381445090517000, subPropA\:filttest\:sdf _proplinks:false:2@1381445090517000, subPropA\:filttest\:sdf _subents:false:2@1381445090517000, subPropA\:filttest\:sdf _val:false:22@1381445090517000, subPropA\:filttest\:sdf _vallang:false:5@1381445090517000, subPropA\:filttest\:sdf _vallinks:false:2@1381445090517000, subPropA\:filttest\:sdf _valtype:false:4@1381445090517000, subPropA\:filttest\:sdf _valunit:false:1@1381445090517000, subPropA\:filttest\:sdf _vars:false:84@1381445090517000, urn\:bby\:pcm\:ingest\:status:ingeststatus:false:4@1381445090517000, urn\:bby\:pcm\:ingest\:status _flags:false:3@1381445090517000, urn\:bby\:pcm\:ingest\:status _propid:false:36@1381445090517000, urn\:bby\:pcm\:ingest\:status _proplinks:false:2@1381445090517000, urn\:bby\:pcm\:ingest\:status _subents:false:2@1381445090517000, urn\:bby\:pcm\:ingest\:status _vallinks:false:2@1381445090517000, urn\:bby\:pcm\:ingest\:status _vars:false:70@1381445090517000, ] ) ) ... does that mean the CF metadata doesn't have the columns for the last one? Or is that just the data value?
          Hide
          Constance Eustace added a comment - - edited

          We expect these queries to work since we do a SELECT * on the rowkey, get a list of returned columns, then generate a bunch of queries that select a random set of column keys, and when a column key exists in the select * but does not get returned in a SELECT WHERE IN..

          FAILING QUERIES:

          ID: f924f134-31e0-11e3-b605-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f924f134-31e0-11e3-b605-001c42000009-CJOB' AND p_prop IN ('_CPSYS_type','CPSYS_name','urn@bby@pcm@job@ingest@content@warn@count','_CPSYS_links')
          ID: b529bc94-31e5-11e3-9bd1-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b529bc94-31e5-11e3-9bd1-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','subPropA@filttest@sdf','_CPSYS_name','_CPSYS_type')
          ID: a4a023f3-31ef-11e3-84cd-001c42000009-CJOB NOMATCH: [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@complete@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a4a023f3-31ef-11e3-84cd-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','urn@bby@pcm@ingest@status','__CPSYS_type','urn@bby@pcm@job@sourceparty@reference@id')
          ID: b81e318b-31f2-11e3-bc88-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b81e318b-31f2-11e3-bc88-001c42000009-CJOB' AND p_prop IN ('_CPSYS_links','CPSYS_name','_CPSYS_type','urn@bby@pcm@job@ingest@content@complete@count')
          ID: 58a032e8-31f1-11e3-84cd-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '58a032e8-31f1-11e3-84cd-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','urn@bby@pcm@ingest@data','__CPSYS_links','urn@bby@pcm@job@sourceparty@reference@id')
          ID: 4a3ba684-31e2-11e3-9fbe-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '4a3ba684-31e2-11e3-9fbe-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','subPropA@filttest@sdf','_CPSYS_type','_CPSYS_name')
          ID: 4a456a81-31e2-11e3-9fbe-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@hold@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '4a456a81-31e2-11e3-9fbe-001c42000009-CJOB' AND p_prop IN ('_CPSYS_type','subPropA@filttest@sdf','_CPSYS_links','urn@bby@pcm@job@ingest@content@hold@count')
          ID: b51ff891-31e5-11e3-9bd1-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@complete@count, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@fail@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b51ff891-31e5-11e3-9bd1-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@fail@count','_CPSYS_name','_CPSYS_type','urn@bby@pcm@job@ingest@content@complete@count')
          ID: b96383c2-31e7-11e3-9392-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b96383c2-31e7-11e3-9392-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','_CPSYS_name','urn@bby@pcm@job@ingest@content@complete@count','_CPSYS_type')
          ID: f6748b53-31ed-11e3-ab0d-001c42000009-CJOB NOMATCH: [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@fail@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f6748b53-31ed-11e3-ab0d-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@sourceparty@reference@id','urn@bby@pcm@ingest@status','urn@bby@pcm@job@ingest@content@fail@count','__CPSYS_type')
          ID: b951d081-31e7-11e3-9392-001c42000009-CJOB NOMATCH: [DISABLED__SYSTEM__BYPASS__PROP__SUBSET__, __CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count] to [DISABLED__SYSTEM__BYPASS__PROP__SUBSET__, __CPSYS_links, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b951d081-31e7-11e3-9392-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','DISABLED_SYSTEMBYPASSPROPSUBSET','subPropA@filttest@sdf','_CPSYS_links')
          ID: 6bc17ea0-31ed-11e3-89d8-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6bc17ea0-31ed-11e3-89d8-001c42000009-CJOB' AND p_prop IN ('_CPSYS_type','CPSYS_name','_CPSYS_links','urn@bby@pcm@ingest@data')
          ID: d4241266-31f1-11e3-b75f-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'd4241266-31f1-11e3-b75f-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@warn@count','_CPSYS_name','_CPSYS_type','subPropA@filttest@sdf')
          ID: d7312824-31ed-11e3-961d-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'd7312824-31ed-11e3-961d-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_CPSYS_links','_CPSYS_name','urn@bby@pcm@job@ingest@content@success@count')
          ID: 0164c793-31f4-11e3-a64e-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '0164c793-31f4-11e3-a64e-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','_CPSYS_name','_CPSYS_type','urn@bby@pcm@job@sourceparty@reference@id')
          ID: f923dfc1-31e0-11e3-b605-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data, urn@bby@pcm@job@ingest@content@hold@count] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f923dfc1-31e0-11e3-b605-001c42000009-CJOB' AND p_prop IN ('_CPSYS_type','_CPSYS_name','urn@bby@pcm@ingest@data','urn@bby@pcm@job@ingest@content@hold@count')
          ID: d6e5bdb7-31e1-11e3-9912-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'd6e5bdb7-31e1-11e3-9912-001c42000009-CJOB' AND p_prop IN ('_CPSYS_type','_CPSYS_links','urn@bby@pcm@job@ingest@content@warn@count','subPropA@filttest@sdf')
          ID: c8955af0-31e5-11e3-a033-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'c8955af0-31e5-11e3-a033-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','_CPSYS_name','CPSYS_links','_CPSYS_type')
          ID: 2a370676-31f3-11e3-bf8b-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '2a370676-31f3-11e3-bf8b-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','urn@bby@pcm@job@sourceparty@reference@id','subPropA@filttest@sdf','__CPSYS_links')
          ID: 63e93721-31ef-11e3-b50f-001c42000009-CJOB NOMATCH: [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@fail@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '63e93721-31ef-11e3-b50f-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@sourceparty@reference@id','urn@bby@pcm@job@ingest@content@fail@count','urn@bby@pcm@ingest@status','__CPSYS_type')
          ID: a3aa36c4-31e5-11e3-ad9c-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@hold@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a3aa36c4-31e5-11e3-ad9c-001c42000009-CJOB' AND p_prop IN ('_CPSYS_type','urn@bby@pcm@job@ingest@content@hold@count','CPSYS_links','_CPSYS_name')
          ID: 9b9fb120-31e6-11e3-a6c8-001c42000009-CJOB NOMATCH: [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@job@sourceparty@uid] to [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@sourceparty@uid] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '9b9fb120-31e6-11e3-a6c8-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','urn@bby@pcm@ingest@data','__CPSYS_name','urn@bby@pcm@job@sourceparty@uid')
          ID: f4d19773-31f2-11e3-b478-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@data] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f4d19773-31f2-11e3-b478-001c42000009-CJOB' AND p_prop IN ('_CPSYS_name','subPropA@filttest@sdf','urn@bby@pcm@ingest@data','_CPSYS_type')
          ID: bc6c9cb3-31f5-11e3-a5b8-001c42000009-CJOB NOMATCH: [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'bc6c9cb3-31f5-11e3-a5b8-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','urn@bby@pcm@ingest@status','subPropA@filttest@sdf','__CPSYS_type')
          ID: 043b4025-31f4-11e3-a64e-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@hold@count, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@hold@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '043b4025-31f4-11e3-a64e-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','urn@bby@pcm@job@ingest@content@hold@count','_CPSYS_name','_CPSYS_type')
          ID: b95c30c3-31e7-11e3-9392-001c42000009-CJOB NOMATCH: [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b95c30c3-31e7-11e3-9392-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','subPropA@filttest@sdf','__CPSYS_type','urn@bby@pcm@job@ingest@content@warn@count')
          ID: 4a44f552-31e2-11e3-9fbe-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '4a44f552-31e2-11e3-9fbe-001c42000009-CJOB' AND p_prop IN ('_CPSYS_links','subPropA@filttest@sdf','_CPSYS_name','urn@bby@pcm@job@ingest@content@success@count')
          ID: 9b93a333-31e6-11e3-a6c8-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '9b93a333-31e6-11e3-a6c8-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_CPSYS_links','_CPSYS_name','urn@bby@pcm@job@ingest@content@warn@count')
          ID: 559c1dd4-31f5-11e3-9684-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '559c1dd4-31f5-11e3-9684-001c42000009-CJOB' AND p_prop IN ('_CPSYS_links','urn@bby@pcm@ingest@status','urn@bby@pcm@job@ingest@content@success@count','_CPSYS_name')
          ID: 014e8073-31f4-11e3-a64e-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '014e8073-31f4-11e3-a64e-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','urn@bby@pcm@ingest@status','__CPSYS_links','subPropA@filttest@sdf')
          ID: 8a715897-31e6-11e3-9d0e-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '8a715897-31e6-11e3-9d0e-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','__CPSYS_links','urn@bby@pcm@job@ingest@content@complete@count','subPropA@filttest@sdf')
          ID: b96a1372-31e7-11e3-9392-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b96a1372-31e7-11e3-9392-001c42000009-CJOB' AND p_prop IN ('_CPSYS_name','CPSYS_links','urn@bby@pcm@job@ingest@content@complete@count','_CPSYS_type')
          ID: 6bbc9ca1-31ed-11e3-89d8-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, urn@bby@pcm@job@ingest@content@hold@count, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, urn@bby@pcm@job@ingest@content@success@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6bbc9ca1-31ed-11e3-89d8-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@hold@count','_CPSYS_name','_CPSYS_links','urn@bby@pcm@job@ingest@content@success@count')
          ID: a1731f91-31f2-11e3-bc88-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a1731f91-31f2-11e3-bc88-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@status','__CPSYS_links','subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@success@count')
          ID: b95d4232-31e7-11e3-9392-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b95d4232-31e7-11e3-9392-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@fail@count','subPropA@filttest@sdf','_CPSYS_links','_CPSYS_type')
          ID: a49d16b1-31ef-11e3-84cd-001c42000009-CJOB NOMATCH: [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a49d16b1-31ef-11e3-84cd-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@status','__CPSYS_name','urn@bby@pcm@job@ingest@content@success@count','subPropA@filttest@sdf')
          ID: a4c49be4-31ef-11e3-84cd-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a4c49be4-31ef-11e3-84cd-001c42000009-CJOB' AND p_prop IN ('_CPSYS_name','CPSYS_type','_CPSYS_links','urn@bby@pcm@job@ingest@content@success@count')
          ID: 26d00223-31fd-11e3-82e6-001c42000009-CJOB NOMATCH: [subPropA:filttest:sdf, urn:bby:pcm:ingest:status, urn@bby@pcm@job@ingest@content@complete@count, urn@bby@pcm@job@ingest@content@fail@count] to [subPropA:filttest:sdf, urn:bby:pcm:ingest:status, urn@bby@pcm@job@ingest@content@fail@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '26d00223-31fd-11e3-82e6-001c42000009-CJOB' AND p_prop IN ('subPropA:filttest:sdf','urn@bby@pcm@job@ingest@content@fail@count','urn@bby@pcm@job@ingest@content@complete@count','urn:bby:pcm:ingest:status')
          ID: f67e4f54-31ed-11e3-ab0d-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f67e4f54-31ed-11e3-ab0d-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@complete@count','_CPSYS_name','_CPSYS_links')
          ID: a38879c3-31ec-11e3-b2d5-001c42000009-CJOB NOMATCH: [subPropA@filttest@sdf, urn:bby:pcm:job:sourceparty:reference:id, urn@bby@pcm@job@ingest@content@fail@count, urn@bby@pcm@job@sourceparty@reference@id] to [subPropA@filttest@sdf, urn:bby:pcm:job:sourceparty:reference:id, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a38879c3-31ec-11e3-b2d5-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@sourceparty@reference@id','subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@fail@count','urn:bby:pcm:job:sourceparty:reference:id')
          ID: 97233243-31fd-11e3-97f7-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA:filttest:sdf, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, subPropA:filttest:sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '97233243-31fd-11e3-97f7-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','subPropA:filttest:sdf','_CPSYS_links','_CPSYS_name')
          ID: f9142853-31e0-11e3-b605-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f9142853-31e0-11e3-b605-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@fail@count','_CPSYS_type','_CPSYS_links','subPropA@filttest@sdf')
          ID: 2170bdd3-31e7-11e3-a934-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '2170bdd3-31e7-11e3-a934-001c42000009-CJOB' AND p_prop IN ('_CPSYS_links','urn@bby@pcm@ingest@status','urn@bby@pcm@job@ingest@content@warn@count','_CPSYS_name')
          ID: d0c65120-31e0-11e3-aa5a-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'd0c65120-31e0-11e3-aa5a-001c42000009-CJOB' AND p_prop IN ('_CPSYS_name','subPropA@filttest@sdf','_CPSYS_links','urn@bby@pcm@job@ingest@content@complete@count')
          ID: f4dedde4-31f2-11e3-b478-001c42000009-CJOB NOMATCH: [__CPSYS_name, urn:bby:pcm:ingest:status, urn@bby@pcm@ingest@data, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_name, urn:bby:pcm:ingest:status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f4dedde4-31f2-11e3-b478-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','__CPSYS_name','urn:bby:pcm:ingest:status','urn@bby@pcm@job@sourceparty@reference@id')
          ID: 4c8eebb3-31ea-11e3-9c73-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, bby@submission@SourceIP, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, bby@submission@SourceIP] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '4c8eebb3-31ea-11e3-9c73-001c42000009-CJOB' AND p_prop IN ('_CPSYS_name','bby@submission@SourceIP','_CPSYS_links','urn@bby@pcm@ingest@data')
          ID: b61aa4e2-31e8-11e3-8ea5-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b61aa4e2-31e8-11e3-8ea5-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@warn@count','_CPSYS_type','_CPSYS_name')
          ID: d0c2ceb0-31e0-11e3-aa5a-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'd0c2ceb0-31e0-11e3-aa5a-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_CPSYS_name','urn@bby@pcm@job@ingest@content@warn@count','_CPSYS_type')
          ID: 21759fd4-31e7-11e3-a934-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '21759fd4-31e7-11e3-a934-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_CPSYS_links','urn@bby@pcm@job@ingest@content@complete@count','_CPSYS_name')
          ID: 6efa2222-31f2-11e3-b7b8-001c42000009-CJOB NOMATCH: [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6efa2222-31f2-11e3-b7b8-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@fail@count','urn@bby@pcm@ingest@status')
          ID: f9120571-31e0-11e3-b605-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f9120571-31e0-11e3-b605-001c42000009-CJOB' AND p_prop IN ('_CPSYS_type','_CPSYS_links','subPropA@filttest@sdf','urn@bby@pcm@ingest@data')
          ID: 0c39c361-31e9-11e3-87e6-001c42000009-CJOB NOMATCH: [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '0c39c361-31e9-11e3-87e6-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@fail@count','__CPSYS_type','urn@bby@pcm@ingest@status','subPropA@filttest@sdf')
          ID: c8933813-31e5-11e3-a033-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'c8933813-31e5-11e3-a033-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_CPSYS_name','_CPSYS_links','urn@bby@pcm@ingest@data')
          ID: 559b0c62-31f5-11e3-9684-001c42000009-CJOB NOMATCH: [__CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '559b0c62-31f5-11e3-9684-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','urn@bby@pcm@job@sourceparty@reference@id','__CPSYS_name','urn@bby@pcm@ingest@status')
          ID: b81fdf39-31f2-11e3-bc88-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b81fdf39-31f2-11e3-bc88-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','_CPSYS_name','_CPSYS_type','urn@bby@pcm@ingest@status')
          ID: f92629b1-31e0-11e3-b605-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f92629b1-31e0-11e3-b605-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','_CPSYS_name','CPSYS_links','_CPSYS_type')
          ID: 6ef89b88-31f2-11e3-b7b8-001c42000009-CJOB NOMATCH: [__CPSYS_links, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@complete@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_links, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6ef89b88-31f2-11e3-b7b8-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','urn@bby@pcm@ingest@status','urn@bby@pcm@job@sourceparty@reference@id','__CPSYS_links')
          ID: 03995bd0-31e9-11e3-aa9a-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '03995bd0-31e9-11e3-aa9a-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_CPSYS_type','_CPSYS_links','urn@bby@pcm@job@ingest@content@warn@count')
          ID: a3a70275-31e5-11e3-ad9c-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a3a70275-31e5-11e3-ad9c-001c42000009-CJOB' AND p_prop IN ('_CPSYS_links','subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@success@count','_CPSYS_type')
          ID: 2a318832-31f3-11e3-bf8b-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '2a318832-31f3-11e3-bf8b-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','urn@bby@pcm@job@sourceparty@reference@id','subPropA@filttest@sdf','__CPSYS_links')
          ID: 2c23bb46-31f3-11e3-bf8b-001c42000009-CJOB NOMATCH: [__CPSYS_links, urn@bby@pcm@job@ingest@content@hold@count, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_links, urn@bby@pcm@job@ingest@content@hold@count, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '2c23bb46-31f3-11e3-bf8b-001c42000009-CJOB' AND p_prop IN ('__CPSYS_links','urn@bby@pcm@job@ingest@content@success@count','urn@bby@pcm@job@sourceparty@reference@id','urn@bby@pcm@job@ingest@content@hold@count')
          ID: 25c066e0-31fd-11e3-82e6-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, urn:bby:pcm:job:sourceparty:reference:id, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, urn:bby:pcm:job:sourceparty:reference:id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '25c066e0-31fd-11e3-82e6-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','_CPSYS_links','_CPSYS_name','urn:bby:pcm:job:sourceparty:reference:id')
          ID: b95c09b0-31e7-11e3-9392-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b95c09b0-31e7-11e3-9392-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_CPSYS_links','_CPSYS_type','urn@bby@pcm@job@ingest@content@complete@count')
          ID: 6efa9758-31f2-11e3-b7b8-001c42000009-CJOB NOMATCH: [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6efa9758-31f2-11e3-b7b8-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','__CPSYS_type','urn@bby@pcm@ingest@status','urn@bby@pcm@job@ingest@content@warn@count')
          ID: 3b3612a1-31e8-11e3-af24-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '3b3612a1-31e8-11e3-af24-001c42000009-CJOB' AND p_prop IN ('__CPSYS_links','urn@bby@pcm@job@ingest@content@fail@count','urn@bby@pcm@job@ingest@content@warn@count','subPropA@filttest@sdf')
          ID: 3b210402-31e8-11e3-af24-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '3b210402-31e8-11e3-af24-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@fail@count','_CPSYS_links','subPropA@filttest@sdf','_CPSYS_name')
          ID: b52a58d0-31e5-11e3-9bd1-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b52a58d0-31e5-11e3-9bd1-001c42000009-CJOB' AND p_prop IN ('_CPSYS_type','CPSYS_name','urn@bby@pcm@job@ingest@content@warn@count','_CPSYS_links')
          ID: 88c56831-31e4-11e3-b700-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __SYSTEM__BYPASS__PROP__SUBSET__, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, __SYSTEM__BYPASS__PROP__SUBSET__] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '88c56831-31e4-11e3-b700-001c42000009-CJOB' AND p_prop IN ('_CPSYS_name','SYSTEMBYPASSPROPSUBSET','_CPSYS_links','urn@bby@pcm@job@ingest@content@complete@count')
          ID: 95bfc4f7-31ed-11e3-a7cc-001c42000009-CJOB NOMATCH: [DISABLED__SYSTEM__BYPASS__PROP__SUBSET__, __CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@hold@count] to [DISABLED__SYSTEM__BYPASS__PROP__SUBSET__, __CPSYS_name, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '95bfc4f7-31ed-11e3-a7cc-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@hold@count','_CPSYS_name','urn@bby@pcm@ingest@status','DISABLEDSYSTEMBYPASSPROPSUBSET_')
          ID: dc084823-31ec-11e3-8db9-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'dc084823-31ec-11e3-8db9-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@sourceparty@reference@id','__CPSYS_links','urn@bby@pcm@ingest@data','subPropA@filttest@sdf')
          ID: f667ba14-31ed-11e3-ab0d-001c42000009-CJOB NOMATCH: [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f667ba14-31ed-11e3-ab0d-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','__CPSYS_type','urn@bby@pcm@ingest@status','urn@bby@pcm@job@ingest@content@fail@count')
          ID: a189b4d3-31f2-11e3-bc88-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a189b4d3-31f2-11e3-bc88-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@fail@count','_CPSYS_links','_CPSYS_type')
          ID: d410ff91-31f1-11e3-b75f-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'd410ff91-31f1-11e3-b75f-001c42000009-CJOB' AND p_prop IN ('_CPSYS_links','urn@bby@pcm@ingest@status','_CPSYS_name','urn@bby@pcm@job@ingest@content@success@count')
          ID: 63c2ea71-31ef-11e3-b50f-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@hold@count] to [__CPSYS_links, __CPSYS_type, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '63c2ea71-31ef-11e3-b50f-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@status','urn@bby@pcm@job@ingest@content@hold@count','_CPSYS_type','_CPSYS_links')
          ID: f67e2843-31ed-11e3-ab0d-001c42000009-CJOB NOMATCH: [__CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@hold@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f67e2843-31ed-11e3-ab0d-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','urn@bby@pcm@job@ingest@content@hold@count','urn@bby@pcm@ingest@status','urn@bby@pcm@job@sourceparty@reference@id')
          ID: 50dfa730-31f5-11e3-9684-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, urn@bby@pcm@ingest@data, urn@bby@pcm@job@ingest@content@hold@count] to [__CPSYS_links, __CPSYS_type, urn@bby@pcm@ingest@data] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '50dfa730-31f5-11e3-9684-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@hold@count','_CPSYS_type','_CPSYS_links','urn@bby@pcm@ingest@data')
          ID: a3b6e0f1-31e5-11e3-ad9c-001c42000009-CJOB NOMATCH: [__CPSYS_links, bby:submission:SourceIP, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, bby:submission:SourceIP, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a3b6e0f1-31e5-11e3-ad9c-001c42000009-CJOB' AND p_prop IN ('bby:submission:SourceIP','__CPSYS_links','urn@bby@pcm@job@ingest@content@warn@count','subPropA@filttest@sdf')
          ID: 4a367663-31e2-11e3-9fbe-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '4a367663-31e2-11e3-9fbe-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@fail@count','subPropA@filttest@sdf','_CPSYS_name','_CPSYS_type')
          ID: 6ee7f9b0-31f2-11e3-b7b8-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6ee7f9b0-31f2-11e3-b7b8-001c42000009-CJOB' AND p_prop IN ('_CPSYS_type','urn@bby@pcm@job@ingest@content@warn@count','CPSYS_name','_CPSYS_links')
          ID: 7d5d1c25-31e6-11e3-a91b-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '7d5d1c25-31e6-11e3-a91b-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_CPSYS_links','urn@bby@pcm@job@ingest@content@warn@count','_CPSYS_type')
          ID: a3aafa11-31e5-11e3-ad9c-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a3aafa11-31e5-11e3-ad9c-001c42000009-CJOB' AND p_prop IN ('_CPSYS_links','_CPSYS_type','urn@bby@pcm@job@ingest@content@fail@count','subPropA@filttest@sdf')
          ID: f4c9a830-31f2-11e3-b478-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_type, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f4c9a830-31f2-11e3-b478-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@status','_CPSYS_links','urn@bby@pcm@job@ingest@content@warn@count','_CPSYS_type')
          ID: b52d6616-31e5-11e3-9bd1-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, dct@dateSubmitted, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, dct@dateSubmitted] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b52d6616-31e5-11e3-9bd1-001c42000009-CJOB' AND p_prop IN ('dct@dateSubmitted','_CPSYS_name','urn@bby@pcm@ingest@data','_CPSYS_links')
          ID: 63c693f2-31ef-11e3-b50f-001c42000009-CJOB NOMATCH: [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '63c693f2-31ef-11e3-b50f-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@status','__CPSYS_type','urn@bby@pcm@job@sourceparty@reference@id','urn@bby@pcm@job@ingest@content@success@count')
          ID: bf4d6bf1-31e9-11e3-aed5-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'bf4d6bf1-31e9-11e3-aed5-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','_CPSYS_name','CPSYS_type','_CPSYS_links')
          ID: f91783b3-31e0-11e3-b605-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f91783b3-31e0-11e3-b605-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','__CPSYS_links','urn@bby@pcm@ingest@data','subPropA@filttest@sdf')
          ID: b51f5c51-31e5-11e3-9bd1-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@warn@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b51f5c51-31e5-11e3-9bd1-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','urn@bby@pcm@job@ingest@content@warn@count','_CPSYS_name','_CPSYS_type')
          ID: bf518aa4-31e9-11e3-aed5-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'bf518aa4-31e9-11e3-aed5-001c42000009-CJOB' AND p_prop IN ('_CPSYS_name','urn@bby@pcm@job@ingest@content@complete@count','CPSYS_links','_CPSYS_type')
          ID: 107a6c95-31e4-11e3-b927-001c42000009-CJOB NOMATCH: [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '107a6c95-31e4-11e3-b927-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@warn@count','__CPSYS_type','urn@bby@pcm@job@ingest@content@success@count','subPropA@filttest@sdf')
          ID: 043c0373-31f4-11e3-a64e-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '043c0373-31f4-11e3-a64e-001c42000009-CJOB' AND p_prop IN ('_CPSYS_links','CPSYS_type','_CPSYS_name','urn@bby@pcm@job@ingest@content@success@count')
          ID: a37c44c2-31ec-11e3-b2d5-001c42000009-CJOB NOMATCH: [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a37c44c2-31ec-11e3-b2d5-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','urn@bby@pcm@job@ingest@content@success@count','urn@bby@pcm@job@ingest@content@warn@count','subPropA@filttest@sdf')
          ID: 6e697382-31f5-11e3-8ee2-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@data, urn@bby@pcm@job@ingest@content@hold@count] to [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@data] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6e697382-31f5-11e3-8ee2-001c42000009-CJOB' AND p_prop IN ('_CPSYS_links','_CPSYS_name','urn@bby@pcm@job@ingest@content@hold@count','urn@bby@pcm@ingest@data')
          ID: 9b961433-31e6-11e3-a6c8-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@ingest@status@detail] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@status@detail] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '9b961433-31e6-11e3-a6c8-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@status@detail','subPropA@filttest@sdf','urn@bby@pcm@ingest@data','__CPSYS_links')
          ID: 923ee850-31fd-11e3-97f7-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '923ee850-31fd-11e3-97f7-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@warn@count','_CPSYS_name','CPSYS_type','_CPSYS_links')
          ID: 924d6742-31fd-11e3-97f7-001c42000009-CJOB NOMATCH: [__CPSYS_name, subPropA:filttest:sdf, urn:bby:pcm:ingest:status, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_name, subPropA:filttest:sdf, urn:bby:pcm:ingest:status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','urn:bby:pcm:ingest:status','subPropA:filttest:sdf','urn@bby@pcm@job@ingest@content@complete@count')
          ID: 7217dab0-31e3-11e3-9dbf-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '7217dab0-31e3-11e3-9dbf-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_CPSYS_name','_CPSYS_links','urn@bby@pcm@ingest@data')
          ID: 3b372413-31e8-11e3-af24-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '3b372413-31e8-11e3-af24-001c42000009-CJOB' AND p_prop IN ('_CPSYS_links','_CPSYS_name','urn@bby@pcm@job@ingest@content@complete@count','subPropA@filttest@sdf')
          :-->wayfair_submission.entity_job stats: 10495 succeeded, 97 failed

          Show
          Constance Eustace added a comment - - edited We expect these queries to work since we do a SELECT * on the rowkey, get a list of returned columns, then generate a bunch of queries that select a random set of column keys, and when a column key exists in the select * but does not get returned in a SELECT WHERE IN.. FAILING QUERIES: ID: f924f134-31e0-11e3-b605-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f924f134-31e0-11e3-b605-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_type',' CPSYS_name','urn@bby@pcm@job@ingest@content@warn@count',' _CPSYS_links') ID: b529bc94-31e5-11e3-9bd1-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b529bc94-31e5-11e3-9bd1-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','subPropA@filttest@sdf','_ CPSYS_name',' _CPSYS_type') ID: a4a023f3-31ef-11e3-84cd-001c42000009-CJOB NOMATCH: [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@complete@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a4a023f3-31ef-11e3-84cd-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','urn@bby@pcm@ingest@status','__CPSYS_type','urn@bby@pcm@job@sourceparty@reference@id') ID: b81e318b-31f2-11e3-bc88-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b81e318b-31f2-11e3-bc88-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_links',' CPSYS_name',' _CPSYS_type','urn@bby@pcm@job@ingest@content@complete@count') ID: 58a032e8-31f1-11e3-84cd-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '58a032e8-31f1-11e3-84cd-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','urn@bby@pcm@ingest@data','__CPSYS_links','urn@bby@pcm@job@sourceparty@reference@id') ID: 4a3ba684-31e2-11e3-9fbe-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '4a3ba684-31e2-11e3-9fbe-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','subPropA@filttest@sdf','_ CPSYS_type',' _CPSYS_name') ID: 4a456a81-31e2-11e3-9fbe-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@hold@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '4a456a81-31e2-11e3-9fbe-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_type','subPropA@filttest@sdf',' _CPSYS_links','urn@bby@pcm@job@ingest@content@hold@count') ID: b51ff891-31e5-11e3-9bd1-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@complete@count, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@fail@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b51ff891-31e5-11e3-9bd1-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@fail@count','_ CPSYS_name',' _CPSYS_type','urn@bby@pcm@job@ingest@content@complete@count') ID: b96383c2-31e7-11e3-9392-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b96383c2-31e7-11e3-9392-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','_ CPSYS_name','urn@bby@pcm@job@ingest@content@complete@count',' _CPSYS_type') ID: f6748b53-31ed-11e3-ab0d-001c42000009-CJOB NOMATCH: [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@fail@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f6748b53-31ed-11e3-ab0d-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@sourceparty@reference@id','urn@bby@pcm@ingest@status','urn@bby@pcm@job@ingest@content@fail@count','__CPSYS_type') ID: b951d081-31e7-11e3-9392-001c42000009-CJOB NOMATCH: [DISABLED__SYSTEM__BYPASS__PROP__SUBSET__, __CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count] to [DISABLED__SYSTEM__BYPASS__PROP__SUBSET__, __CPSYS_links, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b951d081-31e7-11e3-9392-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','DISABLED_ SYSTEM BYPASS PROP SUBSET ','subPropA@filttest@sdf',' _CPSYS_links') ID: 6bc17ea0-31ed-11e3-89d8-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6bc17ea0-31ed-11e3-89d8-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_type',' CPSYS_name',' _CPSYS_links','urn@bby@pcm@ingest@data') ID: d4241266-31f1-11e3-b75f-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'd4241266-31f1-11e3-b75f-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@warn@count','_ CPSYS_name',' _CPSYS_type','subPropA@filttest@sdf') ID: d7312824-31ed-11e3-961d-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'd7312824-31ed-11e3-961d-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_ CPSYS_links',' _CPSYS_name','urn@bby@pcm@job@ingest@content@success@count') ID: 0164c793-31f4-11e3-a64e-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '0164c793-31f4-11e3-a64e-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','_ CPSYS_name',' _CPSYS_type','urn@bby@pcm@job@sourceparty@reference@id') ID: f923dfc1-31e0-11e3-b605-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data, urn@bby@pcm@job@ingest@content@hold@count] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f923dfc1-31e0-11e3-b605-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_type',' _CPSYS_name','urn@bby@pcm@ingest@data','urn@bby@pcm@job@ingest@content@hold@count') ID: d6e5bdb7-31e1-11e3-9912-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'd6e5bdb7-31e1-11e3-9912-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_type',' _CPSYS_links','urn@bby@pcm@job@ingest@content@warn@count','subPropA@filttest@sdf') ID: c8955af0-31e5-11e3-a033-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'c8955af0-31e5-11e3-a033-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','_ CPSYS_name',' CPSYS_links',' _CPSYS_type') ID: 2a370676-31f3-11e3-bf8b-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '2a370676-31f3-11e3-bf8b-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','urn@bby@pcm@job@sourceparty@reference@id','subPropA@filttest@sdf','__CPSYS_links') ID: 63e93721-31ef-11e3-b50f-001c42000009-CJOB NOMATCH: [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@fail@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '63e93721-31ef-11e3-b50f-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@sourceparty@reference@id','urn@bby@pcm@job@ingest@content@fail@count','urn@bby@pcm@ingest@status','__CPSYS_type') ID: a3aa36c4-31e5-11e3-ad9c-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@hold@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a3aa36c4-31e5-11e3-ad9c-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_type','urn@bby@pcm@job@ingest@content@hold@count',' CPSYS_links',' _CPSYS_name') ID: 9b9fb120-31e6-11e3-a6c8-001c42000009-CJOB NOMATCH: [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@job@sourceparty@uid] to [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@sourceparty@uid] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '9b9fb120-31e6-11e3-a6c8-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','urn@bby@pcm@ingest@data','__CPSYS_name','urn@bby@pcm@job@sourceparty@uid') ID: f4d19773-31f2-11e3-b478-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@data] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f4d19773-31f2-11e3-b478-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_name','subPropA@filttest@sdf','urn@bby@pcm@ingest@data',' _CPSYS_type') ID: bc6c9cb3-31f5-11e3-a5b8-001c42000009-CJOB NOMATCH: [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'bc6c9cb3-31f5-11e3-a5b8-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','urn@bby@pcm@ingest@status','subPropA@filttest@sdf','__CPSYS_type') ID: 043b4025-31f4-11e3-a64e-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@hold@count, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@hold@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '043b4025-31f4-11e3-a64e-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','urn@bby@pcm@job@ingest@content@hold@count','_ CPSYS_name',' _CPSYS_type') ID: b95c30c3-31e7-11e3-9392-001c42000009-CJOB NOMATCH: [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b95c30c3-31e7-11e3-9392-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','subPropA@filttest@sdf','__CPSYS_type','urn@bby@pcm@job@ingest@content@warn@count') ID: 4a44f552-31e2-11e3-9fbe-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '4a44f552-31e2-11e3-9fbe-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_links','subPropA@filttest@sdf',' _CPSYS_name','urn@bby@pcm@job@ingest@content@success@count') ID: 9b93a333-31e6-11e3-a6c8-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '9b93a333-31e6-11e3-a6c8-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_ CPSYS_links',' _CPSYS_name','urn@bby@pcm@job@ingest@content@warn@count') ID: 559c1dd4-31f5-11e3-9684-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '559c1dd4-31f5-11e3-9684-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_links','urn@bby@pcm@ingest@status','urn@bby@pcm@job@ingest@content@success@count',' _CPSYS_name') ID: 014e8073-31f4-11e3-a64e-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '014e8073-31f4-11e3-a64e-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','urn@bby@pcm@ingest@status','__CPSYS_links','subPropA@filttest@sdf') ID: 8a715897-31e6-11e3-9d0e-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '8a715897-31e6-11e3-9d0e-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','__CPSYS_links','urn@bby@pcm@job@ingest@content@complete@count','subPropA@filttest@sdf') ID: b96a1372-31e7-11e3-9392-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b96a1372-31e7-11e3-9392-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_name',' CPSYS_links','urn@bby@pcm@job@ingest@content@complete@count',' _CPSYS_type') ID: 6bbc9ca1-31ed-11e3-89d8-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, urn@bby@pcm@job@ingest@content@hold@count, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, urn@bby@pcm@job@ingest@content@success@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6bbc9ca1-31ed-11e3-89d8-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@hold@count','_ CPSYS_name',' _CPSYS_links','urn@bby@pcm@job@ingest@content@success@count') ID: a1731f91-31f2-11e3-bc88-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a1731f91-31f2-11e3-bc88-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@status','__CPSYS_links','subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@success@count') ID: b95d4232-31e7-11e3-9392-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b95d4232-31e7-11e3-9392-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@fail@count','subPropA@filttest@sdf','_ CPSYS_links',' _CPSYS_type') ID: a49d16b1-31ef-11e3-84cd-001c42000009-CJOB NOMATCH: [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a49d16b1-31ef-11e3-84cd-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@status','__CPSYS_name','urn@bby@pcm@job@ingest@content@success@count','subPropA@filttest@sdf') ID: a4c49be4-31ef-11e3-84cd-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a4c49be4-31ef-11e3-84cd-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_name',' CPSYS_type',' _CPSYS_links','urn@bby@pcm@job@ingest@content@success@count') ID: 26d00223-31fd-11e3-82e6-001c42000009-CJOB NOMATCH: [subPropA:filttest:sdf, urn:bby:pcm:ingest:status, urn@bby@pcm@job@ingest@content@complete@count, urn@bby@pcm@job@ingest@content@fail@count] to [subPropA:filttest:sdf, urn:bby:pcm:ingest:status, urn@bby@pcm@job@ingest@content@fail@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '26d00223-31fd-11e3-82e6-001c42000009-CJOB' AND p_prop IN ('subPropA:filttest:sdf','urn@bby@pcm@job@ingest@content@fail@count','urn@bby@pcm@job@ingest@content@complete@count','urn:bby:pcm:ingest:status') ID: f67e4f54-31ed-11e3-ab0d-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f67e4f54-31ed-11e3-ab0d-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@complete@count','_ CPSYS_name',' _CPSYS_links') ID: a38879c3-31ec-11e3-b2d5-001c42000009-CJOB NOMATCH: [subPropA@filttest@sdf, urn:bby:pcm:job:sourceparty:reference:id, urn@bby@pcm@job@ingest@content@fail@count, urn@bby@pcm@job@sourceparty@reference@id] to [subPropA@filttest@sdf, urn:bby:pcm:job:sourceparty:reference:id, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a38879c3-31ec-11e3-b2d5-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@sourceparty@reference@id','subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@fail@count','urn:bby:pcm:job:sourceparty:reference:id') ID: 97233243-31fd-11e3-97f7-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA:filttest:sdf, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, subPropA:filttest:sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '97233243-31fd-11e3-97f7-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','subPropA:filttest:sdf','_ CPSYS_links',' _CPSYS_name') ID: f9142853-31e0-11e3-b605-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f9142853-31e0-11e3-b605-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@fail@count','_ CPSYS_type',' _CPSYS_links','subPropA@filttest@sdf') ID: 2170bdd3-31e7-11e3-a934-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '2170bdd3-31e7-11e3-a934-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_links','urn@bby@pcm@ingest@status','urn@bby@pcm@job@ingest@content@warn@count',' _CPSYS_name') ID: d0c65120-31e0-11e3-aa5a-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'd0c65120-31e0-11e3-aa5a-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_name','subPropA@filttest@sdf',' _CPSYS_links','urn@bby@pcm@job@ingest@content@complete@count') ID: f4dedde4-31f2-11e3-b478-001c42000009-CJOB NOMATCH: [__CPSYS_name, urn:bby:pcm:ingest:status, urn@bby@pcm@ingest@data, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_name, urn:bby:pcm:ingest:status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f4dedde4-31f2-11e3-b478-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','__CPSYS_name','urn:bby:pcm:ingest:status','urn@bby@pcm@job@sourceparty@reference@id') ID: 4c8eebb3-31ea-11e3-9c73-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, bby@submission@SourceIP, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, bby@submission@SourceIP] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '4c8eebb3-31ea-11e3-9c73-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_name','bby@submission@SourceIP',' _CPSYS_links','urn@bby@pcm@ingest@data') ID: b61aa4e2-31e8-11e3-8ea5-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b61aa4e2-31e8-11e3-8ea5-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@warn@count','_ CPSYS_type',' _CPSYS_name') ID: d0c2ceb0-31e0-11e3-aa5a-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'd0c2ceb0-31e0-11e3-aa5a-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_ CPSYS_name','urn@bby@pcm@job@ingest@content@warn@count',' _CPSYS_type') ID: 21759fd4-31e7-11e3-a934-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '21759fd4-31e7-11e3-a934-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_ CPSYS_links','urn@bby@pcm@job@ingest@content@complete@count',' _CPSYS_name') ID: 6efa2222-31f2-11e3-b7b8-001c42000009-CJOB NOMATCH: [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6efa2222-31f2-11e3-b7b8-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@fail@count','urn@bby@pcm@ingest@status') ID: f9120571-31e0-11e3-b605-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f9120571-31e0-11e3-b605-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_type',' _CPSYS_links','subPropA@filttest@sdf','urn@bby@pcm@ingest@data') ID: 0c39c361-31e9-11e3-87e6-001c42000009-CJOB NOMATCH: [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '0c39c361-31e9-11e3-87e6-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@fail@count','__CPSYS_type','urn@bby@pcm@ingest@status','subPropA@filttest@sdf') ID: c8933813-31e5-11e3-a033-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'c8933813-31e5-11e3-a033-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_ CPSYS_name',' _CPSYS_links','urn@bby@pcm@ingest@data') ID: 559b0c62-31f5-11e3-9684-001c42000009-CJOB NOMATCH: [__CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '559b0c62-31f5-11e3-9684-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','urn@bby@pcm@job@sourceparty@reference@id','__CPSYS_name','urn@bby@pcm@ingest@status') ID: b81fdf39-31f2-11e3-bc88-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b81fdf39-31f2-11e3-bc88-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','_ CPSYS_name',' _CPSYS_type','urn@bby@pcm@ingest@status') ID: f92629b1-31e0-11e3-b605-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f92629b1-31e0-11e3-b605-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','_ CPSYS_name',' CPSYS_links',' _CPSYS_type') ID: 6ef89b88-31f2-11e3-b7b8-001c42000009-CJOB NOMATCH: [__CPSYS_links, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@complete@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_links, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6ef89b88-31f2-11e3-b7b8-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','urn@bby@pcm@ingest@status','urn@bby@pcm@job@sourceparty@reference@id','__CPSYS_links') ID: 03995bd0-31e9-11e3-aa9a-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '03995bd0-31e9-11e3-aa9a-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_ CPSYS_type',' _CPSYS_links','urn@bby@pcm@job@ingest@content@warn@count') ID: a3a70275-31e5-11e3-ad9c-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a3a70275-31e5-11e3-ad9c-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_links','subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@success@count',' _CPSYS_type') ID: 2a318832-31f3-11e3-bf8b-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '2a318832-31f3-11e3-bf8b-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@data','urn@bby@pcm@job@sourceparty@reference@id','subPropA@filttest@sdf','__CPSYS_links') ID: 2c23bb46-31f3-11e3-bf8b-001c42000009-CJOB NOMATCH: [__CPSYS_links, urn@bby@pcm@job@ingest@content@hold@count, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_links, urn@bby@pcm@job@ingest@content@hold@count, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '2c23bb46-31f3-11e3-bf8b-001c42000009-CJOB' AND p_prop IN ('__CPSYS_links','urn@bby@pcm@job@ingest@content@success@count','urn@bby@pcm@job@sourceparty@reference@id','urn@bby@pcm@job@ingest@content@hold@count') ID: 25c066e0-31fd-11e3-82e6-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, urn:bby:pcm:job:sourceparty:reference:id, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, urn:bby:pcm:job:sourceparty:reference:id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '25c066e0-31fd-11e3-82e6-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@complete@count','_ CPSYS_links',' _CPSYS_name','urn:bby:pcm:job:sourceparty:reference:id') ID: b95c09b0-31e7-11e3-9392-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b95c09b0-31e7-11e3-9392-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_ CPSYS_links',' _CPSYS_type','urn@bby@pcm@job@ingest@content@complete@count') ID: 6efa9758-31f2-11e3-b7b8-001c42000009-CJOB NOMATCH: [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6efa9758-31f2-11e3-b7b8-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','__CPSYS_type','urn@bby@pcm@ingest@status','urn@bby@pcm@job@ingest@content@warn@count') ID: 3b3612a1-31e8-11e3-af24-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '3b3612a1-31e8-11e3-af24-001c42000009-CJOB' AND p_prop IN ('__CPSYS_links','urn@bby@pcm@job@ingest@content@fail@count','urn@bby@pcm@job@ingest@content@warn@count','subPropA@filttest@sdf') ID: 3b210402-31e8-11e3-af24-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '3b210402-31e8-11e3-af24-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@fail@count','_ CPSYS_links','subPropA@filttest@sdf',' _CPSYS_name') ID: b52a58d0-31e5-11e3-9bd1-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b52a58d0-31e5-11e3-9bd1-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_type',' CPSYS_name','urn@bby@pcm@job@ingest@content@warn@count',' _CPSYS_links') ID: 88c56831-31e4-11e3-b700-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __SYSTEM__BYPASS__PROP__SUBSET__, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, __SYSTEM__BYPASS__PROP__SUBSET__] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '88c56831-31e4-11e3-b700-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_name',' SYSTEM BYPASS PROP SUBSET ',' _CPSYS_links','urn@bby@pcm@job@ingest@content@complete@count') ID: 95bfc4f7-31ed-11e3-a7cc-001c42000009-CJOB NOMATCH: [DISABLED__SYSTEM__BYPASS__PROP__SUBSET__, __CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@hold@count] to [DISABLED__SYSTEM__BYPASS__PROP__SUBSET__, __CPSYS_name, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '95bfc4f7-31ed-11e3-a7cc-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@hold@count','_ CPSYS_name','urn@bby@pcm@ingest@status','DISABLED SYSTEM BYPASS PROP SUBSET _') ID: dc084823-31ec-11e3-8db9-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'dc084823-31ec-11e3-8db9-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@sourceparty@reference@id','__CPSYS_links','urn@bby@pcm@ingest@data','subPropA@filttest@sdf') ID: f667ba14-31ed-11e3-ab0d-001c42000009-CJOB NOMATCH: [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f667ba14-31ed-11e3-ab0d-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','__CPSYS_type','urn@bby@pcm@ingest@status','urn@bby@pcm@job@ingest@content@fail@count') ID: a189b4d3-31f2-11e3-bc88-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a189b4d3-31f2-11e3-bc88-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','urn@bby@pcm@job@ingest@content@fail@count','_ CPSYS_links',' _CPSYS_type') ID: d410ff91-31f1-11e3-b75f-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'd410ff91-31f1-11e3-b75f-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_links','urn@bby@pcm@ingest@status',' _CPSYS_name','urn@bby@pcm@job@ingest@content@success@count') ID: 63c2ea71-31ef-11e3-b50f-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@hold@count] to [__CPSYS_links, __CPSYS_type, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '63c2ea71-31ef-11e3-b50f-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@status','urn@bby@pcm@job@ingest@content@hold@count','_ CPSYS_type',' _CPSYS_links') ID: f67e2843-31ed-11e3-ab0d-001c42000009-CJOB NOMATCH: [__CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@hold@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_name, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f67e2843-31ed-11e3-ab0d-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','urn@bby@pcm@job@ingest@content@hold@count','urn@bby@pcm@ingest@status','urn@bby@pcm@job@sourceparty@reference@id') ID: 50dfa730-31f5-11e3-9684-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, urn@bby@pcm@ingest@data, urn@bby@pcm@job@ingest@content@hold@count] to [__CPSYS_links, __CPSYS_type, urn@bby@pcm@ingest@data] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '50dfa730-31f5-11e3-9684-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@hold@count','_ CPSYS_type',' _CPSYS_links','urn@bby@pcm@ingest@data') ID: a3b6e0f1-31e5-11e3-ad9c-001c42000009-CJOB NOMATCH: [__CPSYS_links, bby:submission:SourceIP, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, bby:submission:SourceIP, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a3b6e0f1-31e5-11e3-ad9c-001c42000009-CJOB' AND p_prop IN ('bby:submission:SourceIP','__CPSYS_links','urn@bby@pcm@job@ingest@content@warn@count','subPropA@filttest@sdf') ID: 4a367663-31e2-11e3-9fbe-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_name, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '4a367663-31e2-11e3-9fbe-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@fail@count','subPropA@filttest@sdf','_ CPSYS_name',' _CPSYS_type') ID: 6ee7f9b0-31f2-11e3-b7b8-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6ee7f9b0-31f2-11e3-b7b8-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_type','urn@bby@pcm@job@ingest@content@warn@count',' CPSYS_name',' _CPSYS_links') ID: 7d5d1c25-31e6-11e3-a91b-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '7d5d1c25-31e6-11e3-a91b-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_ CPSYS_links','urn@bby@pcm@job@ingest@content@warn@count',' _CPSYS_type') ID: a3aafa11-31e5-11e3-ad9c-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@fail@count] to [__CPSYS_links, __CPSYS_type, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a3aafa11-31e5-11e3-ad9c-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_links',' _CPSYS_type','urn@bby@pcm@job@ingest@content@fail@count','subPropA@filttest@sdf') ID: f4c9a830-31f2-11e3-b478-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_type, urn@bby@pcm@ingest@status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f4c9a830-31f2-11e3-b478-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@status','_ CPSYS_links','urn@bby@pcm@job@ingest@content@warn@count',' _CPSYS_type') ID: b52d6616-31e5-11e3-9bd1-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, dct@dateSubmitted, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, dct@dateSubmitted] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b52d6616-31e5-11e3-9bd1-001c42000009-CJOB' AND p_prop IN ('dct@dateSubmitted','_ CPSYS_name','urn@bby@pcm@ingest@data',' _CPSYS_links') ID: 63c693f2-31ef-11e3-b50f-001c42000009-CJOB NOMATCH: [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@sourceparty@reference@id] to [__CPSYS_type, urn@bby@pcm@ingest@status, urn@bby@pcm@job@sourceparty@reference@id] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '63c693f2-31ef-11e3-b50f-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@status','__CPSYS_type','urn@bby@pcm@job@sourceparty@reference@id','urn@bby@pcm@job@ingest@content@success@count') ID: bf4d6bf1-31e9-11e3-aed5-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'bf4d6bf1-31e9-11e3-aed5-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','_ CPSYS_name',' CPSYS_type',' _CPSYS_links') ID: f91783b3-31e0-11e3-b605-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'f91783b3-31e0-11e3-b605-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','__CPSYS_links','urn@bby@pcm@ingest@data','subPropA@filttest@sdf') ID: b51f5c51-31e5-11e3-9bd1-001c42000009-CJOB NOMATCH: [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@warn@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'b51f5c51-31e5-11e3-9bd1-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@success@count','urn@bby@pcm@job@ingest@content@warn@count','_ CPSYS_name',' _CPSYS_type') ID: bf518aa4-31e9-11e3-aed5-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'bf518aa4-31e9-11e3-aed5-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_name','urn@bby@pcm@job@ingest@content@complete@count',' CPSYS_links',' _CPSYS_type') ID: 107a6c95-31e4-11e3-b927-001c42000009-CJOB NOMATCH: [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_type, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '107a6c95-31e4-11e3-b927-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@warn@count','__CPSYS_type','urn@bby@pcm@job@ingest@content@success@count','subPropA@filttest@sdf') ID: 043c0373-31f4-11e3-a64e-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@success@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '043c0373-31f4-11e3-a64e-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_links',' CPSYS_type',' _CPSYS_name','urn@bby@pcm@job@ingest@content@success@count') ID: a37c44c2-31ec-11e3-b2d5-001c42000009-CJOB NOMATCH: [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@success@count, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@warn@count] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = 'a37c44c2-31ec-11e3-b2d5-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','urn@bby@pcm@job@ingest@content@success@count','urn@bby@pcm@job@ingest@content@warn@count','subPropA@filttest@sdf') ID: 6e697382-31f5-11e3-8ee2-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@data, urn@bby@pcm@job@ingest@content@hold@count] to [__CPSYS_links, __CPSYS_name, urn@bby@pcm@ingest@data] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '6e697382-31f5-11e3-8ee2-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_links',' _CPSYS_name','urn@bby@pcm@job@ingest@content@hold@count','urn@bby@pcm@ingest@data') ID: 9b961433-31e6-11e3-a6c8-001c42000009-CJOB NOMATCH: [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@data, urn@bby@pcm@ingest@status@detail] to [__CPSYS_links, subPropA@filttest@sdf, urn@bby@pcm@ingest@status@detail] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '9b961433-31e6-11e3-a6c8-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@ingest@status@detail','subPropA@filttest@sdf','urn@bby@pcm@ingest@data','__CPSYS_links') ID: 923ee850-31fd-11e3-97f7-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, __CPSYS_type, urn@bby@pcm@job@ingest@content@warn@count] to [__CPSYS_links, __CPSYS_name, __CPSYS_type] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '923ee850-31fd-11e3-97f7-001c42000009-CJOB' AND p_prop IN ('urn@bby@pcm@job@ingest@content@warn@count','_ CPSYS_name',' CPSYS_type',' _CPSYS_links') ID: 924d6742-31fd-11e3-97f7-001c42000009-CJOB NOMATCH: [__CPSYS_name, subPropA:filttest:sdf, urn:bby:pcm:ingest:status, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_name, subPropA:filttest:sdf, urn:bby:pcm:ingest:status] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','urn:bby:pcm:ingest:status','subPropA:filttest:sdf','urn@bby@pcm@job@ingest@content@complete@count') ID: 7217dab0-31e3-11e3-9dbf-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@ingest@data] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '7217dab0-31e3-11e3-9dbf-001c42000009-CJOB' AND p_prop IN ('subPropA@filttest@sdf','_ CPSYS_name',' _CPSYS_links','urn@bby@pcm@ingest@data') ID: 3b372413-31e8-11e3-af24-001c42000009-CJOB NOMATCH: [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf, urn@bby@pcm@job@ingest@content@complete@count] to [__CPSYS_links, __CPSYS_name, subPropA@filttest@sdf] CQL: SELECT * FROM wayfair_submission.entity_job WHERE e_entid = '3b372413-31e8-11e3-af24-001c42000009-CJOB' AND p_prop IN ('_ CPSYS_links',' _CPSYS_name','urn@bby@pcm@job@ingest@content@complete@count','subPropA@filttest@sdf') :-->wayfair_submission.entity_job stats: 10495 succeeded, 97 failed
          Hide
          Jonathan Ellis added a comment -

          I got lost in the update soup. Did you ever get to "here's how to reproduce" or are we still at "yep, it's broke on my machine?"

          Show
          Jonathan Ellis added a comment - I got lost in the update soup. Did you ever get to "here's how to reproduce" or are we still at "yep, it's broke on my machine?"
          Hide
          Constance Eustace added a comment - - edited

          Here is the confirmed/known information:

          Reproduction has occured in three environments, all single-node (we are moving to three-node rings soon). (Cassandra 2.0.1 for our Ubuntu AWS QA and Dev environments, 1.2.8 Cassandra on my local OSX laptop). Rowcaching is disabled since we saw problems with rowcaching, but keycaching is on. Since restarts and nodetool cache invalidation do nothing, we don't think it is keycaching problems.

          It seems to occur with large ingestions/insertion jobs combined with some update calls coming to existing data. The insertion jobs can be in totally different keyspaces from what we've seen in two of the occurrences. I reproduced locally by doing an insert of 350,000 rowkeys to one keyspace table while running a program that did 1 update/second to a random rowkey/column in another keyspace table.

          The total volume here is only around a million rowkeys or so. There seems to be a bare minimum of something in the range of a million rows before we start seeing problems. Our INSERTs are done as fast as we can, usually at CL.ONE, the updates are more intermittent as processing threads detect and begin processing "Jobs".

          Once we've detected problematic rows we attempted: restarts, nodetool cache invalidations, and nodetool repairs all did nothing to fix the problem. Doing a COPY TO / COPY FROM temporarily fixes the problem, but then it begins to develop again.

          Knowing that :'s are used in the composition of column names, we have also tried @ signs as separators in our columns, but there are located rows/queries with all @ separators exhibiting the problem. There also seems to be a large incidence of our column names ending in "count" that are problematic, but it appears there are a few detected NOMATCHes that don't involve count columns (but there may still be disruption under the hood in those queries).

          ----------------------------------------------------------------------------------------------------------------------------------------

          The problem is detected by doing this:

          • SELECT rowkeys. for each rowkey:
          • SELECT columnkey FROM the rowkey to get all the columnkeys of the row (this appears to work reliably, as does SELECT *, as seen in individual TRACE logs)
          • calculate and perform 16 different random permutations of column key sets and do SELECT * FROM table WHERE rowkey = $rowkey AND columnkey IN ($columnKeySubset)
          • if any of those fail, output the failures along with the select statements and the column keys that WERE returned.

          Different "corrupted" rowkeys will return different subsets of column keys even if they have all the column keys. So if both Row 1 and Row 2 have A B C D E, and we select B,D,E, some return B,D,E, some return BE, some return BD, some return B, etc...

          you can see those NOMATCH outputs above, and samples were further confirmed through CQLSH to confirm it isn't a cassandra-jdbc driver issue.

          ----------------------------------------------------------------------------------------------------------------------------------------

          Above the NOMATCHes, you can see a TRACE output of execution of these two queries seems to show the problem occurring:

          You can see the Thrift calls returning all the columns/composite columns for column keys for the query without the IN clause:

          SELECT * FROM wyfr_submission.entity_job where e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' ;

          __CPSYS_links
          __CPSYS_name
          __CPSYS_type
          subPropA\:filttest\:sdf
          urn\:bby\:pcm\:ingest\:status
          urn\:bby\:pcm\:ingest\:status:ingeststatus
          urn\:bby\:pcm\:job\:sourceparty\:reference\:id
          urn@bby@pcm@job@ingest@content@complete@count

          But doing

          SELECT * FROM wyfr_submission.entity_job WHERE e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','urn:bby:pcm:ingest:status','subPropA:filttest:sdf','urn@bby@pcm@job@ingest@content@complete@count')

          __CPSYS_name
          urn\:bby\:pcm\:ingest\:status
          subPropA\:filttest\:sdf

          ... it doesn't find urn@bby@pcm@job@ingest@content@complete@count

          ----------------------------------------------------------------------------------------------------------------------------------------

          The count columns such as 'urn@bby@pcm@job@ingest@content@complete@count' are not counter type columns. The actual counter values are on a different table, the column keys are just property existence indicators in an Entity-Attribute-Value datamodel.

          ----------------------------------------------------------------------------------------------------------------------------------------

          Finally, we also saw this strangeness:

          SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:key') :: returns 0 rows (expected)
          SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:key', 'exists:column:A', 'exists:column:B') :: returns 0 rows (sparse storage model should still return the existent column keys, right?)

          BUT, doing a simple shortening of the nonexistent key by a bit:

          SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:', 'exists:column:A', 'exists:column:B') :: returns existent 2 rows... huh?

          Then we put another random character at the end of the nonexistent, such as 'J':

          SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:J', 'exists:column:A', 'exists:column:B') :: returns 0 rows...

          But put A, B, or C and 2 rows come back:

          SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:C', 'exists:column:A', 'exists:column:B') :: returns 2 rows...

          ----------------------------------------------------------------------------------------------------------------------------------------

          So this appears to be:

          1) hashing related
          2) uses a Range Slice at the thrift level based on tracing. Hm, this may be an assumption...
          3) so this seems to indicate CASS must be starting from one point and ending at another, but is picking the wrong start and end point from the IN clause that leaves out requested columns.

          This would explain why random nonexistent columns would impact the returned column keys. Manipulating the nonexistent column keys makes the starting/finishing hash change, which excludes some or all columns. Now, why this works for a while then degrades... well I speculate:

          1) while data hasn't been flushed to SSTables / In memory, it all works fine, but the flush changes the hashes or something like that
          2) compaction messes things up.

          ----------------------------------------------------------------------------------------------------------------------------------------

          We have it locally reproduced in an environment I can toggle controlled TRACE/DEBUG logs as well as attach Eclipse debugging to set remote breakpoints and step through. I've already done that once and will attempt a couple more times today to see what the problem may be. If you have any advice for likely areas of the code that the problem may emanate from, I would love to know. I have a couple of days I can devote to investigation, isolation, and root cause analysis...

          ----------------------------------------------------------------------------------------------------------------------------------------

          Our current workaround is to select all columnkeys for rows. This sucks, since it transfers too much extra information, and we are a content system, so some of these values can be big (1K or bigger for descriptions, HTML, etc).

          Show
          Constance Eustace added a comment - - edited Here is the confirmed/known information: Reproduction has occured in three environments, all single-node (we are moving to three-node rings soon). (Cassandra 2.0.1 for our Ubuntu AWS QA and Dev environments, 1.2.8 Cassandra on my local OSX laptop). Rowcaching is disabled since we saw problems with rowcaching, but keycaching is on. Since restarts and nodetool cache invalidation do nothing, we don't think it is keycaching problems. It seems to occur with large ingestions/insertion jobs combined with some update calls coming to existing data. The insertion jobs can be in totally different keyspaces from what we've seen in two of the occurrences. I reproduced locally by doing an insert of 350,000 rowkeys to one keyspace table while running a program that did 1 update/second to a random rowkey/column in another keyspace table. The total volume here is only around a million rowkeys or so. There seems to be a bare minimum of something in the range of a million rows before we start seeing problems. Our INSERTs are done as fast as we can, usually at CL.ONE, the updates are more intermittent as processing threads detect and begin processing "Jobs". Once we've detected problematic rows we attempted: restarts, nodetool cache invalidations, and nodetool repairs all did nothing to fix the problem. Doing a COPY TO / COPY FROM temporarily fixes the problem, but then it begins to develop again. Knowing that :'s are used in the composition of column names, we have also tried @ signs as separators in our columns, but there are located rows/queries with all @ separators exhibiting the problem. There also seems to be a large incidence of our column names ending in "count" that are problematic, but it appears there are a few detected NOMATCHes that don't involve count columns (but there may still be disruption under the hood in those queries). ---------------------------------------------------------------------------------------------------------------------------------------- The problem is detected by doing this: SELECT rowkeys. for each rowkey: SELECT columnkey FROM the rowkey to get all the columnkeys of the row (this appears to work reliably, as does SELECT *, as seen in individual TRACE logs) calculate and perform 16 different random permutations of column key sets and do SELECT * FROM table WHERE rowkey = $rowkey AND columnkey IN ($columnKeySubset) if any of those fail, output the failures along with the select statements and the column keys that WERE returned. Different "corrupted" rowkeys will return different subsets of column keys even if they have all the column keys. So if both Row 1 and Row 2 have A B C D E, and we select B,D,E, some return B,D,E, some return BE, some return BD, some return B, etc... you can see those NOMATCH outputs above, and samples were further confirmed through CQLSH to confirm it isn't a cassandra-jdbc driver issue. ---------------------------------------------------------------------------------------------------------------------------------------- Above the NOMATCHes, you can see a TRACE output of execution of these two queries seems to show the problem occurring: You can see the Thrift calls returning all the columns/composite columns for column keys for the query without the IN clause: SELECT * FROM wyfr_submission.entity_job where e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' ; __CPSYS_links __CPSYS_name __CPSYS_type subPropA\:filttest\:sdf urn\:bby\:pcm\:ingest\:status urn\:bby\:pcm\:ingest\:status:ingeststatus urn\:bby\:pcm\:job\:sourceparty\:reference\:id urn@bby@pcm@job@ingest@content@complete@count But doing SELECT * FROM wyfr_submission.entity_job WHERE e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','urn:bby:pcm:ingest:status','subPropA:filttest:sdf','urn@bby@pcm@job@ingest@content@complete@count') __CPSYS_name urn\:bby\:pcm\:ingest\:status subPropA\:filttest\:sdf ... it doesn't find urn@bby@pcm@job@ingest@content@complete@count ---------------------------------------------------------------------------------------------------------------------------------------- The count columns such as 'urn@bby@pcm@job@ingest@content@complete@count' are not counter type columns. The actual counter values are on a different table, the column keys are just property existence indicators in an Entity-Attribute-Value datamodel. ---------------------------------------------------------------------------------------------------------------------------------------- Finally, we also saw this strangeness: SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:key') :: returns 0 rows (expected) SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:key', 'exists:column:A', 'exists:column:B') :: returns 0 rows (sparse storage model should still return the existent column keys, right?) BUT, doing a simple shortening of the nonexistent key by a bit: SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:', 'exists:column:A', 'exists:column:B') :: returns existent 2 rows... huh? Then we put another random character at the end of the nonexistent, such as 'J': SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:J', 'exists:column:A', 'exists:column:B') :: returns 0 rows... But put A, B, or C and 2 rows come back: SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:C', 'exists:column:A', 'exists:column:B') :: returns 2 rows... ---------------------------------------------------------------------------------------------------------------------------------------- So this appears to be: 1) hashing related 2) uses a Range Slice at the thrift level based on tracing. Hm, this may be an assumption... 3) so this seems to indicate CASS must be starting from one point and ending at another, but is picking the wrong start and end point from the IN clause that leaves out requested columns. This would explain why random nonexistent columns would impact the returned column keys. Manipulating the nonexistent column keys makes the starting/finishing hash change, which excludes some or all columns. Now, why this works for a while then degrades... well I speculate: 1) while data hasn't been flushed to SSTables / In memory, it all works fine, but the flush changes the hashes or something like that 2) compaction messes things up. ---------------------------------------------------------------------------------------------------------------------------------------- We have it locally reproduced in an environment I can toggle controlled TRACE/DEBUG logs as well as attach Eclipse debugging to set remote breakpoints and step through. I've already done that once and will attempt a couple more times today to see what the problem may be. If you have any advice for likely areas of the code that the problem may emanate from, I would love to know. I have a couple of days I can devote to investigation, isolation, and root cause analysis... ---------------------------------------------------------------------------------------------------------------------------------------- Our current workaround is to select all columnkeys for rows. This sucks, since it transfers too much extra information, and we are a content system, so some of these values can be big (1K or bigger for descriptions, HTML, etc).
          Hide
          Jonathan Ellis added a comment -

          "here's how to reproduce" means, tell me how to reproduce starting with CREATE and INSERT.

          Show
          Jonathan Ellis added a comment - "here's how to reproduce" means, tell me how to reproduce starting with CREATE and INSERT.
          Hide
          Constance Eustace added a comment -

          Do you have an email I can send table dumps, audit code, etc to?

          Show
          Constance Eustace added a comment - Do you have an email I can send table dumps, audit code, etc to?
          Hide
          Jonathan Ellis added a comment -

          If you narrow it down to a single table and example you should be able to post it here instead of giving us a needle in a haystack.

          Show
          Jonathan Ellis added a comment - If you narrow it down to a single table and example you should be able to post it here instead of giving us a needle in a haystack.
          Hide
          Constance Eustace added a comment -

          If I send you the data and you load it, the problem doesn't exhibit.

          You have to run updates to one table while concurrently doing large inserts to another keyspace/table. Several hundred thousand to a million rows seems to need to be written before it presents itself, then a ~1% error rate that steadily grows is detected via auditing.

          Then we see the error.

          I have specific examples visibile to me on my machine. How do you want me to show you the data? I can provide rowkeys, queries, and snapshots/etc, but it only presents itself after a certain volume has occurred... I am not familiar with the behavior of the database at a low level but this seems to indicate a threshold where staged data (commit log, in-memory structures, etc) are flushed/persisted...

          Show
          Constance Eustace added a comment - If I send you the data and you load it, the problem doesn't exhibit. You have to run updates to one table while concurrently doing large inserts to another keyspace/table. Several hundred thousand to a million rows seems to need to be written before it presents itself, then a ~1% error rate that steadily grows is detected via auditing. Then we see the error. I have specific examples visibile to me on my machine. How do you want me to show you the data? I can provide rowkeys, queries, and snapshots/etc, but it only presents itself after a certain volume has occurred... I am not familiar with the behavior of the database at a low level but this seems to indicate a threshold where staged data (commit log, in-memory structures, etc) are flushed/persisted...
          Hide
          Jonathan Ellis added a comment -

          Then I suggest creating a sample script to reproduce with the Python CQL driver: https://github.com/datastax/python-driver

          I also suggest experimenting with queries that just do a simple SELECT without IN. The code path is the same so IN is probably just increasing the surface area unnecessarily.

          Show
          Jonathan Ellis added a comment - Then I suggest creating a sample script to reproduce with the Python CQL driver: https://github.com/datastax/python-driver I also suggest experimenting with queries that just do a simple SELECT without IN. The code path is the same so IN is probably just increasing the surface area unnecessarily.
          Hide
          Constance Eustace added a comment -

          I know Groovy... I'll write a script that kicks off simultaneous update, insert, and audit threads and see if I can reproduce using that.

          Show
          Constance Eustace added a comment - I know Groovy... I'll write a script that kicks off simultaneous update, insert, and audit threads and see if I can reproduce using that.
          Hide
          Constance Eustace added a comment -

          I'll create a github project...

          Show
          Constance Eustace added a comment - I'll create a github project...
          Hide
          Constance Eustace added a comment - - edited

          I've apparently reproduced with the decoupled scripts. Cassandra crashed, had an extensive replay, then the problem reared its head.

          I think the problem may be lots and lots of small update BATCH statements. I didn't see it with direct statements or prepared statements, but when I did BATCH, then it appears something happened.

          I'm going to try to definitively reproduce, then setup the github...

          Show
          Constance Eustace added a comment - - edited I've apparently reproduced with the decoupled scripts. Cassandra crashed, had an extensive replay, then the problem reared its head. I think the problem may be lots and lots of small update BATCH statements. I didn't see it with direct statements or prepared statements, but when I did BATCH, then it appears something happened. I'm going to try to definitively reproduce, then setup the github...
          Hide
          Constance Eustace added a comment - - edited

          Here is the code I'm using to reproduce, it's a SpringSource Tool Suite groovy project...

          https://github.com/cowarlydragon/CASS-6137

          But we just got a node restart that has compaction errors due to counter column use. This example does not have counter columns in the schema, but there were existing keyspaces with counters in my run, so I'm going to clean my entire local schema and re-run.

          The error seems to take a bit of time to appear. I ran yesterday on and off while writing the reproduction code and it appeared only after a node crash, but I ran it last night overnight and it was appearing without crash in the morning after about 6 hours of running.

          I've dropped all my user keyspaces and I'll try to reproduce that way and see how long it takes to appear.

          Show
          Constance Eustace added a comment - - edited Here is the code I'm using to reproduce, it's a SpringSource Tool Suite groovy project... https://github.com/cowarlydragon/CASS-6137 But we just got a node restart that has compaction errors due to counter column use. This example does not have counter columns in the schema, but there were existing keyspaces with counters in my run, so I'm going to clean my entire local schema and re-run. The error seems to take a bit of time to appear. I ran yesterday on and off while writing the reproduction code and it appeared only after a node crash, but I ran it last night overnight and it was appearing without crash in the morning after about 6 hours of running. I've dropped all my user keyspaces and I'll try to reproduce that way and see how long it takes to appear.
          Hide
          Constance Eustace added a comment - - edited

          It reproduced in 20 minutes on my bare schema. I did NOT do a full teardown/reinstall of cassandra.

          So the counter columns don't matter.

          I think it's the BATCHed INSERT update mutation (non-insertion) calls. I did runs of this with prepared and literal statements and didn't see this. The BATCH wrapping seems to trigger it though.

          Show
          Constance Eustace added a comment - - edited It reproduced in 20 minutes on my bare schema. I did NOT do a full teardown/reinstall of cassandra. So the counter columns don't matter. I think it's the BATCHed INSERT update mutation (non-insertion) calls. I did runs of this with prepared and literal statements and didn't see this. The BATCH wrapping seems to trigger it though.
          Hide
          Constance Eustace added a comment -

          My output:

          SCHEMA created
          INSERT INIT Completed
          AUDIT:: BEGIN
          UPDATE start
          INSERT start
          INSERT at 10000
          INSERT at 20000
          INSERT at 30000
          INSERT at 40000
          INSERT at 50000
          INSERT at 60000
          INSERT at 70000
          INSERT at 80000
          INSERT at 90000
          INSERT at 100000
          INSERT at 110000
          INSERT at 120000
          INSERT at 130000
          INSERT at 140000
          INSERT at 150000
          UPDATE count: 1000
          INSERT at 160000
          INSERT at 170000
          INSERT at 180000
          INSERT at 190000
          INSERT at 200000
          INSERT at 210000
          INSERT at 220000
          INSERT at 230000
          INSERT at 240000
          INSERT at 250000
          INSERT at 260000
          INSERT at 270000
          INSERT at 280000
          INSERT at 290000
          INSERT at 300000
          INSERT at 310000
          AUDIT::END stats: 20000 succeeded, 0 failed
          AUDIT:: BEGIN
          UPDATE count: 2000
          INSERT at 320000
          INSERT at 330000
          INSERT at 340000
          INSERT time: 230353
          INSERT start
          INSERT at 10000
          INSERT at 20000
          INSERT at 30000
          INSERT at 40000
          INSERT at 50000
          INSERT at 60000
          INSERT at 70000
          INSERT at 80000
          UPDATE count: 3000
          INSERT at 90000
          INSERT at 100000
          INSERT at 110000
          INSERT at 120000
          INSERT at 130000
          INSERT at 140000
          INSERT at 150000
          INSERT at 160000
          INSERT at 170000
          INSERT at 180000
          NOMATCH ID: a82ffea0-30e1-468e-9936-aca174211730_CJOB FOR: [dbpedia:Stock_keeping_unit, urn:bby:pcm:ingest:data, urn:bby:pcm:ingest:status, urn:bby:pcm:ingest:status:detail] to [dbpedia:Stock_keeping_unit, urn:bby:pcm:ingest:data, urn:bby:pcm:ingest:status] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = 'a82ffea0-30e1-468e-9936-aca174211730_CJOB' AND p_prop IN ('urn:bby:pcm:ingest:status:detail','urn:bby:pcm:ingest:data','dbpedia:Stock_keeping_unit','urn:bby:pcm:ingest:status')
          NOMATCH ID: 41a33e18-9e53-4888-a30f-5eba3ba0ce8c_CJOB FOR: [DISABLED__SYSTEM__BYPASS__PROP__SUBSET__, a:count, b:count, b:count:a] to [DISABLED__SYSTEM__BYPASS__PROP__SUBSET__, a:count, b:count] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = '41a33e18-9e53-4888-a30f-5eba3ba0ce8c_CJOB' AND p_prop IN ('a:count','DISABLED_SYSTEMBYPASSPROPSUBSET_','b:count:a','b:count')
          INSERT at 190000
          INSERT at 200000
          INSERT at 210000
          INSERT at 220000
          UPDATE count: 4000
          INSERT at 230000
          AUDIT::END stats: 19998 succeeded, 2 failed
          AUDIT:: BEGIN
          NOMATCH ID: edc42c0b-bd1e-4c75-b34d-9accce2a2f6d_CJOB FOR: [__CPSYS_links, urn:bby:pcm:attribute:miid, urn:bby:pcm:attribute:pimcat, urn:bby:pcm:definition:describes] to [__CPSYS_links, urn:bby:pcm:attribute:miid, urn:bby:pcm:attribute:pimcat] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = 'edc42c0b-bd1e-4c75-b34d-9accce2a2f6d_CJOB' AND p_prop IN ('urn:bby:pcm:attribute:pimcat','urn:bby:pcm:definition:describes','urn:bby:pcm:attribute:miid','__CPSYS_links')
          INSERT at 240000
          INSERT at 250000
          INSERT at 260000
          INSERT at 270000
          INSERT at 280000
          INSERT at 290000
          INSERT at 300000
          INSERT at 310000
          INSERT at 320000
          INSERT at 330000
          UPDATE count: 5000
          INSERT at 340000
          NOMATCH ID: e8901a5d-da47-4550-aa06-3f0770fec040_CJOB FOR: [urn:bby:pcm:job:ingest:content:fail:count, urn:bby:pcm:job:ingest:content:hold:count, urn:bby:pcm:job:ingest:content:success:count, urn:bby:pcm:job:sourceparty:id] to [urn:bby:pcm:job:ingest:content:fail:count, urn:bby:pcm:job:ingest:content:hold:count, urn:bby:pcm:job:sourceparty:id] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = 'e8901a5d-da47-4550-aa06-3f0770fec040_CJOB' AND p_prop IN ('urn:bby:pcm:job:sourceparty:id','urn:bby:pcm:job:ingest:content:hold:count','urn:bby:pcm:job:ingest:content:success:count','urn:bby:pcm:job:ingest:content:fail:count')
          NOMATCH ID: d87d20eb-29cc-481d-affa-0f0de6bd01de_CJOB FOR: [b:count:b, b:count:c, bby:submission:SourceIP, urn:bby:pcm:ingest:status:detail] to [b:count:b, b:count:c, urn:bby:pcm:ingest:status:detail] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = 'd87d20eb-29cc-481d-affa-0f0de6bd01de_CJOB' AND p_prop IN ('urn:bby:pcm:ingest:status:detail','bby:submission:SourceIP','b:count:b','b:count:c')
          NOMATCH ID: 421a6eee-4210-4940-a330-c26794b3ad08_CJOB FOR: [SDjfk:3829jdsf:sdjfl:8sd98hsdf:nnji, dct:dateSubmitted, urn:bby:pcm:attribute:miid, urn:bby:pcm:attribute:pimcat] to [SDjfk:3829jdsf:sdjfl:8sd98hsdf:nnji, dct:dateSubmitted, urn:bby:pcm:attribute:miid] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = '421a6eee-4210-4940-a330-c26794b3ad08_CJOB' AND p_prop IN ('urn:bby:pcm:attribute:miid','urn:bby:pcm:attribute:pimcat','dct:dateSubmitted','SDjfk:3829jdsf:sdjfl:8sd98hsdf:nnji')
          INSERT time: 882330
          INSERT start
          INSERT at 10000
          INSERT at 20000
          INSERT at 30000
          INSERT at 40000
          INSERT at 50000
          INSERT at 60000
          INSERT at 70000
          INSERT at 80000
          INSERT at 90000
          INSERT at 100000
          NOMATCH ID: ef1e1596-d32a-4196-ba44-7d8b78ce2631_CJOB FOR: [__CPSYS_type, a:count, b:count, urn:bby:pcm:job:ingest:content:warn:count] to [__CPSYS_type, a:count, urn:bby:pcm:job:ingest:content:warn:count] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = 'ef1e1596-d32a-4196-ba44-7d8b78ce2631_CJOB' AND p_prop IN ('b:count','a:count','__CPSYS_type','urn:bby:pcm:job:ingest:content:warn:count')
          NOMATCH ID: 7559528b-d210-489b-b101-40fa89a6a49b_CJOB FOR: [dbpedia:Global_Trade_Item_Number, dct:dateSubmitted, urn:bby:pcm:attribute:miid, urn:bby:pcm:job:ingest:content:warn:count] to [dbpedia:Global_Trade_Item_Number, dct:dateSubmitted, urn:bby:pcm:job:ingest:content:warn:count] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = '7559528b-d210-489b-b101-40fa89a6a49b_CJOB' AND p_prop IN ('dbpedia:Global_Trade_Item_Number','urn:bby:pcm:attribute:miid','dct:dateSubmitted','urn:bby:pcm:job:ingest:content:warn:count')
          UPDATE count: 6000
          INSERT at 110000
          NOMATCH ID: 4a5bfdd6-9f1d-43ee-8e3f-ff094be23454_CJOB FOR: [urn:bby:pcm:attribute:pimcat, urn:bby:pcm:job:sourceparty:id, urn:bby:pcm:job:sourceparty:reference:id, urn:bby:pcm:job:sourceparty:uid] to [urn:bby:pcm:attribute:pimcat, urn:bby:pcm:job:sourceparty:id, urn:bby:pcm:job:sourceparty:reference:id] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = '4a5bfdd6-9f1d-43ee-8e3f-ff094be23454_CJOB' AND p_prop IN ('urn:bby:pcm:job:sourceparty:uid','urn:bby:pcm:attribute:pimcat','urn:bby:pcm:job:sourceparty:reference:id','urn:bby:pcm:job:sourceparty:id')
          NOMATCH ID: 2669ed4b-74c1-4050-96f6-85f02c4d86c8_CJOB FOR: [urn:bby:pcm:definition:describes, urn:bby:pcm:ingest:data, urn:bby:pcm:ingest:status, urn:bby:pcm:job:ingest:content:success:count] to [urn:bby:pcm:definition:describes, urn:bby:pcm:ingest:data, urn:bby:pcm:job:ingest:content:success:count] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = '2669ed4b-74c1-4050-96f6-85f02c4d86c8_CJOB' AND p_prop IN ('urn:bby:pcm:ingest:status','urn:bby:pcm:job:ingest:content:success:count','urn:bby:pcm:definition:describes','urn:bby:pcm:ingest:data')
          INSERT at 120000

          Show
          Constance Eustace added a comment - My output: SCHEMA created INSERT INIT Completed AUDIT:: BEGIN UPDATE start INSERT start INSERT at 10000 INSERT at 20000 INSERT at 30000 INSERT at 40000 INSERT at 50000 INSERT at 60000 INSERT at 70000 INSERT at 80000 INSERT at 90000 INSERT at 100000 INSERT at 110000 INSERT at 120000 INSERT at 130000 INSERT at 140000 INSERT at 150000 UPDATE count: 1000 INSERT at 160000 INSERT at 170000 INSERT at 180000 INSERT at 190000 INSERT at 200000 INSERT at 210000 INSERT at 220000 INSERT at 230000 INSERT at 240000 INSERT at 250000 INSERT at 260000 INSERT at 270000 INSERT at 280000 INSERT at 290000 INSERT at 300000 INSERT at 310000 AUDIT::END stats: 20000 succeeded, 0 failed AUDIT:: BEGIN UPDATE count: 2000 INSERT at 320000 INSERT at 330000 INSERT at 340000 INSERT time: 230353 INSERT start INSERT at 10000 INSERT at 20000 INSERT at 30000 INSERT at 40000 INSERT at 50000 INSERT at 60000 INSERT at 70000 INSERT at 80000 UPDATE count: 3000 INSERT at 90000 INSERT at 100000 INSERT at 110000 INSERT at 120000 INSERT at 130000 INSERT at 140000 INSERT at 150000 INSERT at 160000 INSERT at 170000 INSERT at 180000 NOMATCH ID: a82ffea0-30e1-468e-9936-aca174211730_CJOB FOR: [dbpedia:Stock_keeping_unit, urn:bby:pcm:ingest:data, urn:bby:pcm:ingest:status, urn:bby:pcm:ingest:status:detail] to [dbpedia:Stock_keeping_unit, urn:bby:pcm:ingest:data, urn:bby:pcm:ingest:status] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = 'a82ffea0-30e1-468e-9936-aca174211730_CJOB' AND p_prop IN ('urn:bby:pcm:ingest:status:detail','urn:bby:pcm:ingest:data','dbpedia:Stock_keeping_unit','urn:bby:pcm:ingest:status') NOMATCH ID: 41a33e18-9e53-4888-a30f-5eba3ba0ce8c_CJOB FOR: [DISABLED__SYSTEM__BYPASS__PROP__SUBSET__, a:count, b:count, b:count:a] to [DISABLED__SYSTEM__BYPASS__PROP__SUBSET__, a:count, b:count] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = '41a33e18-9e53-4888-a30f-5eba3ba0ce8c_CJOB' AND p_prop IN ('a:count','DISABLED_ SYSTEM BYPASS PROP SUBSET _','b:count:a','b:count') INSERT at 190000 INSERT at 200000 INSERT at 210000 INSERT at 220000 UPDATE count: 4000 INSERT at 230000 AUDIT::END stats: 19998 succeeded, 2 failed AUDIT:: BEGIN NOMATCH ID: edc42c0b-bd1e-4c75-b34d-9accce2a2f6d_CJOB FOR: [__CPSYS_links, urn:bby:pcm:attribute:miid, urn:bby:pcm:attribute:pimcat, urn:bby:pcm:definition:describes] to [__CPSYS_links, urn:bby:pcm:attribute:miid, urn:bby:pcm:attribute:pimcat] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = 'edc42c0b-bd1e-4c75-b34d-9accce2a2f6d_CJOB' AND p_prop IN ('urn:bby:pcm:attribute:pimcat','urn:bby:pcm:definition:describes','urn:bby:pcm:attribute:miid','__CPSYS_links') INSERT at 240000 INSERT at 250000 INSERT at 260000 INSERT at 270000 INSERT at 280000 INSERT at 290000 INSERT at 300000 INSERT at 310000 INSERT at 320000 INSERT at 330000 UPDATE count: 5000 INSERT at 340000 NOMATCH ID: e8901a5d-da47-4550-aa06-3f0770fec040_CJOB FOR: [urn:bby:pcm:job:ingest:content:fail:count, urn:bby:pcm:job:ingest:content:hold:count, urn:bby:pcm:job:ingest:content:success:count, urn:bby:pcm:job:sourceparty:id] to [urn:bby:pcm:job:ingest:content:fail:count, urn:bby:pcm:job:ingest:content:hold:count, urn:bby:pcm:job:sourceparty:id] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = 'e8901a5d-da47-4550-aa06-3f0770fec040_CJOB' AND p_prop IN ('urn:bby:pcm:job:sourceparty:id','urn:bby:pcm:job:ingest:content:hold:count','urn:bby:pcm:job:ingest:content:success:count','urn:bby:pcm:job:ingest:content:fail:count') NOMATCH ID: d87d20eb-29cc-481d-affa-0f0de6bd01de_CJOB FOR: [b:count:b, b:count:c, bby:submission:SourceIP, urn:bby:pcm:ingest:status:detail] to [b:count:b, b:count:c, urn:bby:pcm:ingest:status:detail] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = 'd87d20eb-29cc-481d-affa-0f0de6bd01de_CJOB' AND p_prop IN ('urn:bby:pcm:ingest:status:detail','bby:submission:SourceIP','b:count:b','b:count:c') NOMATCH ID: 421a6eee-4210-4940-a330-c26794b3ad08_CJOB FOR: [SDjfk:3829jdsf:sdjfl:8sd98hsdf:nnji, dct:dateSubmitted, urn:bby:pcm:attribute:miid, urn:bby:pcm:attribute:pimcat] to [SDjfk:3829jdsf:sdjfl:8sd98hsdf:nnji, dct:dateSubmitted, urn:bby:pcm:attribute:miid] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = '421a6eee-4210-4940-a330-c26794b3ad08_CJOB' AND p_prop IN ('urn:bby:pcm:attribute:miid','urn:bby:pcm:attribute:pimcat','dct:dateSubmitted','SDjfk:3829jdsf:sdjfl:8sd98hsdf:nnji') INSERT time: 882330 INSERT start INSERT at 10000 INSERT at 20000 INSERT at 30000 INSERT at 40000 INSERT at 50000 INSERT at 60000 INSERT at 70000 INSERT at 80000 INSERT at 90000 INSERT at 100000 NOMATCH ID: ef1e1596-d32a-4196-ba44-7d8b78ce2631_CJOB FOR: [__CPSYS_type, a:count, b:count, urn:bby:pcm:job:ingest:content:warn:count] to [__CPSYS_type, a:count, urn:bby:pcm:job:ingest:content:warn:count] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = 'ef1e1596-d32a-4196-ba44-7d8b78ce2631_CJOB' AND p_prop IN ('b:count','a:count','__CPSYS_type','urn:bby:pcm:job:ingest:content:warn:count') NOMATCH ID: 7559528b-d210-489b-b101-40fa89a6a49b_CJOB FOR: [dbpedia:Global_Trade_Item_Number, dct:dateSubmitted, urn:bby:pcm:attribute:miid, urn:bby:pcm:job:ingest:content:warn:count] to [dbpedia:Global_Trade_Item_Number, dct:dateSubmitted, urn:bby:pcm:job:ingest:content:warn:count] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = '7559528b-d210-489b-b101-40fa89a6a49b_CJOB' AND p_prop IN ('dbpedia:Global_Trade_Item_Number','urn:bby:pcm:attribute:miid','dct:dateSubmitted','urn:bby:pcm:job:ingest:content:warn:count') UPDATE count: 6000 INSERT at 110000 NOMATCH ID: 4a5bfdd6-9f1d-43ee-8e3f-ff094be23454_CJOB FOR: [urn:bby:pcm:attribute:pimcat, urn:bby:pcm:job:sourceparty:id, urn:bby:pcm:job:sourceparty:reference:id, urn:bby:pcm:job:sourceparty:uid] to [urn:bby:pcm:attribute:pimcat, urn:bby:pcm:job:sourceparty:id, urn:bby:pcm:job:sourceparty:reference:id] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = '4a5bfdd6-9f1d-43ee-8e3f-ff094be23454_CJOB' AND p_prop IN ('urn:bby:pcm:job:sourceparty:uid','urn:bby:pcm:attribute:pimcat','urn:bby:pcm:job:sourceparty:reference:id','urn:bby:pcm:job:sourceparty:id') NOMATCH ID: 2669ed4b-74c1-4050-96f6-85f02c4d86c8_CJOB FOR: [urn:bby:pcm:definition:describes, urn:bby:pcm:ingest:data, urn:bby:pcm:ingest:status, urn:bby:pcm:job:ingest:content:success:count] to [urn:bby:pcm:definition:describes, urn:bby:pcm:ingest:data, urn:bby:pcm:job:ingest:content:success:count] CQL: SELECT * FROM mayfair_submission.entity_job WHERE e_entid = '2669ed4b-74c1-4050-96f6-85f02c4d86c8_CJOB' AND p_prop IN ('urn:bby:pcm:ingest:status','urn:bby:pcm:job:ingest:content:success:count','urn:bby:pcm:definition:describes','urn:bby:pcm:ingest:data') INSERT at 120000
          Hide
          Jonathan Ellis added a comment -

          Did you try 2.0 HEAD?

          Show
          Jonathan Ellis added a comment - Did you try 2.0 HEAD?
          Hide
          Constance Eustace added a comment -

          We've been in 2.0.1 and 1.2.8 (reproduction github code is cass 1.2.8). Seen it in our Dev/qa aws.

          Does compaction reorder column keys or change the hashes?

          Show
          Constance Eustace added a comment - We've been in 2.0.1 and 1.2.8 (reproduction github code is cass 1.2.8). Seen it in our Dev/qa aws. Does compaction reorder column keys or change the hashes?
          Hide
          Constance Eustace added a comment -

          I'm about to do a mandated OS upgrade. I will try HEAD when it comes back up

          Show
          Constance Eustace added a comment - I'm about to do a mandated OS upgrade. I will try HEAD when it comes back up
          Hide
          Constance Eustace added a comment -

          Still in process of setting up HEAD.

          I reproduced with non-batched updates. The mass inserts are still batched.

          Show
          Constance Eustace added a comment - Still in process of setting up HEAD. I reproduced with non-batched updates. The mass inserts are still batched.
          Hide
          Constance Eustace added a comment -
          Show
          Constance Eustace added a comment - On http://cassandra.apache.org/download/ The bleeding edge development snapshots from Jenkins link https://builds.apache.org/job/Cassandra/lastSuccessfulBuild/artifact/cassandra/build/ 404'd
          Hide
          Jeremiah Jordan added a comment -

          That doesn't exist anymore. If you want a tar.gz for 2.0 HEAD, you have to clone the git repo, checkout cassandra-2.0 and then run "ant artifacts" and it will make a "build/apache-cassandra-<version>-SNAPSHOT-bin.tar.gz"

          Show
          Jeremiah Jordan added a comment - That doesn't exist anymore. If you want a tar.gz for 2.0 HEAD, you have to clone the git repo, checkout cassandra-2.0 and then run "ant artifacts" and it will make a "build/apache-cassandra-<version>-SNAPSHOT-bin.tar.gz"
          Hide
          Constance Eustace added a comment -

          The git sync died halfway through due to our awesome network here.

          I will attempt again.

          Show
          Constance Eustace added a comment - The git sync died halfway through due to our awesome network here. I will attempt again.
          Hide
          Constance Eustace added a comment -

          I have reproduced with HEAD 2_1, about the same amount of time.

          I will try out the native protocol rather than cassandra-jdbc. Perhaps also do a script that calls cqlsh so I'm less environment dependent.

          Show
          Constance Eustace added a comment - I have reproduced with HEAD 2_1, about the same amount of time. I will try out the native protocol rather than cassandra-jdbc. Perhaps also do a script that calls cqlsh so I'm less environment dependent.
          Hide
          Constance Eustace added a comment -

          Has anyone reproduced this besides us?

          Show
          Constance Eustace added a comment - Has anyone reproduced this besides us?
          Hide
          Ashot Golovenko added a comment -

          Oh, I think I've just created a duplicate bug.
          Check this out - https://issues.apache.org/jira/browse/CASSANDRA-6220
          Reproduced on 2.0.1

          Show
          Ashot Golovenko added a comment - Oh, I think I've just created a duplicate bug. Check this out - https://issues.apache.org/jira/browse/CASSANDRA-6220 Reproduced on 2.0.1
          Hide
          Constance Eustace added a comment -

          nodetool scrub did nothing to fix it.

          UPDATE: nodetool compact seems to repair the damage! It obviously probably doesn't prevent the reoccurrence and doesn't allow us to trust the queries...

          Show
          Constance Eustace added a comment - nodetool scrub did nothing to fix it. UPDATE: nodetool compact seems to repair the damage! It obviously probably doesn't prevent the reoccurrence and doesn't allow us to trust the queries...
          Hide
          Constance Eustace added a comment - - edited

          Once the initial datarun is done, then a nodetool compact fixes the initial corruption, my second run is now about 4x further than it's been before and no corruption/bad WHERE IN results have occurred.

          This could be some initial confusion in the internal data structures on newly created keyspaces that lack data, or once the compaction thread catches up.

          EDIT: I've done runs that did 10x the data volume on inserts and 5x the number of update threads and there have been no issues if proper nodetool flush / nodetool compact / nodetool keycacheinvalidate is done.

          Show
          Constance Eustace added a comment - - edited Once the initial datarun is done, then a nodetool compact fixes the initial corruption, my second run is now about 4x further than it's been before and no corruption/bad WHERE IN results have occurred. This could be some initial confusion in the internal data structures on newly created keyspaces that lack data, or once the compaction thread catches up. EDIT: I've done runs that did 10x the data volume on inserts and 5x the number of update threads and there have been no issues if proper nodetool flush / nodetool compact / nodetool keycacheinvalidate is done.
          Hide
          Vito Giuliani added a comment -

          I've hit this bug too in my local environment (using C* 2.0.2), both in the java driver (datastax's) and cqlsh:

          SELECT code, date, price FROM inventory_price_by_day WHERE code='3853853853852' and date in ('2013-11-08', '2013-11-09', '2013-11-10', '2013-11-11', '2013-11-12', '2013-11-13', '2013-11-14', '2013-11-15');
          
           code          | date       | price
          ---------------+------------+-------
           3853853853852 | 2013-11-08 | 66.00
           3853853853852 | 2013-11-09 | 66.00
           3853853853852 | 2013-11-10 | 66.00
           3853853853852 | 2013-11-11 | 66.00
           3853853853852 | 2013-11-12 | 66.00
           3853853853852 | 2013-11-13 | 66.00
           3853853853852 | 2013-11-14 | 66.00
           3853853853852 | 2013-11-15 | 66.00
          
          (8 rows)
          
          SELECT code, date, price FROM inventory_price_by_day WHERE code='3853853853852' and date in ('2013-11-07', '2013-11-08', '2013-11-09', '2013-11-10', '2013-11-11', '2013-11-12', '2013-11-13', '2013-11-14', '2013-11-15');
          
           code          | date       | price
          ---------------+------------+-------
           3853853853852 | 2013-11-15 | 66.00
          
          (1 rows)
          

          (the only difference between the two queries is that the latter includes an additional day)

          I tried to run flush / compact / keycacheinvalidate but they don't seem to have any kind of effect here.
          Enabling tracing, there seems to be a difference in the way the two queries are executed:

           activity                                                                                                                                                                                                                          | timestamp    | source    | source_elapsed
          -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+-----------+----------------
                                                                                                                                                                                                                          execute_cql3_query | 09:23:28,987 | 127.0.0.1 |              0
           Parsing SELECT code, date, price FROM inventory_price_by_day WHERE code='3853853853852' and date in ('2013-11-08', '2013-11-09', '2013-11-10', '2013-11-11', '2013-11-12', '2013-11-13', '2013-11-14', '2013-11-15') LIMIT 10000; | 09:23:28,987 | 127.0.0.1 |             66
                                                                                                                                                                                                                         Preparing statement | 09:23:28,987 | 127.0.0.1 |            161
                                                                                                                                                                                  Executing single-partition query on inventory_price_by_day | 09:23:28,988 | 127.0.0.1 |            467
                                                                                                                                                                                                                Acquiring sstable references | 09:23:28,988 | 127.0.0.1 |            488
                                                                                                                                                                                                                 Merging memtable tombstones | 09:23:28,988 | 127.0.0.1 |            514
                                                                                                                                                                                                                 Key cache hit for sstable 4 | 09:23:28,988 | 127.0.0.1 |            589
                                                                                                                                                                                           Seeking to partition indexed section in data file | 09:23:28,988 | 127.0.0.1 |            604
                                                                                                                                                                   Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | 09:23:28,988 | 127.0.0.1 |            870
                                                                                                                                                                                                  Merging data from memtables and 1 sstables | 09:23:28,988 | 127.0.0.1 |            893
                                                                                                                                                                                                          Read 8 live and 0 tombstoned cells | 09:23:28,988 | 127.0.0.1 |           1045
                                                                                                                                                                                                                            Request complete | 09:23:28,988 | 127.0.0.1 |           1216
          
          
           activity                                                                                                                                                                                                                                        | timestamp    | source    | source_elapsed
          -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+-----------+----------------
                                                                                                                                                                                                                                        execute_cql3_query | 09:23:32,003 | 127.0.0.1 |              0
           Parsing SELECT code, date, price FROM inventory_price_by_day WHERE code='3853853853852' and date in ('2013-11-07', '2013-11-08', '2013-11-09', '2013-11-10', '2013-11-11', '2013-11-12', '2013-11-13', '2013-11-14', '2013-11-15') LIMIT 10000; | 09:23:32,004 | 127.0.0.1 |             76
                                                                                                                                                                                                                                       Preparing statement | 09:23:32,004 | 127.0.0.1 |            184
                                                                                                                                                                                                Executing single-partition query on inventory_price_by_day | 09:23:32,004 | 127.0.0.1 |            491
                                                                                                                                                                                                                              Acquiring sstable references | 09:23:32,004 | 127.0.0.1 |            514
                                                                                                                                                                                                                               Merging memtable tombstones | 09:23:32,004 | 127.0.0.1 |            540
                                                                                                                                                                                 Skipped 1/1 non-slice-intersecting sstables, included 0 due to tombstones | 09:23:32,004 | 127.0.0.1 |            601
                                                                                                                                                                                                                Merging data from memtables and 0 sstables | 09:23:32,004 | 127.0.0.1 |            622
                                                                                                                                                                                                                        Read 1 live and 0 tombstoned cells | 09:23:32,004 | 127.0.0.1 |            757
                                                                                                                                                                                                                                          Request complete | 09:23:32,003 | 127.0.0.1 |            960
          
          

          Notice the "Skipped 1/1 non-slice-intersecting sstables" in the second query, which is interesting given that there's only 1 sstable for that column family.

          Show
          Vito Giuliani added a comment - I've hit this bug too in my local environment (using C* 2.0.2), both in the java driver (datastax's) and cqlsh: SELECT code, date, price FROM inventory_price_by_day WHERE code='3853853853852' and date in ('2013-11-08', '2013-11-09', '2013-11-10', '2013-11-11', '2013-11-12', '2013-11-13', '2013-11-14', '2013-11-15'); code | date | price ---------------+------------+------- 3853853853852 | 2013-11-08 | 66.00 3853853853852 | 2013-11-09 | 66.00 3853853853852 | 2013-11-10 | 66.00 3853853853852 | 2013-11-11 | 66.00 3853853853852 | 2013-11-12 | 66.00 3853853853852 | 2013-11-13 | 66.00 3853853853852 | 2013-11-14 | 66.00 3853853853852 | 2013-11-15 | 66.00 (8 rows) SELECT code, date, price FROM inventory_price_by_day WHERE code='3853853853852' and date in ('2013-11-07', '2013-11-08', '2013-11-09', '2013-11-10', '2013-11-11', '2013-11-12', '2013-11-13', '2013-11-14', '2013-11-15'); code | date | price ---------------+------------+------- 3853853853852 | 2013-11-15 | 66.00 (1 rows) (the only difference between the two queries is that the latter includes an additional day) I tried to run flush / compact / keycacheinvalidate but they don't seem to have any kind of effect here. Enabling tracing, there seems to be a difference in the way the two queries are executed: activity | timestamp | source | source_elapsed -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+-----------+---------------- execute_cql3_query | 09:23:28,987 | 127.0.0.1 | 0 Parsing SELECT code, date, price FROM inventory_price_by_day WHERE code='3853853853852' and date in ('2013-11-08', '2013-11-09', '2013-11-10', '2013-11-11', '2013-11-12', '2013-11-13', '2013-11-14', '2013-11-15') LIMIT 10000; | 09:23:28,987 | 127.0.0.1 | 66 Preparing statement | 09:23:28,987 | 127.0.0.1 | 161 Executing single-partition query on inventory_price_by_day | 09:23:28,988 | 127.0.0.1 | 467 Acquiring sstable references | 09:23:28,988 | 127.0.0.1 | 488 Merging memtable tombstones | 09:23:28,988 | 127.0.0.1 | 514 Key cache hit for sstable 4 | 09:23:28,988 | 127.0.0.1 | 589 Seeking to partition indexed section in data file | 09:23:28,988 | 127.0.0.1 | 604 Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | 09:23:28,988 | 127.0.0.1 | 870 Merging data from memtables and 1 sstables | 09:23:28,988 | 127.0.0.1 | 893 Read 8 live and 0 tombstoned cells | 09:23:28,988 | 127.0.0.1 | 1045 Request complete | 09:23:28,988 | 127.0.0.1 | 1216 activity | timestamp | source | source_elapsed -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+-----------+---------------- execute_cql3_query | 09:23:32,003 | 127.0.0.1 | 0 Parsing SELECT code, date, price FROM inventory_price_by_day WHERE code='3853853853852' and date in ('2013-11-07', '2013-11-08', '2013-11-09', '2013-11-10', '2013-11-11', '2013-11-12', '2013-11-13', '2013-11-14', '2013-11-15') LIMIT 10000; | 09:23:32,004 | 127.0.0.1 | 76 Preparing statement | 09:23:32,004 | 127.0.0.1 | 184 Executing single-partition query on inventory_price_by_day | 09:23:32,004 | 127.0.0.1 | 491 Acquiring sstable references | 09:23:32,004 | 127.0.0.1 | 514 Merging memtable tombstones | 09:23:32,004 | 127.0.0.1 | 540 Skipped 1/1 non-slice-intersecting sstables, included 0 due to tombstones | 09:23:32,004 | 127.0.0.1 | 601 Merging data from memtables and 0 sstables | 09:23:32,004 | 127.0.0.1 | 622 Read 1 live and 0 tombstoned cells | 09:23:32,004 | 127.0.0.1 | 757 Request complete | 09:23:32,003 | 127.0.0.1 | 960 Notice the "Skipped 1/1 non-slice-intersecting sstables" in the second query, which is interesting given that there's only 1 sstable for that column family.
          Hide
          Sylvain Lebresne added a comment -

          Pretty sure this is the same than CASSANDRA-6327 which has been fixed for 2.0.3.

          Show
          Sylvain Lebresne added a comment - Pretty sure this is the same than CASSANDRA-6327 which has been fixed for 2.0.3.

            People

            • Assignee:
              Unassigned
              Reporter:
              Constance Eustace
            • Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development