Apache Gora
  1. Apache Gora
  2. GORA-174

GORA compiler does not handle ["string", "null"] unions in the AVRO schema

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.2.1
    • Fix Version/s: 0.4
    • Component/s: schema
    • Labels:
      None

      Description

      See NUTCH-1477 for description.
      We are getting NPE when using the DataFileAvroStore, in order to avoid that I modified the schema to allow for null values on some fields e.g.

      {"name": "baseUrl", "type": ["string", "null"] }

      however when generating the code for the schema the accessors are not generated by GORA which prevents Nutch from compiling

      1. GORA-174-test.patch
        2 kB
        Julien Nioche
      2. GORA-174.patch
        15 kB
        Alfonso Nishikawa
      3. GORA-174.patch
        16 kB
        Alfonso Nishikawa
      4. GORA-174.patch
        17 kB
        Alfonso Nishikawa
      5. GORA-174.patch
        34 kB
        Alfonso Nishikawa
      6. GORA-174_over_r1462580.patch
        1 kB
        Alfonso Nishikawa

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          In Progress In Progress Open Open
          14h 28m 1 Alfonso Nishikawa 19/Nov/12 21:50
          Open Open In Progress In Progress
          45d 3h 8m 2 Alfonso Nishikawa 11/Dec/12 14:01
          In Progress In Progress Resolved Resolved
          184d 10h 54m 1 Lewis John McGibbney 14/Jun/13 01:56
          Henry Saputra made changes -
          Link This issue is required by NUTCH-1477 [ NUTCH-1477 ]
          Lewis John McGibbney made changes -
          Link This issue is related to GORA-223 [ GORA-223 ]
          Lewis John McGibbney made changes -
          Link This issue is blocked by GORA-223 [ GORA-223 ]
          Hide
          Hudson added a comment -

          Integrated in goraOracle #6 (See https://builds.apache.org/job/goraOracle/6/)
          GORA-174 GORA compiler does not handle [string, null] unions in the AVRO schema (Revision 97cf439fbdc95bd495e74b2e430ad954fb276bea)

          Result = FAILURE
          lewismc :
          Files :

          • gora-tutorial/conf/gora.properties
          • gora-hbase/src/test/conf/gora-hbase-mapping.xml
          • gora-core/src/examples/avro/webpage.json
          • gora-core/src/main/java/org/apache/gora/persistency/impl/PersistentBase.java
          • gora-hbase/src/test/java/org/apache/gora/hbase/store/TestHBaseStore.java
          • gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraColumn.java
          • gora-core/src/examples/avro/employee.json
          • gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java
          • gora-core/src/examples/java/org/apache/gora/examples/generated/Metadata.java
          • gora-hbase/src/main/java/org/apache/gora/hbase/util/HBaseByteInterface.java
          • gora-cassandra/src/test/java/org/apache/gora/cassandra/store/TestCassandraStore.java
          • gora-core/src/main/java/org/apache/gora/compiler/GoraCompiler.java
          • CHANGES.txt
          • gora-core/src/examples/java/org/apache/gora/examples/generated/TokenDatum.java
          • gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMappingManager.java
          • gora-core/src/examples/java/org/apache/gora/examples/mapreduce/WordCount.java
          • gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraResult.java
          • gora-core/src/examples/java/org/apache/gora/examples/generated/Employee.java
          • gora-tutorial/src/main/java/org/apache/gora/tutorial/log/generated/MetricDatum.java
          • gora-core/src/test/java/org/apache/gora/store/DataStoreTestBase.java
          • gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java
          • gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraRow.java
          • gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraClient.java
          • gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMapping.java
          • gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraSubColumn.java
          • gora-tutorial/src/main/java/org/apache/gora/tutorial/log/generated/Pageview.java
          • gora-core/src/test/java/org/apache/gora/store/DataStoreTestUtil.java
          • gora-accumulo/src/test/resources/gora-accumulo-mapping.xml
          • gora-core/src/examples/java/org/apache/gora/examples/WebPageDataCreator.java
          • gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java
          • gora-core/src/test/java/org/apache/gora/mapreduce/MapReduceTestUtils.java
          • gora-cassandra/src/main/java/org/apache/gora/cassandra/serializers/GoraSerializerTypeInferer.java
          • gora-core/src/examples/java/org/apache/gora/examples/generated/WebPage.java
          • pom.xml
          • gora-cassandra/src/test/conf/gora-cassandra-mapping.xml
          Show
          Hudson added a comment - Integrated in goraOracle #6 (See https://builds.apache.org/job/goraOracle/6/ ) GORA-174 GORA compiler does not handle [string, null] unions in the AVRO schema (Revision 97cf439fbdc95bd495e74b2e430ad954fb276bea) Result = FAILURE lewismc : Files : gora-tutorial/conf/gora.properties gora-hbase/src/test/conf/gora-hbase-mapping.xml gora-core/src/examples/avro/webpage.json gora-core/src/main/java/org/apache/gora/persistency/impl/PersistentBase.java gora-hbase/src/test/java/org/apache/gora/hbase/store/TestHBaseStore.java gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraColumn.java gora-core/src/examples/avro/employee.json gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java gora-core/src/examples/java/org/apache/gora/examples/generated/Metadata.java gora-hbase/src/main/java/org/apache/gora/hbase/util/HBaseByteInterface.java gora-cassandra/src/test/java/org/apache/gora/cassandra/store/TestCassandraStore.java gora-core/src/main/java/org/apache/gora/compiler/GoraCompiler.java CHANGES.txt gora-core/src/examples/java/org/apache/gora/examples/generated/TokenDatum.java gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMappingManager.java gora-core/src/examples/java/org/apache/gora/examples/mapreduce/WordCount.java gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraResult.java gora-core/src/examples/java/org/apache/gora/examples/generated/Employee.java gora-tutorial/src/main/java/org/apache/gora/tutorial/log/generated/MetricDatum.java gora-core/src/test/java/org/apache/gora/store/DataStoreTestBase.java gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraRow.java gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraClient.java gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMapping.java gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraSubColumn.java gora-tutorial/src/main/java/org/apache/gora/tutorial/log/generated/Pageview.java gora-core/src/test/java/org/apache/gora/store/DataStoreTestUtil.java gora-accumulo/src/test/resources/gora-accumulo-mapping.xml gora-core/src/examples/java/org/apache/gora/examples/WebPageDataCreator.java gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java gora-core/src/test/java/org/apache/gora/mapreduce/MapReduceTestUtils.java gora-cassandra/src/main/java/org/apache/gora/cassandra/serializers/GoraSerializerTypeInferer.java gora-core/src/examples/java/org/apache/gora/examples/generated/WebPage.java pom.xml gora-cassandra/src/test/conf/gora-cassandra-mapping.xml
          Hide
          Hudson added a comment -

          Integrated in gora-trunk #749 (See https://builds.apache.org/job/gora-trunk/749/)
          GORA-174 GORA compiler does not handle [string, null] unions in the AVRO schema (Revision 1492920)

          Result = SUCCESS
          lewismc : http://svn.apache.org/viewvc/gora/trunk/?view=rev&rev=1492920
          Files :

          • /gora/trunk/CHANGES.txt
          • /gora/trunk/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java
          • /gora/trunk/gora-accumulo/src/test/resources/gora-accumulo-mapping.xml
          • /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraColumn.java
          • /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraResult.java
          • /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraRow.java
          • /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraSubColumn.java
          • /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/serializers/GoraSerializerTypeInferer.java
          • /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraClient.java
          • /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMapping.java
          • /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMappingManager.java
          • /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java
          • /gora/trunk/gora-cassandra/src/test/conf/gora-cassandra-mapping.xml
          • /gora/trunk/gora-cassandra/src/test/java/org/apache/gora/cassandra/store/TestCassandraStore.java
          • /gora/trunk/gora-core/src/examples/avro/employee.json
          • /gora/trunk/gora-core/src/examples/avro/webpage.json
          • /gora/trunk/gora-core/src/examples/java/org/apache/gora/examples/WebPageDataCreator.java
          • /gora/trunk/gora-core/src/examples/java/org/apache/gora/examples/generated/Employee.java
          • /gora/trunk/gora-core/src/examples/java/org/apache/gora/examples/generated/Metadata.java
          • /gora/trunk/gora-core/src/examples/java/org/apache/gora/examples/generated/TokenDatum.java
          • /gora/trunk/gora-core/src/examples/java/org/apache/gora/examples/generated/WebPage.java
          • /gora/trunk/gora-core/src/examples/java/org/apache/gora/examples/mapreduce/WordCount.java
          • /gora/trunk/gora-core/src/main/java/org/apache/gora/compiler/GoraCompiler.java
          • /gora/trunk/gora-core/src/main/java/org/apache/gora/persistency/impl/PersistentBase.java
          • /gora/trunk/gora-core/src/test/java/org/apache/gora/mapreduce/MapReduceTestUtils.java
          • /gora/trunk/gora-core/src/test/java/org/apache/gora/store/DataStoreTestBase.java
          • /gora/trunk/gora-core/src/test/java/org/apache/gora/store/DataStoreTestUtil.java
          • /gora/trunk/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java
          • /gora/trunk/gora-hbase/src/main/java/org/apache/gora/hbase/util/HBaseByteInterface.java
          • /gora/trunk/gora-hbase/src/test/conf/gora-hbase-mapping.xml
          • /gora/trunk/gora-hbase/src/test/java/org/apache/gora/hbase/store/TestHBaseStore.java
          • /gora/trunk/gora-tutorial/conf/gora.properties
          • /gora/trunk/gora-tutorial/src/main/java/org/apache/gora/tutorial/log/generated/MetricDatum.java
          • /gora/trunk/gora-tutorial/src/main/java/org/apache/gora/tutorial/log/generated/Pageview.java
          • /gora/trunk/pom.xml
          Show
          Hudson added a comment - Integrated in gora-trunk #749 (See https://builds.apache.org/job/gora-trunk/749/ ) GORA-174 GORA compiler does not handle [string, null] unions in the AVRO schema (Revision 1492920) Result = SUCCESS lewismc : http://svn.apache.org/viewvc/gora/trunk/?view=rev&rev=1492920 Files : /gora/trunk/CHANGES.txt /gora/trunk/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java /gora/trunk/gora-accumulo/src/test/resources/gora-accumulo-mapping.xml /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraColumn.java /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraResult.java /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraRow.java /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraSubColumn.java /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/serializers/GoraSerializerTypeInferer.java /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraClient.java /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMapping.java /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMappingManager.java /gora/trunk/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java /gora/trunk/gora-cassandra/src/test/conf/gora-cassandra-mapping.xml /gora/trunk/gora-cassandra/src/test/java/org/apache/gora/cassandra/store/TestCassandraStore.java /gora/trunk/gora-core/src/examples/avro/employee.json /gora/trunk/gora-core/src/examples/avro/webpage.json /gora/trunk/gora-core/src/examples/java/org/apache/gora/examples/WebPageDataCreator.java /gora/trunk/gora-core/src/examples/java/org/apache/gora/examples/generated/Employee.java /gora/trunk/gora-core/src/examples/java/org/apache/gora/examples/generated/Metadata.java /gora/trunk/gora-core/src/examples/java/org/apache/gora/examples/generated/TokenDatum.java /gora/trunk/gora-core/src/examples/java/org/apache/gora/examples/generated/WebPage.java /gora/trunk/gora-core/src/examples/java/org/apache/gora/examples/mapreduce/WordCount.java /gora/trunk/gora-core/src/main/java/org/apache/gora/compiler/GoraCompiler.java /gora/trunk/gora-core/src/main/java/org/apache/gora/persistency/impl/PersistentBase.java /gora/trunk/gora-core/src/test/java/org/apache/gora/mapreduce/MapReduceTestUtils.java /gora/trunk/gora-core/src/test/java/org/apache/gora/store/DataStoreTestBase.java /gora/trunk/gora-core/src/test/java/org/apache/gora/store/DataStoreTestUtil.java /gora/trunk/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java /gora/trunk/gora-hbase/src/main/java/org/apache/gora/hbase/util/HBaseByteInterface.java /gora/trunk/gora-hbase/src/test/conf/gora-hbase-mapping.xml /gora/trunk/gora-hbase/src/test/java/org/apache/gora/hbase/store/TestHBaseStore.java /gora/trunk/gora-tutorial/conf/gora.properties /gora/trunk/gora-tutorial/src/main/java/org/apache/gora/tutorial/log/generated/MetricDatum.java /gora/trunk/gora-tutorial/src/main/java/org/apache/gora/tutorial/log/generated/Pageview.java /gora/trunk/pom.xml
          Hide
          Henry Saputra added a comment -

          Cool! Thanks Lewis. Here we go =)

          Show
          Henry Saputra added a comment - Cool! Thanks Lewis. Here we go =)
          Lewis John McGibbney made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Lewis John McGibbney added a comment -

          Committed @revision 1492920 in trunk as part of GORA-242.

          Show
          Lewis John McGibbney added a comment - Committed @revision 1492920 in trunk as part of GORA-242 .
          Hide
          Lewis John McGibbney added a comment -

          Hi Alfonso, I think we've all bee lost in thought. From something seemingly trivial (nutch avsc) it has snowballed into a rather interesting and technically challenging (for me anyway) issue.
          I would like to propose that we implement ONLY what is necessary to satisfy support for ["string", "null"] unions. Of course Accumulo, HBase, Cassandra should also support this. Hence the child issues assigned to this issue.
          Everything else over and above the ["string", "null"] unions should be addressed in subsequent issues.
          We need to watch that our eyes are not bigger than our stomachs here. I've already made this problem for lunch today (I got a big pizza... way too much).
          Thanks for the dialogue.

          Show
          Lewis John McGibbney added a comment - Hi Alfonso, I think we've all bee lost in thought. From something seemingly trivial (nutch avsc) it has snowballed into a rather interesting and technically challenging (for me anyway) issue. I would like to propose that we implement ONLY what is necessary to satisfy support for ["string", "null"] unions. Of course Accumulo, HBase, Cassandra should also support this. Hence the child issues assigned to this issue. Everything else over and above the ["string", "null"] unions should be addressed in subsequent issues. We need to watch that our eyes are not bigger than our stomachs here. I've already made this problem for lunch today (I got a big pizza... way too much). Thanks for the dialogue.
          Hide
          Alfonso Nishikawa added a comment -

          Hi, Lewis.
          Good point. Didn't noticed until now (it seems I was really lost in thought).
          Then, the test testGetNested() is a must, but has to be modified a bit. In that test, the nested record has a 3-types-union and must be converted to a 2-types-union in order to be fully compilant with the specification (only ["null","string"] a.k.a. optional fields). Otherwise, the specification should be changed to:
          "
          support ["null","string"] (a.k.a optional field) in top-most fields of schema (and not inner records).
          "

          Show
          Alfonso Nishikawa added a comment - Hi, Lewis. Good point. Didn't noticed until now (it seems I was really lost in thought). Then, the test testGetNested() is a must, but has to be modified a bit. In that test, the nested record has a 3-types-union and must be converted to a 2-types-union in order to be fully compilant with the specification (only ["null","string"] a.k.a. optional fields). Otherwise, the specification should be changed to: " support ["null","string"] (a.k.a optional field) in top-most fields of schema (and not inner records). "
          Hide
          Lewis John McGibbney added a comment - - edited

          This issue is to support single type Unions within Avro schemas. AFAIK we have committed support for this within all modules in the GORA_174 datastore.
          Recursive nested structures and multi-type Union support are different issues and they are confusing this parent issue very much.
          Unless there is strong argument I propose to close this issue as resolved as well as all children issues relating only to single type union support within Avro schemas.

          Regarding support for multiple union type support and the recursive issue we should open and address these issues elsewhere.

          Show
          Lewis John McGibbney added a comment - - edited This issue is to support single type Unions within Avro schemas. AFAIK we have committed support for this within all modules in the GORA_174 datastore. Recursive nested structures and multi-type Union support are different issues and they are confusing this parent issue very much. Unless there is strong argument I propose to close this issue as resolved as well as all children issues relating only to single type union support within Avro schemas. Regarding support for multiple union type support and the recursive issue we should open and address these issues elsewhere.
          Lewis John McGibbney made changes -
          Fix Version/s 0.4 [ 12319890 ]
          Fix Version/s 0.3 [ 12317954 ]
          Gavin made changes -
          Link This issue is depended upon by NUTCH-1477 [ NUTCH-1477 ]
          Gavin made changes -
          Link This issue blocks NUTCH-1477 [ NUTCH-1477 ]
          Renato Javier Marroquín Mogrovejo made changes -
          Link This issue is blocked by GORA-223 [ GORA-223 ]
          Hide
          Hudson added a comment -

          Integrated in GORA_174 #61 (See https://builds.apache.org/job/GORA_174/61/)
          Committing GORA-206. Step towards putting an end to GORA-174 (Revision 1477735)

          Result = UNSTABLE
          rmarroquin : http://svn.apache.org/viewvc/gora/branches/GORA_174/?view=rev&rev=1477735
          Files :

          • /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraColumn.java
          • /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraResult.java
          • /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraRow.java
          • /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraSubColumn.java
          • /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/serializers/GoraSerializerTypeInferer.java
          • /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraClient.java
          • /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMapping.java
          • /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMappingManager.java
          • /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java
          • /gora/branches/GORA_174/gora-cassandra/src/test/conf/gora-cassandra-mapping.xml
          • /gora/branches/GORA_174/gora-cassandra/src/test/java/org/apache/gora/cassandra/store/TestCassandraStore.java
          • /gora/branches/GORA_174/gora-core/src/examples/avro/employee.json
          Show
          Hudson added a comment - Integrated in GORA_174 #61 (See https://builds.apache.org/job/GORA_174/61/ ) Committing GORA-206 . Step towards putting an end to GORA-174 (Revision 1477735) Result = UNSTABLE rmarroquin : http://svn.apache.org/viewvc/gora/branches/GORA_174/?view=rev&rev=1477735 Files : /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraColumn.java /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraResult.java /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraRow.java /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/query/CassandraSubColumn.java /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/serializers/GoraSerializerTypeInferer.java /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraClient.java /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMapping.java /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMappingManager.java /gora/branches/GORA_174/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java /gora/branches/GORA_174/gora-cassandra/src/test/conf/gora-cassandra-mapping.xml /gora/branches/GORA_174/gora-cassandra/src/test/java/org/apache/gora/cassandra/store/TestCassandraStore.java /gora/branches/GORA_174/gora-core/src/examples/avro/employee.json
          Alfonso Nishikawa made changes -
          Attachment GORA-174_over_r1462580.patch [ 12576622 ]
          Hide
          Alfonso Nishikawa added a comment -

          Updated DataStoreTestUtil#testGetEmployeeWithFields() to test only on column subsets with values. This will a subject to be reviewed later (get() returning nulls or intances).
          The test expects a null when querying for "not set values", the actual behavior.

          Show
          Alfonso Nishikawa added a comment - Updated DataStoreTestUtil#testGetEmployeeWithFields() to test only on column subsets with values. This will a subject to be reviewed later (get() returning nulls or intances). The test expects a null when querying for "not set values", the actual behavior.
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Thanks Alfonso for pushing this through!

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Thanks Alfonso for pushing this through!
          Hide
          Henry Saputra added a comment -

          Thanks for all the hard work Alfonso

          Show
          Henry Saputra added a comment - Thanks for all the hard work Alfonso
          Hide
          Alfonso Nishikawa added a comment - - edited

          Hi, Lewis! Thank you very much for your comments =)
          My fault. I updated both patches to be applied to /trunk #1457365, but from now I will put an incremental since the previous revision (including merges from trunk). Is it ok? Sorry and thank you very much for telling. The way you tell, it will be much easier to prepare the patch! (for this was a bit pain)

          Show
          Alfonso Nishikawa added a comment - - edited Hi, Lewis! Thank you very much for your comments =) My fault. I updated both patches to be applied to /trunk #1457365, but from now I will put an incremental since the previous revision (including merges from trunk). Is it ok? Sorry and thank you very much for telling. The way you tell, it will be much easier to prepare the patch! (for this was a bit pain)
          Hide
          Lewis John McGibbney added a comment -

          As this is committed it is not so much of an issue just now. Its just possibly something for consideration in the future. Given that we need to track provenance of the codebase, and these issues (and attached patches) are critical to that process.
          Thank you so much Alfonso.

          Show
          Lewis John McGibbney added a comment - As this is committed it is not so much of an issue just now. Its just possibly something for consideration in the future. Given that we need to track provenance of the codebase, and these issues (and attached patches) are critical to that process. Thank you so much Alfonso.
          Hide
          Lewis John McGibbney added a comment -

          This Alfonso. Great work on the patches. I wish I earned as much as the boss
          I think the full patches are pretty difficult to absorb given that not all of the patch will be integrated when the commit is made.
          Is it possible for patches (which for these issues will be applied against GORA_174 branch) to only include what you intend to commit for one particular commit as oppose to the entire proposal in one patch.
          For example, if I am not mistaken the new patches for this issue and GORA-207 both include the entire issue patch as oppose to just the parts you wish to add on for each commit.
          Does this make sense? I hope I have explained properly.

          Show
          Lewis John McGibbney added a comment - This Alfonso. Great work on the patches. I wish I earned as much as the boss I think the full patches are pretty difficult to absorb given that not all of the patch will be integrated when the commit is made. Is it possible for patches (which for these issues will be applied against GORA_174 branch) to only include what you intend to commit for one particular commit as oppose to the entire proposal in one patch. For example, if I am not mistaken the new patches for this issue and GORA-207 both include the entire issue patch as oppose to just the parts you wish to add on for each commit. Does this make sense? I hope I have explained properly.
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12574916 ]
          Hide
          Alfonso Nishikawa added a comment - - edited

          Updated with new tests for testing recursive schemas, nested schemas and three-types unions.

          The new tests are:

          DataStoreTestBase#testGetRecusive()
          DataStoreTestBase#testGetDoubleRecursive()
          DataStoreTestBase#testGetNested()
          DataStoreTestBase#testGet3UnionField()

          Commited revision #r1459595 to /branches/GORA_174

          Show
          Alfonso Nishikawa added a comment - - edited Updated with new tests for testing recursive schemas, nested schemas and three-types unions. The new tests are: DataStoreTestBase#testGetRecusive() DataStoreTestBase#testGetDoubleRecursive() DataStoreTestBase#testGetNested() DataStoreTestBase#testGet3UnionField() Commited revision #r1459595 to /branches/GORA_174
          Hide
          Lewis John McGibbney added a comment -

          No bother. I was looking at the avro schemas and just though I would double check. I suppose it comes down to a determination of when you axctually need to use Unions in the fields anyway?

          Show
          Lewis John McGibbney added a comment - No bother. I was looking at the avro schemas and just though I would double check. I suppose it comes down to a determination of when you axctually need to use Unions in the fields anyway?
          Hide
          Alfonso Nishikawa added a comment -

          Hi, Lewis. I checked the code and there's nothing dependent on unions in the tutorial, so nothing should fail. I will have to look how to execute the tutorial... never did before

          Show
          Alfonso Nishikawa added a comment - Hi, Lewis. I checked the code and there's nothing dependent on unions in the tutorial, so nothing should fail. I will have to look how to execute the tutorial... never did before
          Hide
          Lewis John McGibbney added a comment - - edited

          Nice work Alfonso.
          Should there be any changes made to the gora-tutorial module as well?
          We don't want this module falling behind as it is critical to lowering the barrier for newb's

          Show
          Lewis John McGibbney added a comment - - edited Nice work Alfonso. Should there be any changes made to the gora-tutorial module as well? We don't want this module falling behind as it is critical to lowering the barrier for newb's
          Hide
          Hudson added a comment -

          Integrated in GORA_174 #5 (See https://builds.apache.org/job/GORA_174/5/)
          Applied working patch GORA-174 [0] version 12/Mar/13 17:55

          [0] https://issues.apache.org/jira/browse/GORA-174 (Revision 1455623)

          Result = UNSTABLE
          alfonsonishikawa : http://svn.apache.org/viewvc/gora/branches/GORA_174/?view=rev&rev=1455623
          Files :

          • /gora/branches/GORA_174/gora-core/src/examples/avro/webpage.json
          • /gora/branches/GORA_174/gora-core/src/examples/java/org/apache/gora/examples/WebPageDataCreator.java
          • /gora/branches/GORA_174/gora-core/src/examples/java/org/apache/gora/examples/generated/Employee.java
          • /gora/branches/GORA_174/gora-core/src/examples/java/org/apache/gora/examples/generated/Metadata.java
          • /gora/branches/GORA_174/gora-core/src/examples/java/org/apache/gora/examples/generated/TokenDatum.java
          • /gora/branches/GORA_174/gora-core/src/examples/java/org/apache/gora/examples/generated/WebPage.java
          • /gora/branches/GORA_174/gora-core/src/examples/java/org/apache/gora/examples/mapreduce/WordCount.java
          • /gora/branches/GORA_174/gora-core/src/main/java/org/apache/gora/compiler/GoraCompiler.java
          • /gora/branches/GORA_174/gora-core/src/main/java/org/apache/gora/persistency/impl/PersistentBase.java
          • /gora/branches/GORA_174/gora-core/src/test/java/org/apache/gora/mapreduce/MapReduceTestUtils.java
          • /gora/branches/GORA_174/gora-core/src/test/java/org/apache/gora/store/DataStoreTestUtil.java
          Show
          Hudson added a comment - Integrated in GORA_174 #5 (See https://builds.apache.org/job/GORA_174/5/ ) Applied working patch GORA-174 [0] version 12/Mar/13 17:55 [0] https://issues.apache.org/jira/browse/GORA-174 (Revision 1455623) Result = UNSTABLE alfonsonishikawa : http://svn.apache.org/viewvc/gora/branches/GORA_174/?view=rev&rev=1455623 Files : /gora/branches/GORA_174/gora-core/src/examples/avro/webpage.json /gora/branches/GORA_174/gora-core/src/examples/java/org/apache/gora/examples/WebPageDataCreator.java /gora/branches/GORA_174/gora-core/src/examples/java/org/apache/gora/examples/generated/Employee.java /gora/branches/GORA_174/gora-core/src/examples/java/org/apache/gora/examples/generated/Metadata.java /gora/branches/GORA_174/gora-core/src/examples/java/org/apache/gora/examples/generated/TokenDatum.java /gora/branches/GORA_174/gora-core/src/examples/java/org/apache/gora/examples/generated/WebPage.java /gora/branches/GORA_174/gora-core/src/examples/java/org/apache/gora/examples/mapreduce/WordCount.java /gora/branches/GORA_174/gora-core/src/main/java/org/apache/gora/compiler/GoraCompiler.java /gora/branches/GORA_174/gora-core/src/main/java/org/apache/gora/persistency/impl/PersistentBase.java /gora/branches/GORA_174/gora-core/src/test/java/org/apache/gora/mapreduce/MapReduceTestUtils.java /gora/branches/GORA_174/gora-core/src/test/java/org/apache/gora/store/DataStoreTestUtil.java
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12573366 ]
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12573346 ]
          Hide
          Lewis John McGibbney added a comment -

          I'm +1 for this to be committed to the GORA_174 branch, either when you add your fix for the 2 bugs or just now.

          Show
          Lewis John McGibbney added a comment - I'm +1 for this to be committed to the GORA_174 branch, either when you add your fix for the 2 bugs or just now.
          Alfonso Nishikawa made changes -
          Attachment GORA-174v3.patch [ 12554771 ]
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12573124 ]
          Hide
          Alfonso Nishikawa added a comment -

          Working patch upgraded for /trunk (mar 11, 2013). Uploaded again because former was wrong.

          There are at least 2 bugs still not fixed (but not critic), working on them.

          Show
          Alfonso Nishikawa added a comment - Working patch upgraded for /trunk (mar 11, 2013). Uploaded again because former was wrong. There are at least 2 bugs still not fixed (but not critic), working on them.
          Alfonso Nishikawa made changes -
          Comment [ Working patch upgraded to /trunk (mar 11, 2013). ]
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12573123 ]
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12573123 ]
          Alfonso Nishikawa made changes -
          Attachment failed_tests_after_v3.tar.gz [ 12555788 ]
          Alfonso Nishikawa made changes -
          Attachment GORA-174v2.patch [ 12554240 ]
          Hide
          Lewis John McGibbney added a comment -

          Nice on Alfonso.

          Show
          Lewis John McGibbney added a comment - Nice on Alfonso.
          Hide
          Alfonso Nishikawa added a comment - - edited

          Tomorrow I will upload a patch based on /trunk (I forgot it at office) for this issue, and a patch for GORA-207.
          I found more bugs related with this issue, but it is ok to apply tomorrow's patch and create that specific patches in GORA-206
          Thank you!

          Show
          Alfonso Nishikawa added a comment - - edited Tomorrow I will upload a patch based on /trunk (I forgot it at office) for this issue, and a patch for GORA-207 . I found more bugs related with this issue, but it is ok to apply tomorrow's patch and create that specific patches in GORA-206 Thank you!
          Hide
          Lewis John McGibbney added a comment -

          I Alfonso do you wish to apply the current v3 patch to the GORA_174 branch? Although this will break the branch, it means that we can build datastore specific patches on the sub tasks e.g. GORA-206 and GORA-207

          Right now the patch on GORA-206 incorporates the entire patch here, which really confuses the picture of what that issue it trying to address... it is Cassandra specific, whereas this issue is Gora compiler (which has essentially been fixed), we can then open a new issue for the failing test in o.a.g.c.peristent.PersistentSerilization.

          What are your thoughts?

          Show
          Lewis John McGibbney added a comment - I Alfonso do you wish to apply the current v3 patch to the GORA_174 branch? Although this will break the branch, it means that we can build datastore specific patches on the sub tasks e.g. GORA-206 and GORA-207 Right now the patch on GORA-206 incorporates the entire patch here, which really confuses the picture of what that issue it trying to address... it is Cassandra specific, whereas this issue is Gora compiler (which has essentially been fixed), we can then open a new issue for the failing test in o.a.g.c.peristent.PersistentSerilization. What are your thoughts?
          Alfonso Nishikawa made changes -
          Link This issue blocks GORA-109 [ GORA-109 ]
          Alfonso Nishikawa made changes -
          Link This issue blocks GORA-109 [ GORA-109 ]
          Lewis John McGibbney made changes -
          Priority Major [ 3 ] Blocker [ 1 ]
          Hide
          Alfonso Nishikawa added a comment -

          Hi, Renato!
          So sorry! I have been much idle reading the overwhelming Apache documentation and learning git. I will continue with this issue Still needs more work. I have seen Cassandra fails too because is not prepared for unions. I will spend my efforts in this issue
          Sorry for the delay

          Show
          Alfonso Nishikawa added a comment - Hi, Renato! So sorry! I have been much idle reading the overwhelming Apache documentation and learning git. I will continue with this issue Still needs more work. I have seen Cassandra fails too because is not prepared for unions. I will spend my efforts in this issue Sorry for the delay
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Hey guys, have you decided on how to finish testing this patch so we can put it into trunk?

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Hey guys, have you decided on how to finish testing this patch so we can put it into trunk?
          Alfonso Nishikawa made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          Lewis John McGibbney added a comment -

          OK I agree. I am happy about the additional work to the Gora compiler, however you are correct in stating that we need to ensure the repercussions are dealt with elsewhere before integrating into trunk. If you wish to open a new issue(s), please feel free to do so.

          Show
          Lewis John McGibbney added a comment - OK I agree. I am happy about the additional work to the Gora compiler, however you are correct in stating that we need to ensure the repercussions are dealt with elsewhere before integrating into trunk. If you wish to open a new issue(s), please feel free to do so.
          Hide
          Alfonso Nishikawa added a comment -

          Hi, Lewis.
          Still should not be committed since the inclusion of union is still not fully supported by HBase nor Cassandra nor Core (different tests problems).
          Commiting now will make tests fails, so first have to upload a new patch. Al this moment I think all tests passes in my local copy but still not executed all tests at once takes time. I am sure there are hidden bugs because I saw some "switch()" not implementing "case UNION".
          I could upload tomorrow morning a patch for this issue passing tests and we should open another issue to check all related to unions :\ (serializing, deserializing, setting dirty, etc). I don't vote for closing and submitting this half patch :\ but if you think it's better to do so and open another issue, for me is perfect at least issue will be clear

          Show
          Alfonso Nishikawa added a comment - Hi, Lewis. Still should not be committed since the inclusion of union is still not fully supported by HBase nor Cassandra nor Core (different tests problems). Commiting now will make tests fails, so first have to upload a new patch. Al this moment I think all tests passes in my local copy but still not executed all tests at once takes time . I am sure there are hidden bugs because I saw some "switch()" not implementing "case UNION". I could upload tomorrow morning a patch for this issue passing tests and we should open another issue to check all related to unions :\ (serializing, deserializing, setting dirty, etc). I don't vote for closing and submitting this half patch :\ but if you think it's better to do so and open another issue, for me is perfect at least issue will be clear
          Hide
          Lewis John McGibbney added a comment -

          I think the fix for this can be committed. Although I am running into different issues (as commented over in NUTCH-1477) GoraCompiler with the GORA-174v3.patch certainly generates the Java classes properly now with getters and setters.
          We have a problem with the tests though, is this correct? I see your attachment Alfonso.

          Show
          Lewis John McGibbney added a comment - I think the fix for this can be committed. Although I am running into different issues (as commented over in NUTCH-1477 ) GoraCompiler with the GORA-174 v3.patch certainly generates the Java classes properly now with getters and setters. We have a problem with the tests though, is this correct? I see your attachment Alfonso.
          Hide
          Alfonso Nishikawa added a comment - - edited

          Hi, Lewis.
          I noticed now that you have to compile the file webpage.avsc shown in NUTCH-1477, not the one from Nutch branch 2.x.
          Every optional field of type "string" and "bytes" type must be an union of ["null", ...].
          Can you check with that file? I have an updated fail-test-free (I think I found test-proof bugs) on my computer but I wish to check other things of the code and there is nothing related to DataFileAvroStore.
          Thanks!

          Show
          Alfonso Nishikawa added a comment - - edited Hi, Lewis. I noticed now that you have to compile the file webpage.avsc shown in NUTCH-1477 , not the one from Nutch branch 2.x. Every optional field of type "string" and "bytes" type must be an union of ["null", ...] . Can you check with that file? I have an updated fail-test-free (I think I found test-proof bugs) on my computer but I wish to check other things of the code and there is nothing related to DataFileAvroStore. Thanks!
          Hide
          Lewis John McGibbney added a comment -

          I can confirm that after applying the patch to gora-core then using this dependency with Nutch 2.x, the Exception is identical as the one documented in NUTCH-1477. When I compile the webpage avsc's here [0], the webpage class is generated but with no accessors for the fields in with the union case we are concerned with. I am also learning with the GoraCompiler (and Avro stuff) so I do not have a definitive solution to hand just now. I think the serializing and deserializing problems which seem to be a result of introducing union support should be addressed as we encounter them.

          [0] http://svn.apache.org/repos/asf/nutch/branches/2.x/src/gora/

          Show
          Lewis John McGibbney added a comment - I can confirm that after applying the patch to gora-core then using this dependency with Nutch 2.x, the Exception is identical as the one documented in NUTCH-1477 . When I compile the webpage avsc's here [0] , the webpage class is generated but with no accessors for the fields in with the union case we are concerned with. I am also learning with the GoraCompiler (and Avro stuff) so I do not have a definitive solution to hand just now. I think the serializing and deserializing problems which seem to be a result of introducing union support should be addressed as we encounter them. [0] http://svn.apache.org/repos/asf/nutch/branches/2.x/src/gora/
          Alfonso Nishikawa made changes -
          Link This issue contains GORA-188 [ GORA-188 ]
          Hide
          Alfonso Nishikawa added a comment -

          Hi, Lewis.
          Thank you for trying it. Does it give the same exception as in NUTCH-1477?

          Adding union support in the compiler seems to affect serializing and deserializing too (and others - added failed tests in attachments).

          As you told in other thread, this seems to become a pain. I subscribe
          I am working on this issue and will be working all this week Suggestions much accepted and looking forward them.

          Show
          Alfonso Nishikawa added a comment - Hi, Lewis. Thank you for trying it. Does it give the same exception as in NUTCH-1477 ? Adding union support in the compiler seems to affect serializing and deserializing too (and others - added failed tests in attachments). As you told in other thread, this seems to become a pain. I subscribe I am working on this issue and will be working all this week Suggestions much accepted and looking forward them.
          Hide
          Lewis John McGibbney added a comment -

          Hi Alfonso, at the beginning of the weekend I spent time working between the patched GoraCompiler and Nutch 2.x. AS Julien suggested in his initial problem description "however when generating the code for the schema the accessors are not generated by GORA which prevents Nutch from compiling", this still seems to be the case even when using the GORA-174v3.patch so I am not sure that the patch is properly fixing this issue.

          Show
          Lewis John McGibbney added a comment - Hi Alfonso, at the beginning of the weekend I spent time working between the patched GoraCompiler and Nutch 2.x. AS Julien suggested in his initial problem description "however when generating the code for the schema the accessors are not generated by GORA which prevents Nutch from compiling", this still seems to be the case even when using the GORA-174 v3.patch so I am not sure that the patch is properly fixing this issue.
          Alfonso Nishikawa made changes -
          Link This issue is related to GORA-183 [ GORA-183 ]
          Alfonso Nishikawa made changes -
          Link This issue is duplicated by GORA-183 [ GORA-183 ]
          Hide
          Henry Saputra added a comment -

          +1 for closing GORA-183 as dup of GORA-174

          Looks like the fix for GORA-174 should be able to resolve the issue. Alfonso, if thats correct please close GORA-183 as dup

          Show
          Henry Saputra added a comment - +1 for closing GORA-183 as dup of GORA-174 Looks like the fix for GORA-174 should be able to resolve the issue. Alfonso, if thats correct please close GORA-183 as dup
          Alfonso Nishikawa made changes -
          Attachment failed_tests_after_v3.tar.gz [ 12555788 ]
          Hide
          Alfonso Nishikawa added a comment -

          Uploaded failed tests after applying GORA-174v3.patch

          Show
          Alfonso Nishikawa added a comment - Uploaded failed tests after applying GORA-174 v3.patch
          Hide
          Alfonso Nishikawa added a comment -

          Checked all tests. "mvn test" shows all passed, but checking gora-xxx/target/surefire-reports/ realized there where the following failed tests:

          • Gora-core: org.apache.gora.mapreduce.TestPersistentSerialization - testSerdeWebPage. GORA-188
          • Gora-accumulo: org.apache.gora.accumulo.store.AccumuloStoreTest - testUpdate. java.lang.IllegalArgumentException: Unknown type UNION and testing classes related with "null" values.
          • Gora-cassanda: NPE related with "null" values.
          • Gora-hbase: [java.lang.IllegalArgumentException: Unknown type UNION, NPE, ... shown in GORA-183

          Maybe we should close GORA-183 and solve it here.

          Show
          Alfonso Nishikawa added a comment - Checked all tests. "mvn test" shows all passed, but checking gora-xxx/target/surefire-reports/ realized there where the following failed tests: Gora-core: org.apache.gora.mapreduce.TestPersistentSerialization - testSerdeWebPage. GORA-188 Gora-accumulo: org.apache.gora.accumulo.store.AccumuloStoreTest - testUpdate. java.lang.IllegalArgumentException: Unknown type UNION and testing classes related with "null" values. Gora-cassanda: NPE related with "null" values. Gora-hbase: [java.lang.IllegalArgumentException: Unknown type UNION, NPE, ... shown in GORA-183 Maybe we should close GORA-183 and solve it here.
          Alfonso Nishikawa made changes -
          Attachment GORA-174v3.patch [ 12554771 ]
          Hide
          Alfonso Nishikawa added a comment -

          ========= patch released =============
          GORA-174v3.patch

          Released for /trunk although not much differences.

          Show
          Alfonso Nishikawa added a comment - ========= patch released ============= GORA-174 v3.patch Released for /trunk although not much differences.
          Hide
          Alfonso Nishikawa added a comment -

          Patch is for gora 0.2.1 . Will adapt it to /trunk and upload again.

          Show
          Alfonso Nishikawa added a comment - Patch is for gora 0.2.1 . Will adapt it to /trunk and upload again.
          Gavin made changes -
          Workflow jira [ 12731627 ] no-reopen-closed, patch-avail [ 12735388 ]
          Alfonso Nishikawa made changes -
          Link This issue is required by GORA-189 [ GORA-189 ]
          Hide
          Alfonso Nishikawa added a comment -

          Seems all right Thanks!! Hope it works with Nutch.

          Show
          Alfonso Nishikawa added a comment - Seems all right Thanks!! Hope it works with Nutch.
          Hide
          Alfonso Nishikawa added a comment - - edited

          – deleted

          Show
          Alfonso Nishikawa added a comment - - edited – deleted
          Alfonso Nishikawa made changes -
          Comment [ Thanks :) hope it works! ]
          Hide
          Lewis John McGibbney added a comment -

          Hi, I'll try the new patch both with gora trunk and with the Nutch 2.x InjectorJob.

          Show
          Lewis John McGibbney added a comment - Hi, I'll try the new patch both with gora trunk and with the Nutch 2.x InjectorJob.
          Alfonso Nishikawa made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          Alfonso Nishikawa made changes -
          Attachment GORA-174v2.patch [ 12554240 ]
          Hide
          Alfonso Nishikawa added a comment -

          ============================= PATCH AVAILABLE ========================
          GORA-174v2.patch

          Includes @Julien's GORA-174-test.patch

          Show
          Alfonso Nishikawa added a comment - ============================= PATCH AVAILABLE ======================== GORA-174 v2.patch Includes @Julien's GORA-174 -test.patch
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12554158 ]
          Alfonso Nishikawa made changes -
          Comment [ Output of mvn clean test uploaded (patch-tests_applied). Only applied "GORA-174-test.patch". ]
          Alfonso Nishikawa made changes -
          Attachment patch-tests_applied.out.tgz [ 12554190 ]
          Hide
          Alfonso Nishikawa added a comment -

          @Julien: You are right, now seems to (not)work in Eclipse as you say. Thanks.
          Actually what your test shows is that a mandatory field fails if you write a null. Otherwise would be optional. At the same time will show that an optional field is optional, but that is not something 100% pedendent on the compiler But many many thanks! I was getting mad with this tests.
          I will finish fixing the tests, since there are other places where CONTENT[i] (=null) is being accessed as an array.

          Show
          Alfonso Nishikawa added a comment - @Julien: You are right, now seems to (not)work in Eclipse as you say. Thanks. Actually what your test shows is that a mandatory field fails if you write a null. Otherwise would be optional. At the same time will show that an optional field is optional, but that is not something 100% pedendent on the compiler But many many thanks! I was getting mad with this tests. I will finish fixing the tests, since there are other places where CONTENT [i] (=null) is being accessed as an array.
          Hide
          Julien Nioche added a comment -

          I just took a fresh copy from the GIT repo, applied the patch GORA-174-test.patch, cd gora-core and called 'mvn clean test' and I getting

          Failed tests: testQuery(org.apache.gora.avro.store.TestAvroStore): expected:<0> but was:<10>
          testQueryBinaryEncoder(org.apache.gora.avro.store.TestAvroStore): expected:<0> but was:<10>
          testQuery(org.apache.gora.avro.store.TestDataFileAvroStore): expected:<0> but was:<10>
          testQueryBinaryEncoder(org.apache.gora.avro.store.TestDataFileAvroStore): expected:<0> but was:<10>

          Tests in error:
          testCountQuery(org.apache.gora.avro.mapreduce.TestDataFileAvroStoreMapReduce)

          Running the tests without applying the patch GORA-174-test.patch works fine.

          Show
          Julien Nioche added a comment - I just took a fresh copy from the GIT repo, applied the patch GORA-174 -test.patch, cd gora-core and called 'mvn clean test' and I getting Failed tests: testQuery(org.apache.gora.avro.store.TestAvroStore): expected:<0> but was:<10> testQueryBinaryEncoder(org.apache.gora.avro.store.TestAvroStore): expected:<0> but was:<10> testQuery(org.apache.gora.avro.store.TestDataFileAvroStore): expected:<0> but was:<10> testQueryBinaryEncoder(org.apache.gora.avro.store.TestDataFileAvroStore): expected:<0> but was:<10> Tests in error: testCountQuery(org.apache.gora.avro.mapreduce.TestDataFileAvroStoreMapReduce) Running the tests without applying the patch GORA-174 -test.patch works fine.
          Hide
          Alfonso Nishikawa added a comment -

          @Julien: after analyzing your test classes patch, I think they work as expected: before your changes, the class WebPage exists, and method WebPage#setContent() too. After your patch, without regenerating the class WebPage, the method WebPage#setContent() too, so WebPage#setContent(null) will not fail. DataFileAvroStore success writing a null. No problem.

          To make it fail must: modify webpage.json, regenerate WebPage, and will not compile because the method WebPage#setContent() will not exist.

          Show
          Alfonso Nishikawa added a comment - @Julien: after analyzing your test classes patch, I think they work as expected: before your changes, the class WebPage exists, and method WebPage#setContent() too. After your patch, without regenerating the class WebPage, the method WebPage#setContent() too, so WebPage#setContent(null) will not fail. DataFileAvroStore success writing a null. No problem. To make it fail must: modify webpage.json, regenerate WebPage, and will not compile because the method WebPage#setContent() will not exist.
          Hide
          Alfonso Nishikawa added a comment - - edited

          @Henry: yes, that is

          GORA-174.patch is the fix, while GORA-174-test.patch is the patch for JUnit uploaded by Julien. Now I am trying to understand the JUnit part of the project (takes me some time )

          Show
          Alfonso Nishikawa added a comment - - edited @Henry: yes, that is GORA-174 .patch is the fix, while GORA-174 -test.patch is the patch for JUnit uploaded by Julien. Now I am trying to understand the JUnit part of the project (takes me some time )
          Hide
          Henry Saputra added a comment -

          So GORA-174-test.patch file is the right patch to try?

          Show
          Henry Saputra added a comment - So GORA-174 -test.patch file is the right patch to try?
          Alfonso Nishikawa made changes -
          Attachment patch-tests_applied.out.tgz [ 12554190 ]
          Hide
          Julien Nioche added a comment -

          No probs. Even if you apply both patches the mvn tests will still fail - hence my comments about the code generation above.

          Show
          Julien Nioche added a comment - No probs. Even if you apply both patches the mvn tests will still fail - hence my comments about the code generation above.
          Hide
          Alfonso Nishikawa added a comment - - edited

          Oh, oh, sorry applied both patches
          I'm doing it wrong. Will try it again. Maybe I generated the classes by command. I am with other things too and don't remember :'(

          Show
          Alfonso Nishikawa added a comment - - edited Oh, oh, sorry applied both patches I'm doing it wrong. Will try it again. Maybe I generated the classes by command. I am with other things too and don't remember :'(
          Hide
          Julien Nioche added a comment -

          I don't know why it does not fail with Eclipse but the testing should be done with 'mvn clean test' and in this particular case we DO want the tests to fail as a proof that we need your patch and that it works.

          Would be great to clarify what generates the Java code from the AVRO schemas in gora-core/src/examples/avro and when. Thoughts anyone?

          Show
          Julien Nioche added a comment - I don't know why it does not fail with Eclipse but the testing should be done with 'mvn clean test' and in this particular case we DO want the tests to fail as a proof that we need your patch and that it works. Would be great to clarify what generates the Java code from the AVRO schemas in gora-core/src/examples/avro and when. Thoughts anyone?
          Hide
          Alfonso Nishikawa added a comment -

          I applied your patch and passed tests of JUnit in Eclipse (org.apache.gora.avro.store.TestDataFileAvroStore)

          Show
          Alfonso Nishikawa added a comment - I applied your patch and passed tests of JUnit in Eclipse (org.apache.gora.avro.store.TestDataFileAvroStore)
          Hide
          Julien Nioche added a comment -

          Do you mean that it does not fail when calling 'mvn clean test'? If so did you regenerate the java code from the modified AVRO schema manually?

          Show
          Julien Nioche added a comment - Do you mean that it does not fail when calling 'mvn clean test'? If so did you regenerate the java code from the modified AVRO schema manually?
          Hide
          Alfonso Nishikawa added a comment -

          Thanks for the tests patch.
          In my source copy (downloaded .tgz), this line of the patch:

          @@ -116,9 +116,11 @@ public class WebPageDataCreator {
          

          should be

          @@ -115,9 +115,11 @@ public class WebPageDataCreator {
          

          but patched successfuly with "patch -p0 -l -i GORA-174-test.patch".
          Seems to work for me.

          Show
          Alfonso Nishikawa added a comment - Thanks for the tests patch. In my source copy (downloaded .tgz), this line of the patch: @@ -116,9 +116,11 @@ public class WebPageDataCreator { should be @@ -115,9 +115,11 @@ public class WebPageDataCreator { but patched successfuly with "patch -p0 -l -i GORA-174 -test.patch". Seems to work for me.
          Hide
          Julien Nioche added a comment -

          Will test your patch with the Nutch schema

          no comment means that I have not had the time to test it

          you can use the patch I submitted earlier to reproduce the issue yourself within GORA independently of Nutch

          Show
          Julien Nioche added a comment - Will test your patch with the Nutch schema no comment means that I have not had the time to test it you can use the patch I submitted earlier to reproduce the issue yourself within GORA independently of Nutch
          Hide
          Alfonso Nishikawa added a comment -

          Ah, ok, thanks for the information DataFileAvroStore, didn't notice.

          Show
          Alfonso Nishikawa added a comment - Ah, ok, thanks for the information DataFileAvroStore, didn't notice.
          Hide
          Alfonso Nishikawa added a comment -

          Does it work? Doesn't it work? Now still gives the same exception?

          Show
          Alfonso Nishikawa added a comment - Does it work? Doesn't it work? Now still gives the same exception?
          Hide
          Julien Nioche added a comment -

          DataFileAvroStore - see description of this issue and of NUTCH-1477

          Show
          Julien Nioche added a comment - DataFileAvroStore - see description of this issue and of NUTCH-1477
          Alfonso Nishikawa made changes -
          Link This issue is related to GORA-183 [ GORA-183 ]
          Hide
          Alfonso Nishikawa added a comment -

          Remember that in HBase will not work (issue GORA-183). What backend are you using?

          Show
          Alfonso Nishikawa added a comment - Remember that in HBase will not work (issue GORA-183 ). What backend are you using?
          Julien Nioche made changes -
          Attachment GORA-174-test.patch [ 12554160 ]
          Hide
          Julien Nioche added a comment -

          Will test your patch with the Nutch schema. In the meantime here is a patch for the tests that currently crashes when we add a null value to the schema. For some reason the java code is not generated prior to running the tests. Hopefully this test should pass after fixing this issue.

          Show
          Julien Nioche added a comment - Will test your patch with the Nutch schema. In the meantime here is a patch for the tests that currently crashes when we add a null value to the schema. For some reason the java code is not generated prior to running the tests. Hopefully this test should pass after fixing this issue.
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12554158 ]
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12554157 ]
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12554157 ]
          Hide
          Alfonso Nishikawa added a comment -

          Patch uploaded. @Julien: try it and tell if it works for you.

          Show
          Alfonso Nishikawa added a comment - Patch uploaded. @Julien: try it and tell if it works for you.
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12554139 ]
          Hide
          Alfonso Nishikawa added a comment -

          Hi, Julien. Hold a second. Found some minor bug that stops compiling.

          Show
          Alfonso Nishikawa added a comment - Hi, Julien. Hold a second. Found some minor bug that stops compiling.
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12554139 ]
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12554088 ]
          Alfonso Nishikawa made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Alfonso Nishikawa made changes -
          Assignee Alfonso Nishikawa [ alfonso.nishikawa ]
          Alfonso Nishikawa made changes -
          Comment [ Generate getters/setters for union types. ]
          Hide
          Alfonso Nishikawa added a comment - - edited

          Hi, Julien.
          Try the patch and tell me if it solves your problem (I tested, but who knows...).
          Sadly, you will get the exception in GORA-183. Will try to fix it soon.
          --edited: GORA-183 affects HBase backend. What are you using?

          Show
          Alfonso Nishikawa added a comment - - edited Hi, Julien. Try the patch and tell me if it solves your problem (I tested, but who knows...). Sadly, you will get the exception in GORA-183 . Will try to fix it soon. --edited: GORA-183 affects HBase backend. What are you using?
          Alfonso Nishikawa made changes -
          Attachment GORA-174.patch [ 12554088 ]
          Lewis John McGibbney made changes -
          Fix Version/s 0.3 [ 12317954 ]
          Hide
          Julien Nioche added a comment -

          anyone left in the project who understands the intricacies of the compiler?

          Show
          Julien Nioche added a comment - anyone left in the project who understands the intricacies of the compiler?
          Julien Nioche made changes -
          Field Original Value New Value
          Link This issue blocks NUTCH-1477 [ NUTCH-1477 ]
          Julien Nioche created issue -

            People

            • Assignee:
              Alfonso Nishikawa
              Reporter:
              Julien Nioche
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development