Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.2
    • Fix Version/s: 0.4
    • Component/s: build process, storage
    • Labels:
      None

      Description

      I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work...

      1. GORA-94v9.patch
        40 kB
        Lewis John McGibbney
      2. GORA-94v8.patch
        77 kB
        yasin tamer
      3. GORA-94v7.patch
        48 kB
        Lewis John McGibbney
      4. GORA-94v6.diff
        6 kB
        yasin tamer
      5. GORA-94-v4.patch
        395 kB
        Lewis John McGibbney
      6. GORA-94-v3.patch
        406 kB
        Ed Kohlwey
      7. GORA-94-v2.patch
        13 kB
        Lewis John McGibbney
      8. GORA-94v12.patch
        39 kB
        Alparslan Avcı
      9. GORA-94v11.patch
        12 kB
        Alparslan Avcı
      10. GORA-94v10.patch
        21 kB
        Alparslan Avcı
      11. GORA-94.patch
        10 kB
        Lewis John McGibbney
      12. GORA-94.DataStoreTestUtil.patch
        3 kB
        Renato Javier Marroquín Mogrovejo
      13. GORA_94v5.patch
        301 kB
        Lewis John McGibbney

        Issue Links

          Activity

          Hide
          Lewis John McGibbney added a comment -

          Ultimately merged and committed to trunk codebase @r1586888

          Show
          Lewis John McGibbney added a comment - Ultimately merged and committed to trunk codebase @r1586888
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Committed r1579573.
          Going one step at the time

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Committed r1579573. Going one step at the time
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Updated version of the test patch.

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Updated version of the test patch.
          Hide
          Alparslan Avcı added a comment -

          Hi Renato Javier Marroquín Mogrovejo,

          +1 for the commit!

          I hope we will re-think on handling nested structures and find better ways in future (escpecially for gora-0.5 ). This issue is not a problem for now. We can keep moving.

          Show
          Alparslan Avcı added a comment - Hi Renato Javier Marroquín Mogrovejo , +1 for the commit! I hope we will re-think on handling nested structures and find better ways in future (escpecially for gora-0.5 ). This issue is not a problem for now. We can keep moving.
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Hi Alparslan Avcı,

          I did try without using that question about the qualifier, and I think what you say is valid but not in the case of dealing with 2-nested level structure as the one we are working with. That check helps deciding on what needs to be deleted when the map is inside another structure. Maybe this was not intended to work this way, but I think this is the way this is working. You might want to confirm this, and then we can open a new JIRA for fixing this "unexpected behaviour" when serializing nested structures in Gora-HBase. Maybe it's better if we move this discussion over to GORA-246.
          And sure I will take those TODOs from the patch and commit

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Hi Alparslan Avcı , I did try without using that question about the qualifier, and I think what you say is valid but not in the case of dealing with 2-nested level structure as the one we are working with. That check helps deciding on what needs to be deleted when the map is inside another structure. Maybe this was not intended to work this way, but I think this is the way this is working. You might want to confirm this, and then we can open a new JIRA for fixing this "unexpected behaviour" when serializing nested structures in Gora-HBase. Maybe it's better if we move this discussion over to GORA-246 . And sure I will take those TODOs from the patch and commit
          Hide
          Alparslan Avcı added a comment -

          Thanks for the updates Renato Javier Marroquín Mogrovejo!

          Just two more comments; firstly I still think that we do not need to check if the qualifier parameter is null or not at case MAP in HBaseStore.addPutsAndDeletes() method as I mentioned in GORA-246. And would you please remove the TODO's in DataStoreTestUtil? There is no need for them anymore.

          Show
          Alparslan Avcı added a comment - Thanks for the updates Renato Javier Marroquín Mogrovejo ! Just two more comments; firstly I still think that we do not need to check if the qualifier parameter is null or not at case MAP in HBaseStore.addPutsAndDeletes() method as I mentioned in GORA-246 . And would you please remove the TODO's in DataStoreTestUtil ? There is no need for them anymore.
          Hide
          Talat UYARER added a comment -

          Hey Renato Javier Marroquín Mogrovejo,

          You are right. We create new issue for this. Actually I didnt know what should i do.

          Show
          Talat UYARER added a comment - Hey Renato Javier Marroquín Mogrovejo , You are right. We create new issue for this. Actually I didnt know what should i do.
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Hi Alparslan Avcı,

          The patch posted in GORA-246 contains just changes for gora-hbase and I have uploaded a rebased version for GORA-245. So if you could test them out, you would make my day because I could commit them without worries

          Talat UYARER Maybe we should open a different JIRA issue for this one (but add a relation with GORA-94), so we can keep on handling specific patches on specific issues. Wdyt?

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Hi Alparslan Avcı , The patch posted in GORA-246 contains just changes for gora-hbase and I have uploaded a rebased version for GORA-245 . So if you could test them out, you would make my day because I could commit them without worries Talat UYARER Maybe we should open a different JIRA issue for this one (but add a relation with GORA-94 ), so we can keep on handling specific patches on specific issues. Wdyt?
          Hide
          Talat UYARER added a comment -

          In Employee based test we set nullable ssn field. But We use ssn for rowkey. Ssn is not nullable.

          Show
          Talat UYARER added a comment - In Employee based test we set nullable ssn field. But We use ssn for rowkey. Ssn is not nullable.
          Hide
          Alparslan Avcı added a comment -

          Hi Renato Javier Marroquín Mogrovejo,

          The GORA-94.DataStoreTestUtil.patch works smootly, thanks. However, GORA-246.FailingTest.patch and GORA-245.final.patch contains changes on the same code. Would you please reorganize the patches in GORA-245 and GORA-246?

          Show
          Alparslan Avcı added a comment - Hi Renato Javier Marroquín Mogrovejo , The GORA-94.DataStoreTestUtil.patch works smootly, thanks. However, GORA-246.FailingTest.patch and GORA-245.final.patch contains changes on the same code. Would you please reorganize the patches in GORA-245 and GORA-246 ?
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Patch containing the modifications needed for testUpdateWebPageRemoveMapEntry to work as expected.
          This should be applied first, and then the other patches for gora-hbase and gora-cassandra. Please check it out, so we can decide if we commit this or not. Thanks!

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Patch containing the modifications needed for testUpdateWebPageRemoveMapEntry to work as expected. This should be applied first, and then the other patches for gora-hbase and gora-cassandra. Please check it out, so we can decide if we commit this or not. Thanks!
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Hey Talat UYARER,

          I have been looking into the failing test (testUpdateWebPageRemoveMapEntry), but it doesn't seem that the problem is with the delete methods as we are not using any. The problem seems to be in the update process of the value. Some how the new attributes being updated are not considered as new ones I will keep on investigating on this, and keep you posted.
          Thanks!

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Hey Talat UYARER , I have been looking into the failing test (testUpdateWebPageRemoveMapEntry), but it doesn't seem that the problem is with the delete methods as we are not using any. The problem seems to be in the update process of the value. Some how the new attributes being updated are not considered as new ones I will keep on investigating on this, and keep you posted. Thanks!
          Hide
          Talat UYARER added a comment - - edited

          If you are available work on this, you can work Renato Javier Marroquín Mogrovejo. I guess I will not work until 24.00 GMT+2 tonight.

          However Cassandra's problem is not simple. Clear methos is not work. Because Cassandra has not delete methods. This is main problem But If you want to learn anything I can help you.

          Show
          Talat UYARER added a comment - - edited If you are available work on this, you can work Renato Javier Marroquín Mogrovejo . I guess I will not work until 24.00 GMT+2 tonight. However Cassandra's problem is not simple. Clear methos is not work. Because Cassandra has not delete methods. This is main problem But If you want to learn anything I can help you.
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Nice Talat UYARER! So are you done with this? I was just getting ready to hack it but if not I can move on something else.
          So do you want me work on this? or do you want to finish this one man?

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Nice Talat UYARER ! So are you done with this? I was just getting ready to hack it but if not I can move on something else. So do you want me work on this? or do you want to finish this one man?
          Hide
          Talat UYARER added a comment - - edited

          You are right Alparslan Avcı. I am working on Cassandra's Test. I will be glad if you commit it Renato Javier Marroquín Mogrovejo.

          Show
          Talat UYARER added a comment - - edited You are right Alparslan Avcı . I am working on Cassandra's Test. I will be glad if you commit it Renato Javier Marroquín Mogrovejo .
          Hide
          Alparslan Avcı added a comment - - edited

          I have already prepared a patch for fix in HBaseStore, but have not uploaded yet. If it is ok for you, you can send your changes in DataStoreTestUtil onto this issue. And after your patch upload, I will upload mine to GORA-246. AFAIK, Talat UYARER is working for Cassandra to fix this problem.

          Show
          Alparslan Avcı added a comment - - edited I have already prepared a patch for fix in HBaseStore, but have not uploaded yet. If it is ok for you, you can send your changes in DataStoreTestUtil onto this issue. And after your patch upload, I will upload mine to GORA-246 . AFAIK, Talat UYARER is working for Cassandra to fix this problem.
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          I updated that and we still have a test failure in CassandraStore and in HBaseStore. I will look into this and commit if I fix them (:

          Show
          Renato Javier Marroquín Mogrovejo added a comment - I updated that and we still have a test failure in CassandraStore and in HBaseStore. I will look into this and commit if I fix them (:
          Hide
          Alparslan Avcı added a comment -

          Yes, you are right. There is no problem with testUpdateWebPageRemoveMapEntry().

          Show
          Alparslan Avcı added a comment - Yes, you are right. There is no problem with testUpdateWebPageRemoveMapEntry() .
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          and you meant only:

          • testUpdateWebPagePutToNotNullableMap
          • testUpdateWebPagePutToNullableMap

          right?
          Because testUpdateWebPageRemoveMapEntry seems fine to me. Wdyt?

          Show
          Renato Javier Marroquín Mogrovejo added a comment - and you meant only: testUpdateWebPagePutToNotNullableMap testUpdateWebPagePutToNullableMap right? Because testUpdateWebPageRemoveMapEntry seems fine to me. Wdyt?
          Hide
          Alparslan Avcı added a comment -

          Thanks Renato Javier Marroquín Mogrovejo

          I hope that after implementing the tests for other Avro field type cases, it will be clearer.

          Show
          Alparslan Avcı added a comment - Thanks Renato Javier Marroquín Mogrovejo I hope that after implementing the tests for other Avro field type cases, it will be clearer.
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Yes I had seen that, but I wondered if there was more than that
          Nicely refactored man, this is much more understandable now. I think I did a clean-up of this method a while ago, but now this looks much better, great work Alparslan Avcı!!

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Yes I had seen that, but I wondered if there was more than that Nicely refactored man, this is much more understandable now. I think I did a clean-up of this method a while ago, but now this looks much better, great work Alparslan Avcı !!
          Hide
          Alparslan Avcı added a comment - - edited

          Hi Renato Javier Marroquín Mogrovejo,

          You can just modify testUpdateWebPagePutToNullableMap() and testUpdateWebPageRemoveMapEntry() methods in DataStoreTestUtil class. The last iterations in these test methods has to be changed in order to get only odd indexed urls and headers. This will revert the test algorithm for testUpdate to the implementation currently in trunk.

          You can also remove TODO's in the class. They are not needed anymore.

          Thanks!

          Show
          Alparslan Avcı added a comment - - edited Hi Renato Javier Marroquín Mogrovejo , You can just modify testUpdateWebPagePutToNullableMap() and testUpdateWebPageRemoveMapEntry() methods in DataStoreTestUtil class. The last iterations in these test methods has to be changed in order to get only odd indexed urls and headers . This will revert the test algorithm for testUpdate to the implementation currently in trunk. You can also remove TODO's in the class. They are not needed anymore. Thanks!
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Hi Alparslan Avcı,

          I am trying to chip into this task as well, and I wanted to start with this:
          "2. Revert the test algorithm for testUpdate to the implementation currently in trunk"
          Any suggestions or caveats I should be aware of?
          Thanks!

          Renato M.

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Hi Alparslan Avcı , I am trying to chip into this task as well, and I wanted to start with this: "2. Revert the test algorithm for testUpdate to the implementation currently in trunk" Any suggestions or caveats I should be aware of? Thanks! Renato M.
          Hide
          Alparslan Avcı added a comment -

          Hi Lewis John McGibbney, sorry for the late answer.

          A simple test at the end, in addition to checking the odd indexed urls in the outlinks map would be for us to get the webpage from the datastore and check the total number of entries in the outlink map. This would mean we add the the algorithm by doing the final check from the persisted object.

          IMHO, since the outlink map is cleared in the tests, the total number of entries in the final map has to be equal to odd indexed urls count. This is already checked in the tests, so we do not need to do a final check, I think. Or, have I misunderstood your comment?

          1. Keep your improvements for adding the following methods
          + DataStoreTestUtil.testUpdateWebPagePutToArray(webPageStore);
          + DataStoreTestUtil.testUpdateWebPagePutToNotNullableMap(webPageStore);
          + DataStoreTestUtil.testUpdateWebPagePutToNullableMap(webPageStore);
          + DataStoreTestUtil.testUpdateWebPageRemoveMapEntry(webPageStore);
          + DataStoreTestUtil.testUpdateWebPageRemoveField(webPageStore);
          2. Revert the test algorithm for testUpdate to the implementation currently in trunk
          3. Fix datastores in accordance with this.

          +1 on this way. I will start fixing the clear() method problem as soon as possible.

          For extra test cases, Talat UYARER and I have worked together about possible combinations of Avro field types (like standart types, nullable unions, 3-type unions, unions with maps etc.) and have seen that there are more than 30 possible combinations. Thus, there are many test cases that have to be specified and implemented like your suggestions. I propose we can move with the present test cases for gora-0.4 and think other cases for next versions. This will also allow us to go into test driven development and give us time for refactoring the ways of implementing UNION types in more detail (like saving the index of unions on db).

          Show
          Alparslan Avcı added a comment - Hi Lewis John McGibbney , sorry for the late answer. A simple test at the end, in addition to checking the odd indexed urls in the outlinks map would be for us to get the webpage from the datastore and check the total number of entries in the outlink map. This would mean we add the the algorithm by doing the final check from the persisted object. IMHO, since the outlink map is cleared in the tests, the total number of entries in the final map has to be equal to odd indexed urls count . This is already checked in the tests, so we do not need to do a final check, I think. Or, have I misunderstood your comment? 1. Keep your improvements for adding the following methods + DataStoreTestUtil.testUpdateWebPagePutToArray(webPageStore); + DataStoreTestUtil.testUpdateWebPagePutToNotNullableMap(webPageStore); + DataStoreTestUtil.testUpdateWebPagePutToNullableMap(webPageStore); + DataStoreTestUtil.testUpdateWebPageRemoveMapEntry(webPageStore); + DataStoreTestUtil.testUpdateWebPageRemoveField(webPageStore); 2. Revert the test algorithm for testUpdate to the implementation currently in trunk 3. Fix datastores in accordance with this. +1 on this way. I will start fixing the clear() method problem as soon as possible. For extra test cases, Talat UYARER and I have worked together about possible combinations of Avro field types (like standart types, nullable unions, 3-type unions, unions with maps etc.) and have seen that there are more than 30 possible combinations. Thus, there are many test cases that have to be specified and implemented like your suggestions. I propose we can move with the present test cases for gora-0.4 and think other cases for next versions. This will also allow us to go into test driven development and give us time for refactoring the ways of implementing UNION types in more detail (like saving the index of unions on db).
          Hide
          Lewis John McGibbney added a comment -

          Joint commit made for GORA-94v11 and GORA-246v5
          Committed @revision 1572170 in GORA_94 branch

          Show
          Lewis John McGibbney added a comment - Joint commit made for GORA-94 v11 and GORA-246 v5 Committed @revision 1572170 in GORA_94 branch
          Hide
          Lewis John McGibbney added a comment -

          Hi Alparslan Avcı, thanks for comments. Apologies about my delay in getting back to you.

          I think, this is the true algorithm for testing the update...

          Yes I agree here. A simple test at the end, in addition to checking the odd indexed urls in the outlinks map would be for us to get the webpage from the datastore and check the total number of entries in the outlink map. This would mean we add the the algorithm by doing the final check from the persisted object. wdyt?

          ...since HBaseStore does not clear the map after DirtyMapWrapper.clear() method is called.

          I think that this is a big that we should correct. #clear methods should remove all entires from the data structure leaving it as an empty data structure (not null).

          I propose that we
          1. Keep your improvements for adding the following methods
          + DataStoreTestUtil.testUpdateWebPagePutToArray(webPageStore);
          + DataStoreTestUtil.testUpdateWebPagePutToNotNullableMap(webPageStore);
          + DataStoreTestUtil.testUpdateWebPagePutToNullableMap(webPageStore);
          + DataStoreTestUtil.testUpdateWebPageRemoveMapEntry(webPageStore);
          + DataStoreTestUtil.testUpdateWebPageRemoveField(webPageStore);
          2. Revert the test algorithm for testUpdate to the implementation currently in trunk
          3. Fix datastores in accordance with this.

          One final comment. I feel that we should maybe add test cases for
          + DataStoreTestUtil.testUpdateWebPagePutToNotNullableArray(webPageStore);
          + DataStoreTestUtil.testUpdateWebPagePutToNullableArray(webPageStore);
          + DataStoreTestUtil.testUpdateWebPageRemoveArrayEntry(webPageStore);

          This way we would really make explicit the semantics for dirtyable data structures.
          If you can comment on the stuff above then I will get to work on these implementations.
          Thanks for your comments.

          Show
          Lewis John McGibbney added a comment - Hi Alparslan Avcı , thanks for comments. Apologies about my delay in getting back to you. I think, this is the true algorithm for testing the update... Yes I agree here. A simple test at the end, in addition to checking the odd indexed urls in the outlinks map would be for us to get the webpage from the datastore and check the total number of entries in the outlink map. This would mean we add the the algorithm by doing the final check from the persisted object. wdyt? ...since HBaseStore does not clear the map after DirtyMapWrapper.clear() method is called. I think that this is a big that we should correct. #clear methods should remove all entires from the data structure leaving it as an empty data structure (not null). I propose that we 1. Keep your improvements for adding the following methods + DataStoreTestUtil.testUpdateWebPagePutToArray(webPageStore); + DataStoreTestUtil.testUpdateWebPagePutToNotNullableMap(webPageStore); + DataStoreTestUtil.testUpdateWebPagePutToNullableMap(webPageStore); + DataStoreTestUtil.testUpdateWebPageRemoveMapEntry(webPageStore); + DataStoreTestUtil.testUpdateWebPageRemoveField(webPageStore); 2. Revert the test algorithm for testUpdate to the implementation currently in trunk 3. Fix datastores in accordance with this. One final comment. I feel that we should maybe add test cases for + DataStoreTestUtil.testUpdateWebPagePutToNotNullableArray(webPageStore); + DataStoreTestUtil.testUpdateWebPagePutToNullableArray(webPageStore); + DataStoreTestUtil.testUpdateWebPageRemoveArrayEntry(webPageStore); This way we would really make explicit the semantics for dirtyable data structures. If you can comment on the stuff above then I will get to work on these implementations. Thanks for your comments.
          Hide
          Alparslan Avcı added a comment -
          webPage.getHeaders().clear(); //TODO clear method does not work

          Actually, this clear method mentioned above is DirtyMapWrapper.clear() method.

          In gora trunk, DataStoreTestUtil.testUpdateWebPage() method works as follows:

          1. puts even indexed urls to outlinks map
          2. flushes the datastore
          3. clears outlinks map
          4. puts odd indexed urls to outlinks map
          5. flushes the datastore
          6. asserts the size of outlinks map is equal to odd indexed urls count.

          I think, this is the true algorithm for testing the update. However, in GORA_94 branch's DataStoreTestUtil.testUpdateWebPage() method; it asserts the size of outlinks map is equal to the total urls count at the last step. The incremental of j variable has been somehow changed. I think, it was changed in order to pass the tests since HBaseStore does not clear the map after DirtyMapWrapper.clear() method is called.

          Those TODO's are all about this issue. In my opinion, we have to change the test algorithm back to the old version and fix the stores according to this. You can also check my latest comment and patch in GORA-246 for more info.

          Show
          Alparslan Avcı added a comment - webPage.getHeaders().clear(); //TODO clear method does not work Actually, this clear method mentioned above is DirtyMapWrapper.clear() method. In gora trunk, DataStoreTestUtil.testUpdateWebPage() method works as follows: puts even indexed urls to outlinks map flushes the datastore clears outlinks map puts odd indexed urls to outlinks map flushes the datastore asserts the size of outlinks map is equal to odd indexed urls count . I think, this is the true algorithm for testing the update. However, in GORA_94 branch's DataStoreTestUtil.testUpdateWebPage() method; it asserts the size of outlinks map is equal to the total urls count at the last step. The incremental of j variable has been somehow changed. I think, it was changed in order to pass the tests since HBaseStore does not clear the map after DirtyMapWrapper.clear() method is called. Those TODO's are all about this issue. In my opinion, we have to change the test algorithm back to the old version and fix the stores according to this. You can also check my latest comment and patch in GORA-246 for more info.
          Hide
          Lewis John McGibbney added a comment -

          the builder initiate this field with null value. So, we always have to set a new HashMap object to headers field at first.

          Duh, sorry about this one, I just didn't check the default value in the avro schema. You are completely right with this one, thank you for clarification.

          I had fixed this problem in GORA-94v10.patch and it seems that it is clearing the dirty states of all fields for now. Would you please elaborate this issue?

          Well the v11 patch includes the following TODO

          +      webPage.getHeaders().clear(); //TODO clear method does not work
          

          I therefore wondered if this TODO should be removed?
          Also I think we should make a decision on the other 2 TODO's regarding increments of local variable j. This should be trivial to implement.
          Once this is done i think we are ready to get this code in to the code base as the tests are well justified given the change in behavior of certain data structures which we have been discussing over the last weeks.

          Show
          Lewis John McGibbney added a comment - the builder initiate this field with null value. So, we always have to set a new HashMap object to headers field at first. Duh, sorry about this one, I just didn't check the default value in the avro schema. You are completely right with this one, thank you for clarification. I had fixed this problem in GORA-94 v10.patch and it seems that it is clearing the dirty states of all fields for now. Would you please elaborate this issue? Well the v11 patch includes the following TODO + webPage.getHeaders().clear(); //TODO clear method does not work I therefore wondered if this TODO should be removed? Also I think we should make a decision on the other 2 TODO's regarding increments of local variable j . This should be trivial to implement. Once this is done i think we are ready to get this code in to the code base as the tests are well justified given the change in behavior of certain data structures which we have been discussing over the last weeks.
          Hide
          Alparslan Avcı added a comment - - edited

          Hi Lewis John McGibbney, thanks for checking.

          Do we need to always setHeaders as follows --> webPage.setHeaders(new HashMap<CharSequence, CharSequence>()); ? What happens if we just access the headers field without initiating a new HashMap? Do we still get NPE? I though the builder build() method would initiate default empty HashMap?

          Since headers field is defined as 'nullable' and has 'null' as default value in JSON,

          {"name": "headers", "type": ["null",{"type": "map", "values": ["null", "string"]}],"default":null}

          the builder initiate this field with null value. So, we always have to set a new HashMap object to headers field at first.

          Do you have any idea what is the current issue with PersistentBase#clear() method? I am puzzled as to why it is not clearing dirty state of object field.

          I had fixed this problem in GORA-94v10.patch and it seems that it is clearing the dirty states of all fields for now. Would you please elaborate this issue?

          Show
          Alparslan Avcı added a comment - - edited Hi Lewis John McGibbney , thanks for checking. Do we need to always setHeaders as follows --> webPage.setHeaders(new HashMap<CharSequence, CharSequence>()); ? What happens if we just access the headers field without initiating a new HashMap? Do we still get NPE? I though the builder build() method would initiate default empty HashMap? Since headers field is defined as 'nullable' and has 'null' as default value in JSON, {"name": "headers", "type": [" null ",{"type": "map", "values": [" null ", "string"]}]," default ": null } the builder initiate this field with null value. So, we always have to set a new HashMap object to headers field at first. Do you have any idea what is the current issue with PersistentBase#clear() method? I am puzzled as to why it is not clearing dirty state of object field. I had fixed this problem in GORA-94 v10.patch and it seems that it is clearing the dirty states of all fields for now. Would you please elaborate this issue?
          Hide
          Lewis John McGibbney added a comment -

          Hi Alparslan Avcı, GORA-94v11.patch is much much better. Only thing it lacks in Javadoc
          Some observations

          • Do we need to always setHeaders as follows --> webPage.setHeaders(new HashMap<CharSequence, CharSequence>()); ? What happens if we just access the headers field without initiating a new HashMap? Do we still get NPE? I though the builder build() method would initiate default empty HashMap?
          • Do you have any idea what is the current issue with PersistentBase#clear() method? I am puzzled as to why it is not clearing dirty state of object field.
          Show
          Lewis John McGibbney added a comment - Hi Alparslan Avcı , GORA-94 v11.patch is much much better. Only thing it lacks in Javadoc Some observations Do we need to always setHeaders as follows --> webPage.setHeaders(new HashMap<CharSequence, CharSequence>()); ? What happens if we just access the headers field without initiating a new HashMap? Do we still get NPE? I though the builder build() method would initiate default empty HashMap? Do you have any idea what is the current issue with PersistentBase#clear() method? I am puzzled as to why it is not clearing dirty state of object field.
          Hide
          Alparslan Avcı added a comment -

          Hi Lewis John McGibbney, I've updated the GORA-94v11.patch and changed the tests. Would you please check and comment?

          Show
          Alparslan Avcı added a comment - Hi Lewis John McGibbney , I've updated the GORA-94 v11.patch and changed the tests. Would you please check and comment?
          Hide
          Alparslan Avcı added a comment -

          Thanks for the comment Lewis John McGibbney. Yes, you are right about JUnit tests. I can divide them into smaller tests that target only one purpose. As you said, it is very important to have maintainable tests in order to get easy and bug-free future implementations.

          Show
          Alparslan Avcı added a comment - Thanks for the comment Lewis John McGibbney . Yes, you are right about JUnit tests. I can divide them into smaller tests that target only one purpose. As you said, it is very important to have maintainable tests in order to get easy and bug-free future implementations.
          Hide
          Lewis John McGibbney added a comment -

          Hi Alparslan Avcı, regarding your comments on 22/Jan/14 12:58 and GORA-94v11.patch, I feel it is clearer to either fully document these new test operations or else make them into new tests. We want the JUnit method names within DataStoreTestBase and subsequently DataStoreTesUtil to reflect the actual tests which are being carried out... not for them to become large, multipurpose, difficult to maintain tests. We want to make them easy to debug and easy to read and infer what is actually going on.
          If we don't succeed in doing this, we run the risk of making future upgrades more difficult as we move on.
          Thanks for the patch.
          wdyt?

          Show
          Lewis John McGibbney added a comment - Hi Alparslan Avcı , regarding your comments on 22/Jan/14 12:58 and GORA-94v11.patch , I feel it is clearer to either fully document these new test operations or else make them into new tests. We want the JUnit method names within DataStoreTestBase and subsequently DataStoreTesUtil to reflect the actual tests which are being carried out... not for them to become large, multipurpose, difficult to maintain tests. We want to make them easy to debug and easy to read and infer what is actually going on. If we don't succeed in doing this, we run the risk of making future upgrades more difficult as we move on. Thanks for the patch. wdyt?
          Hide
          Lewis John McGibbney added a comment -

          GORA-94v12.patch Committed @revision 1560819 in GORA_94 branch. Thank you Alparslan Avcı. I'm going to port GORA-119 here now.

          Show
          Lewis John McGibbney added a comment - GORA-94 v12.patch Committed @revision 1560819 in GORA_94 branch. Thank you Alparslan Avcı . I'm going to port GORA-119 here now.
          Hide
          Lewis John McGibbney added a comment -

          Hi Alparslan Avcı. I also noticed this yesterday when I was porting GORA-119 to GORA_94 branch. I questioned why these were removed from generated data beans. I think it is fine to add these back in. Anyone else have comments?
          One positive for adding these accessors back in is simple, it means that client applications need to change less of their code if they upgrade thier Gora dependency for 0.4 when we release.

          Show
          Lewis John McGibbney added a comment - Hi Alparslan Avcı . I also noticed this yesterday when I was porting GORA-119 to GORA_94 branch. I questioned why these were removed from generated data beans. I think it is fine to add these back in. Anyone else have comments? One positive for adding these accessors back in is simple, it means that client applications need to change less of their code if they upgrade thier Gora dependency for 0.4 when we release.
          Hide
          Alparslan Avcı added a comment -

          Hi all again

          I'm working on using GORA_94 branch on Nutch 2.x and faced a problem. As you know, in order to execute a query with fields, we have to set the appropriate field schema names to the query. So, this makes a neccesity to reach the field schema names from any Persistent object. In the old version of GoraCompiler, we produce an enum that contains all the field schema names of the auto-generated class. I think, we also add this enum in Gora_94's compiler. In the patch I've uploaded, I've changed record.vm and regenerated the example classes.

          Show
          Alparslan Avcı added a comment - Hi all again I'm working on using GORA_94 branch on Nutch 2.x and faced a problem. As you know, in order to execute a query with fields, we have to set the appropriate field schema names to the query. So, this makes a neccesity to reach the field schema names from any Persistent object. In the old version of GoraCompiler, we produce an enum that contains all the field schema names of the auto-generated class. I think, we also add this enum in Gora_94's compiler. In the patch I've uploaded, I've changed record.vm and regenerated the example classes.
          Hide
          Alparslan Avcı added a comment - - edited

          Hi all,

          I've uploaded a patch that contains some tests. I think these are required for all back-ends.

          • Removal of entries by using .put(null) for Map type with nullable entries (Ex.: "type": {"type":"map", "values": ["null", "string"]}

            )

          • .put() and removal of the field by using .set[Fieldname](null) for unions that contain nullable Map type (Ex.: "type": ["null", {"type": "map", "values": "string"}

            ])

          I've also uploaded a patch to GORA-246 in order to pass these tests using HBaseStore.

          Would you please check and comment about the patch?

          Show
          Alparslan Avcı added a comment - - edited Hi all, I've uploaded a patch that contains some tests. I think these are required for all back-ends. Removal of entries by using .put(null) for Map type with nullable entries (Ex.: "type": {"type":"map", "values": ["null", "string"]} ) .put() and removal of the field by using .set [Fieldname] (null) for unions that contain nullable Map type (Ex.: "type": ["null", {"type": "map", "values": "string"} ]) I've also uploaded a patch to GORA-246 in order to pass these tests using HBaseStore. Would you please check and comment about the patch?
          Hide
          Alparslan Avcı added a comment -

          Thank you for your comment about patch Renato Javier Marroquín Mogrovejo and sorry for the late answer.

          You've noticed a good point, but as Martin Kleppmann mentioned in this (http://mail-archives.apache.org/mod_mbox/avro-user/201401.mbox/%3CCAHNLEqv%2BmzCJJPR3Y53WHJg-%2BiOOPgj3%3DxBaM0%3D3NGeamuUjsw%40mail.gmail.com%3E) thread, "In Avro, types are not nullable by default (unlike most programming languages)." So, you have to specify if the field will be null by default.
          Moreover, if a field's default value is not set and the builder pattern is used to create a new object, an AvroRuntimeException like "Field .... type: ... pos: .. not set and has no default value" will be thrown.

          Show
          Alparslan Avcı added a comment - Thank you for your comment about patch Renato Javier Marroquín Mogrovejo and sorry for the late answer. You've noticed a good point, but as Martin Kleppmann mentioned in this ( http://mail-archives.apache.org/mod_mbox/avro-user/201401.mbox/%3CCAHNLEqv%2BmzCJJPR3Y53WHJg-%2BiOOPgj3%3DxBaM0%3D3NGeamuUjsw%40mail.gmail.com%3E ) thread, "In Avro, types are not nullable by default (unlike most programming languages)." So, you have to specify if the field will be null by default. Moreover, if a field's default value is not set and the builder pattern is used to create a new object, an AvroRuntimeException like "Field .... type: ... pos: .. not set and has no default value" will be thrown.
          Hide
          Renato Javier Marroquín Mogrovejo added a comment -

          Hi Alparslan Avcı!

          First of all, thank you very much for your contributions! They are awesome! (:
          Just one thing about your patch, even though I think these changes are already committed. You said that for not facing the NPEs you set the values to its default value, so users will always have to set the fields with at least some value i.e. if the users forget to set these values they will get the NPE, am I understanding this correctly? if so, then we should try fixing the possibility of the users not having to specify a default value if it is null. Do you know what I mean?

          Show
          Renato Javier Marroquín Mogrovejo added a comment - Hi Alparslan Avcı ! First of all, thank you very much for your contributions! They are awesome! (: Just one thing about your patch, even though I think these changes are already committed. You said that for not facing the NPEs you set the values to its default value, so users will always have to set the fields with at least some value i.e. if the users forget to set these values they will get the NPE, am I understanding this correctly? if so, then we should try fixing the possibility of the users not having to specify a default value if it is null. Do you know what I mean?
          Hide
          Alparslan Avcı added a comment -

          Thank you for your help Lewis John McGibbney, I'm happy to contribute.
          I'm working on using GORA_94 branch on Nutch now. I think Talat UYARER would have some effort on GORA-245 in a few days.

          Show
          Alparslan Avcı added a comment - Thank you for your help Lewis John McGibbney , I'm happy to contribute. I'm working on using GORA_94 branch on Nutch now. I think Talat UYARER would have some effort on GORA-245 in a few days.
          Hide
          Lewis John McGibbney added a comment -

          GORA-94v10.patch Committed @revision 1556830 in GORA_94 branch. I also re-compiled the data beans in gora-core and gora-tutorial. All tests pass in gora-core and gora-hbase.
          Thank you very much Alparslan Avcı for the patches and the time.
          If you are interested in working with gora-cassandra. Your input would be very much appreciated over on issue GORA-245.

          Show
          Lewis John McGibbney added a comment - GORA-94 v10.patch Committed @revision 1556830 in GORA_94 branch. I also re-compiled the data beans in gora-core and gora-tutorial. All tests pass in gora-core and gora-hbase. Thank you very much Alparslan Avcı for the patches and the time. If you are interested in working with gora-cassandra. Your input would be very much appreciated over on issue GORA-245 .
          Hide
          Alparslan Avcı added a comment -

          Hi,
          I updated GORA-94v10.patch and now the tests are passed succesfully.

          In addition to previous patch, I changed the PersistentBase.clear() method and make the fields set with their default values instead of null. Lewis John McGibbney, would you also please check this update?

          And also I fixed some of the test objects that are not clearly defined for the tests.

          Show
          Alparslan Avcı added a comment - Hi, I updated GORA-94 v10.patch and now the tests are passed succesfully. In addition to previous patch, I changed the PersistentBase.clear() method and make the fields set with their default values instead of null. Lewis John McGibbney , would you also please check this update? And also I fixed some of the test objects that are not clearly defined for the tests.
          Hide
          Alparslan Avcı added a comment -

          And also about my patch, in order to get NULL value as default for a 'map' type field,

          "type": ["null",

          {"type":"map", "values":"string"}

          ], "default":null

          has to be used as type in .avsc definition. And in order to get 'empty map object' as default for a 'map' type field;

          "type":

          {"type":"map", "values":"string"}

          , "default":{}}

          has to be used.

          Show
          Alparslan Avcı added a comment - And also about my patch, in order to get NULL value as default for a 'map' type field, "type": ["null", {"type":"map", "values":"string"} ], "default":null has to be used as type in .avsc definition. And in order to get 'empty map object' as default for a 'map' type field; "type": {"type":"map", "values":"string"} , "default":{}} has to be used.
          Hide
          Alparslan Avcı added a comment -

          Thanks for the comments Lewis John McGibbney

          I've also noticed the failing JUnit tests. I'll try to fix them too.

          Show
          Alparslan Avcı added a comment - Thanks for the comments Lewis John McGibbney I've also noticed the failing JUnit tests. I'll try to fix them too.
          Hide
          Talat UYARER added a comment -

          Hi Lewis John McGibbney
          I think you forget share link.

          Show
          Talat UYARER added a comment - Hi Lewis John McGibbney I think you forget share link.
          Hide
          Lewis John McGibbney added a comment - - edited

          Hi Alparslan Avcı, firstly thank you very much for your patch and the effort required to look in to this... it is great for the 94 branch and of course for moving this issue to a close

          I refer you to this thread [0] which I asked on user@avro regarding default: null... I'm experimenting with making changes to default values for Dirtyable structures like Map's and Array's but I am getting loads of NPE when I run the JUnit tests locally. There are some bugs lingering here for sure.

          After applying your new patch I am currently still getting many NPE in JUnit tests so there is more work required. I'll keep working on this tonight and see where I can get.

          http://www.mail-archive.com/user%40avro.apache.org/msg02551.html

          Thanks Talat

          Show
          Lewis John McGibbney added a comment - - edited Hi Alparslan Avcı , firstly thank you very much for your patch and the effort required to look in to this... it is great for the 94 branch and of course for moving this issue to a close I refer you to this thread [0] which I asked on user@avro regarding default: null... I'm experimenting with making changes to default values for Dirtyable structures like Map's and Array's but I am getting loads of NPE when I run the JUnit tests locally. There are some bugs lingering here for sure. After applying your new patch I am currently still getting many NPE in JUnit tests so there is more work required. I'll keep working on this tonight and see where I can get. http://www.mail-archive.com/user%40avro.apache.org/msg02551.html Thanks Talat
          Hide
          Alparslan Avcı added a comment -

          Hi,
          When we want to call a 'map' type field (either null or empty map as default), field value comes null (not with an empty map object). This can cause NPE in many points like serializations. I've attached a patch that fixes this issue by using builder methods in new instance creation. I've also had to change record.vm and re-create the avro-generated example classes.
          Lewis John McGibbney, I think this can be more appropriate way of creating objects with default values. Would you please comment about this patch?

          Show
          Alparslan Avcı added a comment - Hi, When we want to call a 'map' type field (either null or empty map as default), field value comes null (not with an empty map object). This can cause NPE in many points like serializations. I've attached a patch that fixes this issue by using builder methods in new instance creation. I've also had to change record.vm and re-create the avro-generated example classes. Lewis John McGibbney , I think this can be more appropriate way of creating objects with default values. Would you please comment about this patch?
          Hide
          Alfonso Nishikawa added a comment -

          Hi!
          I find that GORA-94-v4.patch about the compiler is essentially wrong breaking back-compatibility. Tests should not be changed (if not to add more tests), and I see (not all) interfaces being changed.
          Pretty implementation, but I believe it miss the mark. The compiler should not affect any public part.
          What are your thoughts?

          Show
          Alfonso Nishikawa added a comment - Hi! I find that GORA-94 -v4.patch about the compiler is essentially wrong breaking back-compatibility. Tests should not be changed (if not to add more tests), and I see (not all) interfaces being changed. Pretty implementation, but I believe it miss the mark. The compiler should not affect any public part. What are your thoughts?
          Hide
          yasin tamer added a comment -

          if i could touch a small helps, this would have made happy me thanks for your motivations and supports, special thanks to Lewis

          Show
          yasin tamer added a comment - if i could touch a small helps, this would have made happy me thanks for your motivations and supports, special thanks to Lewis
          Hide
          Henry Saputra added a comment -

          W00t! Great job Yasin and Lewis.

          Show
          Henry Saputra added a comment - W00t! Great job Yasin and Lewis.
          Hide
          Lewis John McGibbney added a comment -

          v9 patch Committed @revision 1538057 in GORA_94 branch HEAD<> Thanks for the input Yasin. Great help Lets move on to HBase

          Show
          Lewis John McGibbney added a comment - v9 patch Committed @revision 1538057 in GORA_94 branch HEAD<> Thanks for the input Yasin. Great help Lets move on to HBase
          Hide
          Lewis John McGibbney added a comment -

          Patch for GORA_94 HEAD. All tests in gora-core pass now.
          yasin tamer we will deal with the HBase upgrade and Avro upgrade in GORA-246. I will upload the HBase specific stuff over there and we can work on it from there OK. Thanks for your persistence with this one Yasin. We are making progress. Effectively this part of the upgrade is now complete and we can move on.

          Show
          Lewis John McGibbney added a comment - Patch for GORA_94 HEAD. All tests in gora-core pass now. yasin tamer we will deal with the HBase upgrade and Avro upgrade in GORA-246 . I will upload the HBase specific stuff over there and we can work on it from there OK. Thanks for your persistence with this one Yasin. We are making progress. Effectively this part of the upgrade is now complete and we can move on.
          Hide
          Lewis John McGibbney added a comment -

          Hi Yasin. I'm working with your patch right now. There is a problem though. I use (and so does Gora) AVN for source code management. This means that your patches cannot be applied cleanly to a local copy of the code.
          Can you please generate your patches with
          git diff --no-prefix > myBeautifulPatch.patch
          This would be a huge favour + help for us trying to apply your patches.
          Thank you so much Yasin.

          Show
          Lewis John McGibbney added a comment - Hi Yasin. I'm working with your patch right now. There is a problem though. I use (and so does Gora) AVN for source code management. This means that your patches cannot be applied cleanly to a local copy of the code. Can you please generate your patches with git diff --no-prefix > myBeautifulPatch.patch This would be a huge favour + help for us trying to apply your patches. Thank you so much Yasin.
          Hide
          yasin tamer added a comment - - edited

          Hi Lewis, after i got last commits then i recreate GORA-94v8.patch which exclude meta-inf and manifest etc..

          Show
          yasin tamer added a comment - - edited Hi Lewis, after i got last commits then i recreate GORA-94 v8.patch which exclude meta-inf and manifest etc..
          Hide
          Lewis John McGibbney added a comment -

          Hi Yasin, thanks for this. Great work. Glad to see that you've moved on considerably with your use of Gora.
          I have some comments:

          • The way that the patch has been created means I cannot apply it locally to my checkout of GORA_94 branch. If you are in $GORA_HOME then you can generate a patch simply by doing svn diff > myBeautifulPatch.patch or if you are in git, then please do git diff --no-prefix > myBeautifulPatch.patch
          • The patch seems to combine numerous things which are not useful at this point. For example, it includes a bunch of MANIFEST information as well as pom.xml configuration to suit. This is not required for the Avro upgrade.
          • Is it possible for us to upgrade ONLY Avro with this umbrella issue? Is it absolutely required that an upgrade of HBase happens at the same time? I was hoping that this would not be the case but can live with it if it is.

          If we can begin with the above then we can move on.
          Finally, did you get a chance to look at the most recent code for the GORA-94 branch? AFAIK the Avro upgrade for gora-core is nearly complete with only one failing test e.g. testSerdeMultipleWebPages

          Again, thanks for your contributions. People like yourself pushing this forward is exactly what we require here.

          Show
          Lewis John McGibbney added a comment - Hi Yasin, thanks for this. Great work. Glad to see that you've moved on considerably with your use of Gora. I have some comments: The way that the patch has been created means I cannot apply it locally to my checkout of GORA_94 branch. If you are in $GORA_HOME then you can generate a patch simply by doing svn diff > myBeautifulPatch.patch or if you are in git, then please do git diff --no-prefix > myBeautifulPatch.patch The patch seems to combine numerous things which are not useful at this point. For example, it includes a bunch of MANIFEST information as well as pom.xml configuration to suit. This is not required for the Avro upgrade. Is it possible for us to upgrade ONLY Avro with this umbrella issue? Is it absolutely required that an upgrade of HBase happens at the same time? I was hoping that this would not be the case but can live with it if it is. If we can begin with the above then we can move on. Finally, did you get a chance to look at the most recent code for the GORA-94 branch? AFAIK the Avro upgrade for gora-core is nearly complete with only one failing test e.g. testSerdeMultipleWebPages Again, thanks for your contributions. People like yourself pushing this forward is exactly what we require here.
          Hide
          yasin tamer added a comment -

          Hi all this v8 test include a lot of fixes and developments, with this patch gora become almost compatible with hbase-0.94.12 and avro-1.7.x, all gora-hbase unit test succes except 2 errors and 16 failures, i will wait yours comments.

          Show
          yasin tamer added a comment - Hi all this v8 test include a lot of fixes and developments, with this patch gora become almost compatible with hbase-0.94.12 and avro-1.7.x, all gora-hbase unit test succes except 2 errors and 16 failures, i will wait yours comments.
          Hide
          Lewis John McGibbney added a comment -

          patch v7 Committed @revision 1535375 in GORA_94 branch
          Two failing tests are
          Tests in error:
          testSerdeMultipleWebPages(org.apache.gora.mapreduce.TestPersistentSerialization): null of map in field outlinks of org.apache.gora.examples.generated.WebPage
          testCountQuery(org.apache.gora.avro.mapreduce.TestDataFileAvroStoreMapReduce)

          Show
          Lewis John McGibbney added a comment - patch v7 Committed @revision 1535375 in GORA_94 branch Two failing tests are Tests in error: testSerdeMultipleWebPages(org.apache.gora.mapreduce.TestPersistentSerialization): null of map in field outlinks of org.apache.gora.examples.generated.WebPage testCountQuery(org.apache.gora.avro.mapreduce.TestDataFileAvroStoreMapReduce)
          Hide
          Lewis John McGibbney added a comment -

          Patch which includes Yasin's improvements and also passes all but 2 tests in gora-core. Once we can get these fixed we will be ready to get other datastores working.

          Show
          Lewis John McGibbney added a comment - Patch which includes Yasin's improvements and also passes all but 2 tests in gora-core. Once we can get these fixed we will be ready to get other datastores working.
          Hide
          Lewis John McGibbney added a comment -

          Hi Yasin, thanks for taking the time to look in to this one. It is my priority to get this upgrade done however I will not be able to hack the code until next week. Thursday at the earliest.
          This is not off my radar but I am just unable to hack the code until then. Thanks Yasin for you contributions.

          Show
          Lewis John McGibbney added a comment - Hi Yasin, thanks for taking the time to look in to this one. It is my priority to get this upgrade done however I will not be able to hack the code until next week. Thursday at the earliest. This is not off my radar but I am just unable to hack the code until then. Thanks Yasin for you contributions.
          Hide
          yasin tamer added a comment - - edited

          Hi Lewis, GORA-94v6.diff include some fixes, we solved stackoverflowexception when compiling employee.json via enqueue mechanism and add _ALL_FIELDS part to record.vm because test classes need _ALL_FIELDS and started to develop hbase module. you must apply this patch to gora-compile I wish to hear your comments.

          Show
          yasin tamer added a comment - - edited Hi Lewis, GORA-94 v6.diff include some fixes, we solved stackoverflowexception when compiling employee.json via enqueue mechanism and add _ALL_FIELDS part to record.vm because test classes need _ALL_FIELDS and started to develop hbase module. you must apply this patch to gora-compile I wish to hear your comments.
          Hide
          Lewis John McGibbney added a comment -

          Newly generated databeans for webpage, tokendatum, metadata, pageview and metricdatum schemas Committed @revision 1517094 in GORA_94 branch

          Show
          Lewis John McGibbney added a comment - Newly generated databeans for webpage, tokendatum, metadata, pageview and metricdatum schemas Committed @revision 1517094 in GORA_94 branch
          Hide
          Lewis John McGibbney added a comment -

          Patch GORA_94v5.patch Committed @revision 1517093 in GORA_94 branch

          Show
          Lewis John McGibbney added a comment - Patch GORA_94v5.patch Committed @revision 1517093 in GORA_94 branch
          Hide
          Lewis John McGibbney added a comment -

          OK I upload a patch which compiles against the GORA_94 branch.

          THIS HAS NOT BEEN TESTED. It does however port all of the gora-core upgrades to our upgraded code.

          It does additionally add all of the gora-cassandra upgrades however these should be avoided for the time being. I have disabled all modules apart from gora-core, the new compiler modules and the tutorial module until we stabalize gora-core.

          There are some major issues here regarding the support for our current Avro schema's in trunk. Especially I refer to the employee schema. There are serious issues here actually. In particular I get a StackOverflowException when attempting to compile the employee.json.

          I really don't know how we can absorb the changes in this patch as thy are quite literally huge. The entire persistency packages is revamped... there are also obvious changes to the way the GoraCompiler works compared to the current compiler in trunk and all of the additions which have been made over recent months.

          It is my intention to just commit this to GORA_94 branch, set up a Jenkins build, and look at what kind of results we get.

          Over time we can bring in everyone who wishes to upgrade individual modules as it is going to need a bit of effort on every module to get this working I think.

          OK I think that's about it. I'll commit (we can roll back no bother at all if it is not the right decision) and then we can get dug in to the upgrade on the 94 branch.

          Thanks

          Show
          Lewis John McGibbney added a comment - OK I upload a patch which compiles against the GORA_94 branch. THIS HAS NOT BEEN TESTED. It does however port all of the gora-core upgrades to our upgraded code. It does additionally add all of the gora-cassandra upgrades however these should be avoided for the time being. I have disabled all modules apart from gora-core, the new compiler modules and the tutorial module until we stabalize gora-core. There are some major issues here regarding the support for our current Avro schema's in trunk. Especially I refer to the employee schema. There are serious issues here actually. In particular I get a StackOverflowException when attempting to compile the employee.json. I really don't know how we can absorb the changes in this patch as thy are quite literally huge. The entire persistency packages is revamped... there are also obvious changes to the way the GoraCompiler works compared to the current compiler in trunk and all of the additions which have been made over recent months. It is my intention to just commit this to GORA_94 branch, set up a Jenkins build, and look at what kind of results we get. Over time we can bring in everyone who wishes to upgrade individual modules as it is going to need a bit of effort on every module to get this working I think. OK I think that's about it. I'll commit (we can roll back no bother at all if it is not the right decision) and then we can get dug in to the upgrade on the 94 branch. Thanks
          Hide
          yasin tamer added a comment -

          ok we are waiting for your commit eagerly, so do you have any time estimation for commit

          Show
          yasin tamer added a comment - ok we are waiting for your commit eagerly, so do you have any time estimation for commit
          Hide
          Lewis John McGibbney added a comment -

          Hi, yeah there is a branch created for this work now which can be found here [0]. This is in sync with trunk and has no 94 stuff committed as of yet. I have been working on a gora-core and cassandra patch which I will push to the branch. It will then be a case of taking the time to work with the patches attached to the sub tasks as part of this issue. That fact that you guys are willing to pick up the hbase stuff is really really great. I am not able to spend half as much time on this as I want to right now... which is really frustrating. I will try and push my changes to the 94 branch ASAP.

          [0] http://svn.apache.org/repos/asf/gora/branches/GORA_94/

          Show
          Lewis John McGibbney added a comment - Hi, yeah there is a branch created for this work now which can be found here [0] . This is in sync with trunk and has no 94 stuff committed as of yet. I have been working on a gora-core and cassandra patch which I will push to the branch. It will then be a case of taking the time to work with the patches attached to the sub tasks as part of this issue. That fact that you guys are willing to pick up the hbase stuff is really really great. I am not able to spend half as much time on this as I want to right now... which is really frustrating. I will try and push my changes to the 94 branch ASAP. [0] http://svn.apache.org/repos/asf/gora/branches/GORA_94/
          Hide
          Henry Saputra added a comment -

          I believe Lewis John McGibbney has created gora/branches/GORA_94/ branch for Avro upgrade

          Show
          Henry Saputra added a comment - I believe Lewis John McGibbney has created gora/branches/GORA_94/ branch for Avro upgrade
          Hide
          yasin tamer added a comment -

          Hi Lewis which branch or tag we can apply most recent patch?

          Show
          yasin tamer added a comment - Hi Lewis which branch or tag we can apply most recent patch?
          Hide
          Lewis John McGibbney added a comment -

          Hi Ed. Now that Dynamodb has been committed and the changes made to the core-module it would be excellent to pick this up. If you are able, it would be excellent to get your contribution reviewed, tested and hopefully committed in the short term. wdyt?

          Show
          Lewis John McGibbney added a comment - Hi Ed. Now that Dynamodb has been committed and the changes made to the core-module it would be excellent to pick this up. If you are able, it would be excellent to get your contribution reviewed, tested and hopefully committed in the short term. wdyt?
          Hide
          Lewis John McGibbney added a comment -

          I just noticed that the most recent (v4) patch also incorporates GORA-160. It would be nice to deal with that one first so I will progress on that basis.

          Show
          Lewis John McGibbney added a comment - I just noticed that the most recent (v4) patch also incorporates GORA-160 . It would be nice to deal with that one first so I will progress on that basis.
          Hide
          Lewis John McGibbney added a comment -

          This patch (once applied) does NOT compile fully. It still gives me 11 compilation errors for the CassandraStore and CassandraClient classes however I thought it best to attach as it updates the pom's the script in /bin and also gives those using SVN a cleaner approach to apply.

          From a Cassandra viewpoint I still have loads of stuff to look at but hopefully I will get around to this soon.

          Show
          Lewis John McGibbney added a comment - This patch (once applied) does NOT compile fully. It still gives me 11 compilation errors for the CassandraStore and CassandraClient classes however I thought it best to attach as it updates the pom's the script in /bin and also gives those using SVN a cleaner approach to apply. From a Cassandra viewpoint I still have loads of stuff to look at but hopefully I will get around to this soon.
          Hide
          Ed Kohlwey added a comment -

          I'll be maintaining this on top of the trunk in my github repository, running the tests after each merge.

          If you can integrate your findings as regression tests in trunk that will go a long way towards getting this merged in.

          I think the tombstone system could possibly be refactored out, but much of this patch depends directly on Avro. Also, much of the code in it is generated, so its not as big and scary as it seems . Nothing should have changed in terms of data compatibility; obviously there's some pretty significant API changes though.

          Do you guys want to do a hangout or IRC chat to discuss?

          Show
          Ed Kohlwey added a comment - I'll be maintaining this on top of the trunk in my github repository, running the tests after each merge. If you can integrate your findings as regression tests in trunk that will go a long way towards getting this merged in. I think the tombstone system could possibly be refactored out, but much of this patch depends directly on Avro. Also, much of the code in it is generated, so its not as big and scary as it seems . Nothing should have changed in terms of data compatibility; obviously there's some pretty significant API changes though. Do you guys want to do a hangout or IRC chat to discuss?
          Hide
          Lewis John McGibbney added a comment -

          Hi.

          {bq}Wow that is HUGE patch.{bq}

          ;0)

          This was also my major concern. I need to be honest and say that for us to pull this one off I think it's going to need to be much more incremental. Inevitably it affects all aspects of the entire Gora framework so I will also step up to try and test AMAP the gora-cassandra stuff.

          Show
          Lewis John McGibbney added a comment - Hi. {bq}Wow that is HUGE patch.{bq} ;0) This was also my major concern. I need to be honest and say that for us to pull this one off I think it's going to need to be much more incremental. Inevitably it affects all aspects of the entire Gora framework so I will also step up to try and test AMAP the gora-cassandra stuff.
          Hide
          Ferdy Galema added a comment -

          Wow that is HUGE patch. I tried to apply on trunk head but there where some merge problems already. I realize that it is a bit difficult to keep such a large patch in sync, so I think it would be best if we try to review and commit this as soon as possible. I will be able to do some testing with HBase next couple of days.

          Show
          Ferdy Galema added a comment - Wow that is HUGE patch. I tried to apply on trunk head but there where some merge problems already. I realize that it is a bit difficult to keep such a large patch in sync, so I think it would be best if we try to review and commit this as soon as possible. I will be able to do some testing with HBase next couple of days.
          Hide
          Ed Kohlwey added a comment - - edited

          I'm getting a 500 when I try to submit this on reviewboard, so here it is as a patch. This is more or less a final version.

          Some notes:
          I added a "tombstone" system to track deletions vs. creating additional state. This is convenient because it provides better compatability with maps and lists - just set the list or map tombstone on list or map elements to mark them as deleted, rather than the previous StatefulMap system. Persistent values now also have tombstones.
          I removed the StatefulMap system in favor of tombstones.
          I also reworked the way that PersistentBase works in order to force immutable Strings and ByteBuffers.

          Show
          Ed Kohlwey added a comment - - edited I'm getting a 500 when I try to submit this on reviewboard, so here it is as a patch. This is more or less a final version. Some notes: I added a "tombstone" system to track deletions vs. creating additional state. This is convenient because it provides better compatability with maps and lists - just set the list or map tombstone on list or map elements to mark them as deleted, rather than the previous StatefulMap system. Persistent values now also have tombstones. I removed the StatefulMap system in favor of tombstones. I also reworked the way that PersistentBase works in order to force immutable Strings and ByteBuffers.
          Hide
          Ed Kohlwey added a comment -

          Ferdy,
          Yes, ByteBuffer (as you note) could be protected to make them immutable. I think I would just do a copy of any buffer passed in and then do a .asReadOnlyBuffer() when the getters are called.
          CharSequences can also be stored as strings using the toString() on the setter to guarantee that the stored copy is immutable. By default Avro will put in Utf8's but we can do some processing on the getters to make sure no users gain access them.

          This might be the best thing for the moment, although it will have negative implications for the garbage collector in a map/reduce type context.

          Show
          Ed Kohlwey added a comment - Ferdy, Yes, ByteBuffer (as you note) could be protected to make them immutable. I think I would just do a copy of any buffer passed in and then do a .asReadOnlyBuffer() when the getters are called. CharSequences can also be stored as strings using the toString() on the setter to guarantee that the stored copy is immutable. By default Avro will put in Utf8's but we can do some processing on the getters to make sure no users gain access them. This might be the best thing for the moment, although it will have negative implications for the garbage collector in a map/reduce type context.
          Hide
          Ferdy Galema added a comment -

          Ok.

          About the CharSequence and ByteBuffer types: Is it possible to make them immutable? Perhaps by overriding the mutators and throw exceptions. I already noticed that you are returning CharSequence (which has no mutator methods in the interface), instead of Utf8. Is this not a hint to the user that it should not be casting/changing the value? ByteBuffer can already be read-only. This might change client usage (only if a client was directly changing the mutable in the first place), but perhaps this is worth the change. Anyway we can always try to solve this problem later and leave the semantics as they currently are.

          I think it is no problem that the format of intermediate output is changed. If there is the case that the final input/output rows have changed then this is something we just have to find out. It might depend on the store. I will test this with the HBaseStore.

          Thanks for the git info. I will manage to create diffs for me. However if you a have final version to be tested I think it's best to upload a patch yourself.

          Show
          Ferdy Galema added a comment - Ok. About the CharSequence and ByteBuffer types: Is it possible to make them immutable? Perhaps by overriding the mutators and throw exceptions. I already noticed that you are returning CharSequence (which has no mutator methods in the interface), instead of Utf8. Is this not a hint to the user that it should not be casting/changing the value? ByteBuffer can already be read-only. This might change client usage (only if a client was directly changing the mutable in the first place), but perhaps this is worth the change. Anyway we can always try to solve this problem later and leave the semantics as they currently are. I think it is no problem that the format of intermediate output is changed. If there is the case that the final input/output rows have changed then this is something we just have to find out. It might depend on the store. I will test this with the HBaseStore. Thanks for the git info. I will manage to create diffs for me. However if you a have final version to be tested I think it's best to upload a patch yourself.
          Hide
          Ed Kohlwey added a comment -

          actually, in a few places I iterate from 1 to n to avoid looking at the dirty byte array. I'm probably going to add a constant somewhere called FIRST_UNMANAGED_FIELD_INDEX or something to make this more explicit.

          I've added some map wrapper classes to the persistency.impl package that track structural changes to maps. There's no good way at the moment to track changes to ByteBuffer, and I'm not totally sure of a way to track changes to CharSequences (I'm currently thinking just extend Utf8). I was considering using AspectJ but the best way to do it would probably be as a runtime bytecode weaving thing which tends to be really slow. The option of having a class that doesn't inherit from ByteBuffer is also an option, but that will probably unnecessarily frustrate users. Another option could be to use a heuristic dirty for these based on a hash thats computed whenever clearDirty() is called.

          Serialization level compatibility is an issue. As I mentioned, this moves storage of the dirty field information into Avro instead of writing it out separately right before the serialized Avro object. But my understanding from a discussion on the mailing list is that that was only actually written as part of the Map/reduce intermediate output. There's also an issue with existing avro objects that don't have the dirty field. In that case I guess you would need to use a resolving decoder to read them in.

          If you check out the code using Git, you have two branches: my branch and the one you want to diff against. Then do

          git diff branch1 branch2 > /path/to/my/new/patch

          You can also use a tag name or a ref id in lieu of either of the branch names (which you can hunt down in git log)

          Or if you tell me what you want the diff against I'll make it and upload it. The current version still has some bugs related to the cassandra patch that just got merged into trunk, and the generated sample classes are missing from the tutorial project.

          Show
          Ed Kohlwey added a comment - actually, in a few places I iterate from 1 to n to avoid looking at the dirty byte array. I'm probably going to add a constant somewhere called FIRST_UNMANAGED_FIELD_INDEX or something to make this more explicit. I've added some map wrapper classes to the persistency.impl package that track structural changes to maps. There's no good way at the moment to track changes to ByteBuffer, and I'm not totally sure of a way to track changes to CharSequences (I'm currently thinking just extend Utf8). I was considering using AspectJ but the best way to do it would probably be as a runtime bytecode weaving thing which tends to be really slow. The option of having a class that doesn't inherit from ByteBuffer is also an option, but that will probably unnecessarily frustrate users. Another option could be to use a heuristic dirty for these based on a hash thats computed whenever clearDirty() is called. Serialization level compatibility is an issue. As I mentioned, this moves storage of the dirty field information into Avro instead of writing it out separately right before the serialized Avro object. But my understanding from a discussion on the mailing list is that that was only actually written as part of the Map/reduce intermediate output. There's also an issue with existing avro objects that don't have the dirty field. In that case I guess you would need to use a resolving decoder to read them in. If you check out the code using Git, you have two branches: my branch and the one you want to diff against. Then do git diff branch1 branch2 > /path/to/my/new/patch You can also use a tag name or a ref id in lieu of either of the branch names (which you can hunt down in git log) Or if you tell me what you want the diff against I'll make it and upload it. The current version still has some bugs related to the cassandra patch that just got merged into trunk, and the generated sample classes are missing from the tutorial project.
          Hide
          Ferdy Galema added a comment - - edited

          Hi,

          Thanks for the elaborate explanation. This is great stuff. I quickly glanced at some changes and notes a few things.

          In HBaseStore, you start iterating from 1 to fields.size(). I think this should be 0.

          About the dirty fields: Does a getMap() and changing some stuff without calling setMap(..) make that map field dirty? (I ask because I see the various putToMap(..) methods are now missing. Currently structural changes to maps are not tracked if I am not mistaken, unless they are done using the putToMap(..) methods, or the setMap method of course).

          Is compatibility an issue here? In the sense that after applying this fix it will change how final serialization is done?

          One final question. What is the easiest way to make a unified diff of all the relevant changes in your git branch, so that I can apply it easily on svn branches. That way I can test it extensively by running my some integration tests.

          Show
          Ferdy Galema added a comment - - edited Hi, Thanks for the elaborate explanation. This is great stuff. I quickly glanced at some changes and notes a few things. In HBaseStore, you start iterating from 1 to fields.size(). I think this should be 0. About the dirty fields: Does a getMap() and changing some stuff without calling setMap(..) make that map field dirty? (I ask because I see the various putToMap(..) methods are now missing. Currently structural changes to maps are not tracked if I am not mistaken, unless they are done using the putToMap(..) methods, or the setMap method of course). Is compatibility an issue here? In the sense that after applying this fix it will change how final serialization is done? One final question. What is the easiest way to make a unified diff of all the relevant changes in your git branch, so that I can apply it easily on svn branches. That way I can test it extensively by running my some integration tests.
          Hide
          Ed Kohlwey added a comment -

          Here's the github diff view - its also a convenient place to put code review comments.
          https://github.com/ekohlwey/gora/compare/trunk...avro-1.7.0

          If its more convenient I can do a reviewboard post too once I'm happy calling the patch more-or-less final.

          The upgrade is non-trivial, but I think most client API's should compile with a few exceptions.

          The first change I made was to refactor the whole Persistent system to remove the StateManager. It was obvious that it had been abandoned since the new and readable states weren't being used; additionally, its my opinion that using a lazy callback or something inside persistent objects would be better than using something like a readable field, and weather or not a field is new is basically the same thing as if all the fields are dirty, so this didn't make sense either.

          The next thing I wanted to do was remove the writing of transient information (the dirtyness) from being a hack on top of Avro. This was necessary since the Avro encoder system is now packaged up in such a way that you really can't override ResolvingDecoder, which was necessary for the previous implementation. I did this by doing a total rework of the Gora compiler so that it extends AvroCompiler. This also makes this class more manageable because the 1.7 Avro compiler uses velocity templates and has become fairly extendable. The new scheme for tracking dirtyness is to augment each schema with a new field, _g_dirty, which keeps track of what fields are dirty.

          I also have started work on making dirty more reliable. Using Utf8 classes and ByteBuffers, it is possible to make a field dirty without the set() method for the field intercepting the mutation. I think this is not the right thing. Structural changes to records, maps, lists, and unions are tracked automatically, however, as are changes on immutable types such as numbers and booleans.

          Most other modifications are trivial. Maps are represented by java.util.Map and arrays are represented by java.util.List now, so all the references to the Avro-ish way of doing things have been updated.

          Show
          Ed Kohlwey added a comment - Here's the github diff view - its also a convenient place to put code review comments. https://github.com/ekohlwey/gora/compare/trunk...avro-1.7.0 If its more convenient I can do a reviewboard post too once I'm happy calling the patch more-or-less final. The upgrade is non-trivial, but I think most client API's should compile with a few exceptions. The first change I made was to refactor the whole Persistent system to remove the StateManager. It was obvious that it had been abandoned since the new and readable states weren't being used; additionally, its my opinion that using a lazy callback or something inside persistent objects would be better than using something like a readable field, and weather or not a field is new is basically the same thing as if all the fields are dirty, so this didn't make sense either. The next thing I wanted to do was remove the writing of transient information (the dirtyness) from being a hack on top of Avro. This was necessary since the Avro encoder system is now packaged up in such a way that you really can't override ResolvingDecoder, which was necessary for the previous implementation. I did this by doing a total rework of the Gora compiler so that it extends AvroCompiler. This also makes this class more manageable because the 1.7 Avro compiler uses velocity templates and has become fairly extendable. The new scheme for tracking dirtyness is to augment each schema with a new field, _ g _dirty, which keeps track of what fields are dirty. I also have started work on making dirty more reliable. Using Utf8 classes and ByteBuffers, it is possible to make a field dirty without the set() method for the field intercepting the mutation. I think this is not the right thing. Structural changes to records, maps, lists, and unions are tracked automatically, however, as are changes on immutable types such as numbers and booleans. Most other modifications are trivial. Maps are represented by java.util.Map and arrays are represented by java.util.List now, so all the references to the Avro-ish way of doing things have been updated.
          Hide
          Lewis John McGibbney added a comment -

          Hi Ed. I wonder if it possible for you to explain a bit about your understanding of whats changing here. I'm not so clued up on the 1.7.X Avro Java API and therefore it would really help if you were able to either point me to a diff of the changes within your fork? Did you have any problems with the upgrade? Thanks very much in advance.

          Show
          Lewis John McGibbney added a comment - Hi Ed. I wonder if it possible for you to explain a bit about your understanding of whats changing here. I'm not so clued up on the 1.7.X Avro Java API and therefore it would really help if you were able to either point me to a diff of the changes within your fork? Did you have any problems with the upgrade? Thanks very much in advance.
          Hide
          Ed Kohlwey added a comment -

          I finally had some time this weekend to sit down and work on this.

          Here's a current snapshot in my git repo. I'd appreciate any comments anyone has.

          https://github.com/ekohlwey/gora

          Show
          Ed Kohlwey added a comment - I finally had some time this weekend to sit down and work on this. Here's a current snapshot in my git repo. I'd appreciate any comments anyone has. https://github.com/ekohlwey/gora
          Hide
          Enis Soztutar added a comment -

          Here you go Ed. I think from now on, you should be able to assign tickets. Let me know otherwise.

          Show
          Enis Soztutar added a comment - Here you go Ed. I think from now on, you should be able to assign tickets. Let me know otherwise.
          Hide
          Ed Kohlwey added a comment -

          I don't think I have permission to assign myself.

          Show
          Ed Kohlwey added a comment - I don't think I have permission to assign myself.
          Hide
          Lewis John McGibbney added a comment -

          Hi Ed. This would be absolutely excellent

          {bq} it will definitely be a patch that touches a lot of things{bq}

          yes your quite right. The Avro datastore stuff within Gora seems to have slightly neglected as of late with users directing more focus towards the other datastore implementations, however in previous board reports we have made a clear cut case to drag the Avro stuff up to date. Please feel free to assign yourself to this and work on it at your leisure. That would be excellent.

          Show
          Lewis John McGibbney added a comment - Hi Ed. This would be absolutely excellent {bq} it will definitely be a patch that touches a lot of things{bq} yes your quite right. The Avro datastore stuff within Gora seems to have slightly neglected as of late with users directing more focus towards the other datastore implementations, however in previous board reports we have made a clear cut case to drag the Avro stuff up to date. Please feel free to assign yourself to this and work on it at your leisure. That would be excellent.
          Hide
          Ed Kohlwey added a comment -

          I've started looking at a port to Avro 1.7 - I've done some initial hacking and it will definitely be a patch that touches a lot of things.

          There's a bunch of cool abstractions in 1.7 that will let us do this a little more easily, like the pluggable velocity templates. I'd be happy to take this on as the assignee if you want.

          Show
          Ed Kohlwey added a comment - I've started looking at a port to Avro 1.7 - I've done some initial hacking and it will definitely be a patch that touches a lot of things. There's a bunch of cool abstractions in 1.7 that will let us do this a little more easily, like the pluggable velocity templates. I'd be happy to take this on as the assignee if you want.
          Hide
          Lewis John McGibbney added a comment -

          We also need to consider core.site.xml when taking this one into consideration.

          http://svn.apache.org/repos/asf/gora/trunk/gora-core/src/test/conf/core-site.xml

          Show
          Lewis John McGibbney added a comment - We also need to consider core.site.xml when taking this one into consideration. http://svn.apache.org/repos/asf/gora/trunk/gora-core/src/test/conf/core-site.xml
          Hide
          Ferdy Galema added a comment -

          Thanks, now this issue is out of the way for the 0.2 release.

          Show
          Ferdy Galema added a comment - Thanks, now this issue is out of the way for the 0.2 release.
          Hide
          Lewis John McGibbney added a comment -

          Ferdy based on the above comments I agree with you. Thanks for reminding me about this thread as well, I'm going to chase it up soon as it's had plenty of time to settle in. I'll remove this as a blocker for GORA-76

          Show
          Lewis John McGibbney added a comment - Ferdy based on the above comments I agree with you. Thanks for reminding me about this thread as well, I'm going to chase it up soon as it's had plenty of time to settle in. I'll remove this as a blocker for GORA-76
          Hide
          Ferdy Galema added a comment -

          Reference to the ResolvingDecoder thread.
          http://mail-archives.apache.org/mod_mbox/avro-user/201202.mbox/%3CCAGaRif0gwXf278-DF63SEu65rEezT6HH3mGN5Gzrrg6N1aQfZw@mail.gmail.com%3E

          I looked into this and unfortunately could not fix all the errors that arise when updating Avro to 1.6.2. I came a long way, but FakeResolvingDecoder is still a bit unclear to me, as well as the getField(...) calls. When the inner workings of Gora are a bit more clear to me, I will continue this update attempt.

          In order to not block Hadoop 1.0, I propose to simply update Hadoop but leave Avro be. Please see my notes for updating Hadoop in GORA-76.

          Show
          Ferdy Galema added a comment - Reference to the ResolvingDecoder thread. http://mail-archives.apache.org/mod_mbox/avro-user/201202.mbox/%3CCAGaRif0gwXf278-DF63SEu65rEezT6HH3mGN5Gzrrg6N1aQfZw@mail.gmail.com%3E I looked into this and unfortunately could not fix all the errors that arise when updating Avro to 1.6.2. I came a long way, but FakeResolvingDecoder is still a bit unclear to me, as well as the getField(...) calls. When the inner workings of Gora are a bit more clear to me, I will continue this update attempt. In order to not block Hadoop 1.0, I propose to simply update Hadoop but leave Avro be. Please see my notes for updating Hadoop in GORA-76 .
          Hide
          Lewis John McGibbney added a comment -

          This is a 3/4 cooked patch. I'm getting around 5 issues @ comoile time and would really appreciate if someone a bit more Avro savvy than myself could check over it and comment/citicise.

          With regards to implementations for the deprecated ResolvingDecoder constructor implementation and init method I am waiting feedbac from the Avro lists.

          Show
          Lewis John McGibbney added a comment - This is a 3/4 cooked patch. I'm getting around 5 issues @ comoile time and would really appreciate if someone a bit more Avro savvy than myself could check over it and comment/citicise. With regards to implementations for the deprecated ResolvingDecoder constructor implementation and init method I am waiting feedbac from the Avro lists.
          Hide
          Lewis John McGibbney added a comment -

          This is a start at addressing the immediate issue of updating Avro libraries, there are only three Gora classes which will not compile. I don't know about tests, I'll hope for the best but prepare for the worst.

          Show
          Lewis John McGibbney added a comment - This is a start at addressing the immediate issue of updating Avro libraries, there are only three Gora classes which will not compile. I don't know about tests, I'll hope for the best but prepare for the worst.

            People

            • Assignee:
              Ed Kohlwey
              Reporter:
              Lewis John McGibbney
            • Votes:
              3 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development