ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-627

zkpython arbitrarily restricts the size of a 'get' to 512 bytes

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: contrib-bindings
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      fix to remove restrictions on get in the zkpython library.

      Description

      Reported on the list:

      "
      I'm working on using ZooKeeper for an internal application at Digg. I've been using the zkpython package and I just noticed that the data I was receiving from a zookeeper.get() call was being truncated. After some quick digging I found that zookeeper.c limits the data returned to 512 characters (see http://svn.apache.org/viewvc/hadoop/zookeeper/tags/release-3.2.2/src/contrib/zkpython/src/c/zookeeper.c?view=markup line 855).

      Is there a reason for this? The only information regarding node size that I've read is that it should not exceed 1MB so this limit seems a bit arbitrary and restrictive.

      Thanks for the great work!

      Rich"

      1. ZOOKEEPER-627.patch
        4 kB
        Henry Robinson
      2. ZOOKEEPER-627.patch
        4 kB
        Henry Robinson
      3. ZOOKEEPER-627.patch
        4 kB
        Henry Robinson

        Activity

        Hide
        Henry Robinson added a comment -

        Attached patch increases default buffer size to 1024*1024 bytes, and adds a fourth parameter to zookeeper.get to specify maximum buffer size.

        Tests included.

        Show
        Henry Robinson added a comment - Attached patch increases default buffer size to 1024*1024 bytes, and adds a fourth parameter to zookeeper.get to specify maximum buffer size. Tests included.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428121/ZOOKEEPER-627.patch
        against trunk revision 891034.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/88/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/88/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/88/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428121/ZOOKEEPER-627.patch against trunk revision 891034. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/88/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/88/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/88/console This message is automatically generated.
        Hide
        Gustavo Niemeyer added a comment -

        Thanks for the fix Henry.

        I think there's a minor detail to sort out before merging it: it looks like the buffer is being freed before it's actually used to build the return value.

        Show
        Gustavo Niemeyer added a comment - Thanks for the fix Henry. I think there's a minor detail to sort out before merging it: it looks like the buffer is being freed before it's actually used to build the return value.
        Hide
        Henry Robinson added a comment -

        This is what happens when you write a patch quickly Good catch, thanks!

        Show
        Henry Robinson added a comment - This is what happens when you write a patch quickly Good catch, thanks!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428181/ZOOKEEPER-627.patch
        against trunk revision 891034.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/91/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428181/ZOOKEEPER-627.patch against trunk revision 891034. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/91/console This message is automatically generated.
        Hide
        Henry Robinson added a comment -

        Re-submitting with --no-prefix, so hudson can apply the patch.

        Show
        Henry Robinson added a comment - Re-submitting with --no-prefix, so hudson can apply the patch.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428198/ZOOKEEPER-627.patch
        against trunk revision 891368.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/92/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/92/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/92/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428198/ZOOKEEPER-627.patch against trunk revision 891368. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/92/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/92/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/92/console This message is automatically generated.
        Hide
        Mahadev konar added a comment -

        +1 the patch looks good.
        btw, henry
        I just noticed that the c code in zookeeper.c has inconsistent formatting. Usually we use 4 space indentation with no tabs (which I like as well )... do we want to fix that sometime (in a different patch obviously) or you want ot keep it as it is?

        Show
        Mahadev konar added a comment - +1 the patch looks good. btw, henry I just noticed that the c code in zookeeper.c has inconsistent formatting. Usually we use 4 space indentation with no tabs (which I like as well )... do we want to fix that sometime (in a different patch obviously) or you want ot keep it as it is?
        Hide
        Gustavo Niemeyer added a comment -

        I vote for fixing it at some point. It'd be nice to fix the style as well (parenthesis, etc), in addition to the spacing.

        Show
        Gustavo Niemeyer added a comment - I vote for fixing it at some point. It'd be nice to fix the style as well (parenthesis, etc), in addition to the spacing.
        Hide
        Henry Robinson added a comment -

        Yes, it needs a good spring clean - both to address the memory allocation issues that Gustavo has highlighted, and the style / spacing. I'll open the JIRA.

        Show
        Henry Robinson added a comment - Yes, it needs a good spring clean - both to address the memory allocation issues that Gustavo has highlighted, and the style / spacing. I'll open the JIRA.
        Hide
        Mahadev konar added a comment -

        great ... ill go ahead and commit this patch for now. The new jira will take care of cleaning it up.

        Show
        Mahadev konar added a comment - great ... ill go ahead and commit this patch for now. The new jira will take care of cleaning it up.
        Hide
        Mahadev konar added a comment -

        I just committed this. thanks henry.

        Show
        Mahadev konar added a comment - I just committed this. thanks henry.
        Hide
        Hudson added a comment -

        Integrated in ZooKeeper-trunk #632 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/632/)

        Show
        Hudson added a comment - Integrated in ZooKeeper-trunk #632 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/632/ )

          People

          • Assignee:
            Henry Robinson
            Reporter:
            Henry Robinson
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development