Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.5.0
    • Component/s: core
    • Labels:
      None

      Description

      Add Voldemort as a service to Whirr. We have a first version ( https://github.com/rsumbaly/whirr/tree/voldemort ) and another one with more features from Kirk ( https://github.com/kirktrue/whirr/tree/voldemort) - More description ( https://groups.google.com/forum/?pli=1#!topic/project-voldemort/l6rb2X7YXEM )

      1. WHIRR-237.v5.patch
        35 kB
        Kirk True
      2. WHIRR-237.v4.patch
        36 kB
        Kirk True
      3. WHIRR-237.v3.patch
        36 kB
        Kirk True
      4. WHIRR-237.v2.patch
        41 kB
        Kirk True
      5. WHIRR-237.v1.patch
        41 kB
        Kirk True
      6. WHIRR-237.patch
        37 kB
        Tom White
      7. WHIRR-237.patch
        37 kB
        Tom White
      8. WHIRR-237.patch
        37 kB
        Andrei Savu
      9. linkedin-voldemort-0.90.RC3.jar
        1.31 MB
        Kirk True

        Activity

        Hide
        Andrei Savu added a comment -

        Awesome! A patch against the current trunk would be great

        Show
        Andrei Savu added a comment - Awesome! A patch against the current trunk would be great
        Hide
        Kirk True added a comment -

        This is the first pass at a patch to implement Voldemort as a target-able service for Whirr.

        There's at least one issue, and that is that the Voldemort client libraries are not in any Maven repositories. Is this a blocker for inclusion?

        Show
        Kirk True added a comment - This is the first pass at a patch to implement Voldemort as a target-able service for Whirr. There's at least one issue, and that is that the Voldemort client libraries are not in any Maven repositories. Is this a blocker for inclusion?
        Hide
        Tom White added a comment -

        > There's at least one issue, and that is that the Voldemort client libraries are not in any Maven repositories. Is this a blocker for inclusion?

        No, Cassandra has the same problem so its libraries are bundled in lib. The best thing is to attach them to this JIRA so they can be checked in at commit time.

        BTW can you regenerate the patch using git's --no-prefix option so it's easier to apply using patch. Thanks.

        Show
        Tom White added a comment - > There's at least one issue, and that is that the Voldemort client libraries are not in any Maven repositories. Is this a blocker for inclusion? No, Cassandra has the same problem so its libraries are bundled in lib. The best thing is to attach them to this JIRA so they can be checked in at commit time. BTW can you regenerate the patch using git's --no-prefix option so it's easier to apply using patch. Thanks.
        Hide
        Kirk True added a comment -

        Version 2 of the patch, which uses --no-prefix and updates a few settings to the latest version of Voldemort.

        Also includes the latest Voldemort JAR.

        Show
        Kirk True added a comment - Version 2 of the patch, which uses --no-prefix and updates a few settings to the latest version of Voldemort. Also includes the latest Voldemort JAR.
        Hide
        Tom White added a comment -

        Generally looks good. A few comments:

        • You don't need VoldemortClientService and VoldemortService since everything is role based now.
        • Change "voldemortclient" to "voldemort-client" by analogy with "hbase-master" etc.
        • Change "whirr.voldemort-tar-url" to "whirr.voldemort.tarball.url" for consistency with other services, like HBase. There are other properties that should be changed too, and you might consider putting them in a constants class (like HBaseConstants) to avoid duplication.
        • There's a spurious reference to Cassandra in pom.xml.
        • There are some tabs and inconsistent indentation (should be 2 spaces) in places.
        • Try running "mvn checkstyle:checkstyle apache-rat:check" to test for style and licensing compliance.

        Thanks for contributing this!

        Show
        Tom White added a comment - Generally looks good. A few comments: You don't need VoldemortClientService and VoldemortService since everything is role based now. Change "voldemortclient" to "voldemort-client" by analogy with "hbase-master" etc. Change "whirr.voldemort-tar-url" to "whirr.voldemort.tarball.url" for consistency with other services, like HBase. There are other properties that should be changed too, and you might consider putting them in a constants class (like HBaseConstants) to avoid duplication. There's a spurious reference to Cassandra in pom.xml. There are some tabs and inconsistent indentation (should be 2 spaces) in places. Try running "mvn checkstyle:checkstyle apache-rat:check" to test for style and licensing compliance. Thanks for contributing this!
        Hide
        Kirk True added a comment - - edited

        Uploaded version 3 of the code. This intentionally removes the Voldemort client cluster as it's probably too specific to my personal use case.

        • You don't need VoldemortClientService and VoldemortService since everything is role based now.

        Done.

        • Change "voldemortclient" to "voldemort-client" by analogy with "hbase-master" etc.

        Done, indirectly.

        • Change "whirr.voldemort-tar-url" to "whirr.voldemort.tarball.url" for consistency with other services, like HBase. There are other properties that should be changed too, and you might consider putting them in a constants class (like HBaseConstants) to avoid duplication.

        Done.

        • There's a spurious reference to Cassandra in pom.xml.

        Done.

        • There are some tabs and inconsistent indentation (should be 2 spaces) in places.

        Done, hopefully. Is there a tool to check this?

        • Try running "mvn checkstyle:checkstyle apache-rat:check" to test for style and licensing compliance.

        Done. Shell scripts have been updated with license.

        Show
        Kirk True added a comment - - edited Uploaded version 3 of the code. This intentionally removes the Voldemort client cluster as it's probably too specific to my personal use case. You don't need VoldemortClientService and VoldemortService since everything is role based now. Done. Change "voldemortclient" to "voldemort-client" by analogy with "hbase-master" etc. Done, indirectly. Change "whirr.voldemort-tar-url" to "whirr.voldemort.tarball.url" for consistency with other services, like HBase. There are other properties that should be changed too, and you might consider putting them in a constants class (like HBaseConstants) to avoid duplication. Done. There's a spurious reference to Cassandra in pom.xml. Done. There are some tabs and inconsistent indentation (should be 2 spaces) in places. Done, hopefully . Is there a tool to check this? Try running "mvn checkstyle:checkstyle apache-rat:check" to test for style and licensing compliance. Done. Shell scripts have been updated with license.
        Hide
        Tom White added a comment -

        Changes look good, integration tests pass, +1. I noticed that there is no slf4j dependency (see ZooKeeper for an example), which means that logs aren't generated. Do you want to fix that here or in a follow on issue?

        Show
        Tom White added a comment - Changes look good, integration tests pass, +1. I noticed that there is no slf4j dependency (see ZooKeeper for an example), which means that logs aren't generated. Do you want to fix that here or in a follow on issue?
        Hide
        Kirk True added a comment -

        Will work on the slf4j issue as part of this Jira. Thanks for catching that.

        Show
        Kirk True added a comment - Will work on the slf4j issue as part of this Jira. Thanks for catching that.
        Hide
        Kirk True added a comment -

        Cumulative patch, v4 includes the missing slf4j dependencies.

        Show
        Kirk True added a comment - Cumulative patch, v4 includes the missing slf4j dependencies.
        Hide
        Kirk True added a comment -

        Patch ready.

        Show
        Kirk True added a comment - Patch ready.
        Hide
        Adrian Cole added a comment -

        +1 looks clean and tests pass on aws-ec2 + cloudservers-us

        Show
        Adrian Cole added a comment - +1 looks clean and tests pass on aws-ec2 + cloudservers-us
        Hide
        Andrei Savu added a comment -

        +1 Looks great! I will run the tests tomorrow and if everything is fine I will commit the patch. Thanks Kirk!

        Show
        Andrei Savu added a comment - +1 Looks great! I will run the tests tomorrow and if everything is fine I will commit the patch. Thanks Kirk!
        Hide
        Andrei Savu added a comment -

        I believe there are some remaining configuration issues. The integration tests are working but the following manual test is failing:

        $ ./bin/whirr launch-cluster --config recipes/voldemort-ec2.properties
        [...]
        Completed setup of Voldemort voldemort with hosts 50.16.77.85,67.202.48.188
        Wrote instances file /home/andrei/.whirr/voldemort/instances
        Started cluster of 2 instances
        Cluster{instances=[Instance{roles=[voldemort], publicAddress=/50.16.77.85, privateAddress=/10.254.205.172, id=us-east-1/i-a6ac0cc9}, Instance{roles=[voldemort], publicAddress=/67.202.48.188, privateAddress=/10.206.54.191, id=us-east-1/i-a4ac0ccb}], configuration={}}
        
        $ ssh 50.16.77.85
        $ sudo su
        $ cd /usr/local/voldemort
        $ ./bin/voldemort-shell.sh test tcp://localhost:6666
        [2011-03-29 13:11:38,910 voldemort.client.DefaultStoreClient] INFO bootstrapping metadata for store test 
        Established connection to test via tcp://localhost:6666
        > put "key" "value"
        [2011-03-29 13:12:01,761 voldemort.store.routed.action.PerformParallelRequests] WARN Error in get versions on node 1(67.202.48.188) 
        voldemort.store.UnreachableStoreException: Failure while checking out socket for 67.202.48.188:6666(vp3): 
        	at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:124)
        	at voldemort.store.socket.SocketStore.requestAsync(SocketStore.java:283)
        	at voldemort.store.socket.SocketStore.submitGetVersionsRequest(SocketStore.java:139)
        	at voldemort.store.routed.action.PerformParallelRequests.execute(PerformParallelRequests.java:164)
        	at voldemort.store.routed.Pipeline.execute(Pipeline.java:214)
        	at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:338)
        	at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:65)
        	at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86)
        	at voldemort.store.serialized.SerializingStore.getVersions(SerializingStore.java:144)
        	at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86)
        	at voldemort.client.DefaultStoreClient.getVersions(DefaultStoreClient.java:155)
        	at voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:201)
        	at voldemort.VoldemortClientShell.processCommands(VoldemortClientShell.java:131)
        	at voldemort.VoldemortClientShell.main(VoldemortClientShell.java:112)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at jline.ConsoleRunner.main(ConsoleRunner.java:69)
        Caused by: java.net.ConnectException: Cannot connect socket 2 for 67.202.48.188:6666 after 508 ms
        	at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:147)
        	at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:49)
        	at voldemort.utils.pool.KeyedResourcePool.attemptGrow(KeyedResourcePool.java:153)
        	at voldemort.utils.pool.KeyedResourcePool.checkoutOrCreateResource(KeyedResourcePool.java:133)
        	at voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:106)
        	at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:119)
        	... 18 more
        [2011-03-29 13:12:01,762 voldemort.cluster.failuredetector.BannagePeriodFailureDetector] WARN Node 1 set as unavailable 
        voldemort.store.UnreachableStoreException: Failure while checking out socket for 67.202.48.188:6666(vp3): 
        	at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:124)
        	at voldemort.store.socket.SocketStore.requestAsync(SocketStore.java:283)
        	at voldemort.store.socket.SocketStore.submitGetVersionsRequest(SocketStore.java:139)
        	at voldemort.store.routed.action.PerformParallelRequests.execute(PerformParallelRequests.java:164)
        	at voldemort.store.routed.Pipeline.execute(Pipeline.java:214)
        	at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:338)
        	at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:65)
        	at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86)
        	at voldemort.store.serialized.SerializingStore.getVersions(SerializingStore.java:144)
        	at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86)
        	at voldemort.client.DefaultStoreClient.getVersions(DefaultStoreClient.java:155)
        	at voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:201)
        	at voldemort.VoldemortClientShell.processCommands(VoldemortClientShell.java:131)
        	at voldemort.VoldemortClientShell.main(VoldemortClientShell.java:112)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at jline.ConsoleRunner.main(ConsoleRunner.java:69)
        Caused by: java.net.ConnectException: Cannot connect socket 2 for 67.202.48.188:6666 after 508 ms
        	at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:147)
        	at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:49)
        	at voldemort.utils.pool.KeyedResourcePool.attemptGrow(KeyedResourcePool.java:153)
        	at voldemort.utils.pool.KeyedResourcePool.checkoutOrCreateResource(KeyedResourcePool.java:133)
        	at voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:106)
        	at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:119)
        	... 18 more
        [2011-03-29 13:12:01,763 voldemort.cluster.failuredetector.BannagePeriodFailureDetector] INFO Node 1 now unavailable 
        [2011-03-29 13:12:01,764 voldemort.client.AbstractStoreClientFactory] INFO Node1 in zone 0 partitionList:[0, 2, 8, 9, 10, 13, 15, 17, 18, 19] has been marked as unavailable, destroying socket pool 
        Exception thrown during operation.
        voldemort.store.InsufficientOperationalNodesException: 1 get versionss required, but only 0 succeeded
        	at voldemort.store.routed.action.PerformSerialRequests.execute(PerformSerialRequests.java:104)
        	at voldemort.store.routed.Pipeline.execute(Pipeline.java:214)
        	at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:338)
        	at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:65)
        	at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86)
        	at voldemort.store.serialized.SerializingStore.getVersions(SerializingStore.java:144)
        	at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86)
        	at voldemort.client.DefaultStoreClient.getVersions(DefaultStoreClient.java:155)
        	at voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:201)
        	at voldemort.VoldemortClientShell.processCommands(VoldemortClientShell.java:131)
        	at voldemort.VoldemortClientShell.main(VoldemortClientShell.java:112)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at jline.ConsoleRunner.main(ConsoleRunner.java:69)
        Caused by: voldemort.store.UnreachableStoreException: Failure while checking out socket for 67.202.48.188:6666(vp3): 
        	at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:124)
        	at voldemort.store.socket.SocketStore.requestAsync(SocketStore.java:283)
        	at voldemort.store.socket.SocketStore.submitGetVersionsRequest(SocketStore.java:139)
        	at voldemort.store.routed.action.PerformParallelRequests.execute(PerformParallelRequests.java:164)
        	... 15 more
        Caused by: java.net.ConnectException: Cannot connect socket 2 for 67.202.48.188:6666 after 508 ms
        	at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:147)
        	at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:49)
        	at voldemort.utils.pool.KeyedResourcePool.attemptGrow(KeyedResourcePool.java:153)
        	at voldemort.utils.pool.KeyedResourcePool.checkoutOrCreateResource(KeyedResourcePool.java:133)
        	at voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:106)
        	at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:119)
        	... 18 more
        
        

        I'm seeing the same error on both machines. I believe you need to use private IPs in configuration files.

        Show
        Andrei Savu added a comment - I believe there are some remaining configuration issues. The integration tests are working but the following manual test is failing: $ ./bin/whirr launch-cluster --config recipes/voldemort-ec2.properties [...] Completed setup of Voldemort voldemort with hosts 50.16.77.85,67.202.48.188 Wrote instances file /home/andrei/.whirr/voldemort/instances Started cluster of 2 instances Cluster{instances=[Instance{roles=[voldemort], publicAddress=/50.16.77.85, privateAddress=/10.254.205.172, id=us-east-1/i-a6ac0cc9}, Instance{roles=[voldemort], publicAddress=/67.202.48.188, privateAddress=/10.206.54.191, id=us-east-1/i-a4ac0ccb}], configuration={}} $ ssh 50.16.77.85 $ sudo su $ cd /usr/local/voldemort $ ./bin/voldemort-shell.sh test tcp://localhost:6666 [2011-03-29 13:11:38,910 voldemort.client.DefaultStoreClient] INFO bootstrapping metadata for store test Established connection to test via tcp://localhost:6666 > put "key" "value" [2011-03-29 13:12:01,761 voldemort.store.routed.action.PerformParallelRequests] WARN Error in get versions on node 1(67.202.48.188) voldemort.store.UnreachableStoreException: Failure while checking out socket for 67.202.48.188:6666(vp3): at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:124) at voldemort.store.socket.SocketStore.requestAsync(SocketStore.java:283) at voldemort.store.socket.SocketStore.submitGetVersionsRequest(SocketStore.java:139) at voldemort.store.routed.action.PerformParallelRequests.execute(PerformParallelRequests.java:164) at voldemort.store.routed.Pipeline.execute(Pipeline.java:214) at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:338) at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:65) at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86) at voldemort.store.serialized.SerializingStore.getVersions(SerializingStore.java:144) at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86) at voldemort.client.DefaultStoreClient.getVersions(DefaultStoreClient.java:155) at voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:201) at voldemort.VoldemortClientShell.processCommands(VoldemortClientShell.java:131) at voldemort.VoldemortClientShell.main(VoldemortClientShell.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at jline.ConsoleRunner.main(ConsoleRunner.java:69) Caused by: java.net.ConnectException: Cannot connect socket 2 for 67.202.48.188:6666 after 508 ms at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:147) at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:49) at voldemort.utils.pool.KeyedResourcePool.attemptGrow(KeyedResourcePool.java:153) at voldemort.utils.pool.KeyedResourcePool.checkoutOrCreateResource(KeyedResourcePool.java:133) at voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:106) at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:119) ... 18 more [2011-03-29 13:12:01,762 voldemort.cluster.failuredetector.BannagePeriodFailureDetector] WARN Node 1 set as unavailable voldemort.store.UnreachableStoreException: Failure while checking out socket for 67.202.48.188:6666(vp3): at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:124) at voldemort.store.socket.SocketStore.requestAsync(SocketStore.java:283) at voldemort.store.socket.SocketStore.submitGetVersionsRequest(SocketStore.java:139) at voldemort.store.routed.action.PerformParallelRequests.execute(PerformParallelRequests.java:164) at voldemort.store.routed.Pipeline.execute(Pipeline.java:214) at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:338) at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:65) at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86) at voldemort.store.serialized.SerializingStore.getVersions(SerializingStore.java:144) at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86) at voldemort.client.DefaultStoreClient.getVersions(DefaultStoreClient.java:155) at voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:201) at voldemort.VoldemortClientShell.processCommands(VoldemortClientShell.java:131) at voldemort.VoldemortClientShell.main(VoldemortClientShell.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at jline.ConsoleRunner.main(ConsoleRunner.java:69) Caused by: java.net.ConnectException: Cannot connect socket 2 for 67.202.48.188:6666 after 508 ms at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:147) at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:49) at voldemort.utils.pool.KeyedResourcePool.attemptGrow(KeyedResourcePool.java:153) at voldemort.utils.pool.KeyedResourcePool.checkoutOrCreateResource(KeyedResourcePool.java:133) at voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:106) at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:119) ... 18 more [2011-03-29 13:12:01,763 voldemort.cluster.failuredetector.BannagePeriodFailureDetector] INFO Node 1 now unavailable [2011-03-29 13:12:01,764 voldemort.client.AbstractStoreClientFactory] INFO Node1 in zone 0 partitionList:[0, 2, 8, 9, 10, 13, 15, 17, 18, 19] has been marked as unavailable, destroying socket pool Exception thrown during operation. voldemort.store.InsufficientOperationalNodesException: 1 get versionss required, but only 0 succeeded at voldemort.store.routed.action.PerformSerialRequests.execute(PerformSerialRequests.java:104) at voldemort.store.routed.Pipeline.execute(Pipeline.java:214) at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:338) at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:65) at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86) at voldemort.store.serialized.SerializingStore.getVersions(SerializingStore.java:144) at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86) at voldemort.client.DefaultStoreClient.getVersions(DefaultStoreClient.java:155) at voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:201) at voldemort.VoldemortClientShell.processCommands(VoldemortClientShell.java:131) at voldemort.VoldemortClientShell.main(VoldemortClientShell.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at jline.ConsoleRunner.main(ConsoleRunner.java:69) Caused by: voldemort.store.UnreachableStoreException: Failure while checking out socket for 67.202.48.188:6666(vp3): at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:124) at voldemort.store.socket.SocketStore.requestAsync(SocketStore.java:283) at voldemort.store.socket.SocketStore.submitGetVersionsRequest(SocketStore.java:139) at voldemort.store.routed.action.PerformParallelRequests.execute(PerformParallelRequests.java:164) ... 15 more Caused by: java.net.ConnectException: Cannot connect socket 2 for 67.202.48.188:6666 after 508 ms at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:147) at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:49) at voldemort.utils.pool.KeyedResourcePool.attemptGrow(KeyedResourcePool.java:153) at voldemort.utils.pool.KeyedResourcePool.checkoutOrCreateResource(KeyedResourcePool.java:133) at voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:106) at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:119) ... 18 more I'm seeing the same error on both machines. I believe you need to use private IPs in configuration files.
        Hide
        Kirk True added a comment -

        Voldemort has able to properly resolve the private IPs given the public ID as a bootstrap URL for me. I'll take a look.

        Thanks for your in-depth testing!

        Show
        Kirk True added a comment - Voldemort has able to properly resolve the private IPs given the public ID as a bootstrap URL for me. I'll take a look. Thanks for your in-depth testing!
        Hide
        Kirk True added a comment -

        Switched to private IP addresses.

        Show
        Kirk True added a comment - Switched to private IP addresses.
        Hide
        Andrei Savu added a comment -

        +1 tested on aws (integration and manual)

        Show
        Andrei Savu added a comment - +1 tested on aws (integration and manual)
        Hide
        Andrei Savu added a comment -

        Guys I'm planning to commit this tomorrow. The patch looks ready to me.

        Show
        Andrei Savu added a comment - Guys I'm planning to commit this tomorrow. The patch looks ready to me.
        Hide
        Tom White added a comment -

        WHIRR-262 broke the latest patch so I've made a one line change to fix it.

        Show
        Tom White added a comment - WHIRR-262 broke the latest patch so I've made a one line change to fix it.
        Hide
        Andrei Savu added a comment -

        I wanted to commit this patch but unfortunately the integration test is failing on cloudservers.

        voldemort.VoldemortException: voldemort.store.UnreachableStoreException: Failure while checking out socket for 50.56.115.48:6667(vp1): 
        [...]
        

        Any ideas why? The environment is quite similar with EC2 (ubuntu on machines with plenty of ram). Tom could this be related to the change you've made?

        Show
        Andrei Savu added a comment - I wanted to commit this patch but unfortunately the integration test is failing on cloudservers. voldemort.VoldemortException: voldemort.store.UnreachableStoreException: Failure while checking out socket for 50.56.115.48:6667(vp1): [...] Any ideas why? The environment is quite similar with EC2 (ubuntu on machines with plenty of ram). Tom could this be related to the change you've made?
        Hide
        Tom White added a comment -

        The error on the node was "Could not determine id for my host 173.203.218.120 against servers 10.177.169.139 10.177.169.13
        8."

        I changed configure_voldermort.sh so it uses eth1 to find a node's self address, and the test passes on Rackspace.

        Show
        Tom White added a comment - The error on the node was "Could not determine id for my host 173.203.218.120 against servers 10.177.169.139 10.177.169.13 8." I changed configure_voldermort.sh so it uses eth1 to find a node's self address, and the test passes on Rackspace.
        Hide
        Andrei Savu added a comment -

        Updated patch to use the FirewallManager and added Voldemort on the website home page. I've also tested the patch on EC2 and cloudservers. I'm going to commit this.

        Show
        Andrei Savu added a comment - Updated patch to use the FirewallManager and added Voldemort on the website home page. I've also tested the patch on EC2 and cloudservers. I'm going to commit this.
        Hide
        Andrei Savu added a comment -

        I've just committed this. Thanks guys!

        Show
        Andrei Savu added a comment - I've just committed this. Thanks guys!

          People

          • Assignee:
            Kirk True
            Reporter:
            Roshan Sumbaly
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development