Whirr
  1. Whirr
  2. WHIRR-288

Add blob store persistence for cluster state

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: None
    • Labels:
      None

      Description

      Right now Whirr persists the current cluster state (instance IDs, public and private IPs and roles) on the local filesystem (~/.whirr/<cluster-name>/instances). It should be possible to save the state as a blob so that it's easily accessible from multiple machines.

      1. WHIRR-288.patch
        16 kB
        Andrei Savu
      2. WHIRR-288.patch
        21 kB
        Andrei Savu
      3. WHIRR-288.patch
        21 kB
        Andrei Savu
      4. WHIRR-288.patch
        22 kB
        Andrei Savu

        Issue Links

          Activity

          Hide
          Andrei Savu added a comment -

          I've just committed this. Guys thanks for helping and reviewing.

          Show
          Andrei Savu added a comment - I've just committed this. Guys thanks for helping and reviewing.
          Hide
          Adrian Cole added a comment -

          +1

          Show
          Adrian Cole added a comment - +1
          Hide
          Andrei Savu added a comment -

          I was using an invalid container name. I've added code for strict format checking (thanks Adrian for help) and I've been able to start a cluster using blob store state persistence. I'm going to commit this.

          Show
          Andrei Savu added a comment - I was using an invalid container name. I've added code for strict format checking (thanks Adrian for help) and I've been able to start a cluster using blob store state persistence. I'm going to commit this.
          Hide
          Andrei Savu added a comment -

          Updated patch with some fixes. Unfortunately it still doesn't work. Adrian, any ideas?

          Show
          Andrei Savu added a comment - Updated patch with some fixes. Unfortunately it still doesn't work. Adrian, any ideas?
          Hide
          Andrei Savu added a comment -

          I want to commit this but unfortunately I'm still seeing some errors (even if the integration tests are passing):

          endpoint.getHost() is null for https://whirr_test.s3.amazonaws.com/whirr-zookeeper
          

          Any ideas? I will investigate more to get it fixed.

          Show
          Andrei Savu added a comment - I want to commit this but unfortunately I'm still seeing some errors (even if the integration tests are passing): endpoint.getHost() is null for https://whirr_test.s3.amazonaws.com/whirr-zookeeper Any ideas? I will investigate more to get it fixed.
          Hide
          Andrei Savu added a comment -

          > Have you thought about using Avro/JSON as the serialization format? It would then be easier to evolve.

          I believe the current format is good enough for now. I suggest we should change it later when the state data model evolves.

          > Have a parser object that has public methods that are easier to test than the ones on ClusterStateStore. They don't really belong there as the ClusterStateStore could use any representation it chooses.
          > Rather than use NoopClusterStateStore, the default might be in-memory (which you implemented elsewhere)?

          I will address these two issues in one step later while improving code testability.

          Show
          Andrei Savu added a comment - > Have you thought about using Avro/JSON as the serialization format? It would then be easier to evolve. I believe the current format is good enough for now. I suggest we should change it later when the state data model evolves. > Have a parser object that has public methods that are easier to test than the ones on ClusterStateStore. They don't really belong there as the ClusterStateStore could use any representation it chooses. > Rather than use NoopClusterStateStore, the default might be in-memory (which you implemented elsewhere)? I will address these two issues in one step later while improving code testability.
          Hide
          Tom White added a comment -

          +1 Looks great.

          A few comments, which we might address in follow-on issues.

          • Have you thought about using Avro/JSON as the serialization format? It would then be easier to evolve.
          • Have a parser object that has public methods that are easier to test than the ones on ClusterStateStore. They don't really belong there as the ClusterStateStore could use any representation it chooses.
          • Rather than use NoopClusterStateStore, the default might be in-memory (which you implemented elsewhere)?
          Show
          Tom White added a comment - +1 Looks great. A few comments, which we might address in follow-on issues. Have you thought about using Avro/JSON as the serialization format? It would then be easier to evolve. Have a parser object that has public methods that are easier to test than the ones on ClusterStateStore. They don't really belong there as the ClusterStateStore could use any representation it chooses. Rather than use NoopClusterStateStore, the default might be in-memory (which you implemented elsewhere)?
          Hide
          Andrei Savu added a comment -

          Added integration test and it works on EC2.

          Show
          Andrei Savu added a comment - Added integration test and it works on EC2.
          Hide
          Andrei Savu added a comment -

          Not tested (yet) patch (it still needs integration tests). Let me know what you think.

          Show
          Andrei Savu added a comment - Not tested (yet) patch (it still needs integration tests). Let me know what you think.

            People

            • Assignee:
              Andrei Savu
              Reporter:
              Andrei Savu
            • Votes:
              1 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development