Accumulo
  1. Accumulo
  2. ACCUMULO-2793

Clean up handling of moving HDFS under Accumulo from non-HA to HA

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.6.2, 1.7.0
    • Component/s: docs
    • Labels:
      None

      Description

      We added a util to handle cluster maintenance (such as going from non-HA HDFS to HA HDFS) in ACCUMULO-1832.

      It should be added to section 13.5 Tools in the user manual, and a reference to that section should be added in the section 12 explanation of the impact of supporting multi-volume setups.

        Issue Links

          Activity

          Hide
          Sean Busbey added a comment -

          Okay I found the explanation of instance.volumes.replacements in section 12.

          A section should still get added to the recovery section 13.3 HDFS troubleshooting for users changing their namenode or nameservice (such as moving from non-HA to HA).

          Show
          Sean Busbey added a comment - Okay I found the explanation of instance.volumes.replacements in section 12. A section should still get added to the recovery section 13.3 HDFS troubleshooting for users changing their namenode or nameservice (such as moving from non-HA to HA).
          Hide
          Sean Busbey added a comment - - edited

          Work in progress steps to change from non-HA to HA:

          assumptions, based on starting with a clean 1.6.0 install on top of non-HA HDFS:

          • original HDFS FS URL is hdfs://namenode.example.com:8020
          • new HA nameservice URL is hdfs://nameservice1
          • you didn't have instance.volumes configured
          • you didn't change the legacy instance dir stuff from /accumulo

          1) use ```$ACCUMULO_HOME/bin/accumulo admin volumes``` to confirm that only the expected volume is present

          Listing volumes referenced in zookeeper
                  Volume : hdfs://namenode.example.com:8020/accumulo
          
          Listing volumes referenced in accumulo.root tablets section
                  Volume : hdfs://namenode.example.com:8020/accumulo
          Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time)
          
          Listing volumes referenced in accumulo.metadata tablets section
                  Volume : hdfs://namenode.example.com:8020/accumulo
          
          Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time)
          

          2) shut down the Accumulo cluster
          3) Transition to HDFS HA with given nameservice above
          4) Edit $ACCUMULO_CONF_DIR/accumulo-site.xml

              <property>
                <name>instance.volumes</name>
                <value>hdfs://namenode.example.com:8020/accumulo,hdfs://nameservice1/accumulo</value>
              </property>
              <property>
                <name>instance.volumes.replacements</name>
                <value>hdfs://namenode.example.com:8020/accumulo hdfs://nameservice1/accumulo</value>
              </property>
          

          5) run $ACCUMULO_HOME/bin/accumulo init --add-volumes
          6) start the Accumulo cluster
          7) Verify that the new volume now shows up

          Listing volumes referenced in zookeeper
                  Volume : hdfs://namenode.example.com:8020/accumulo
                  Volume : hdfs://nameservice1/accumulo
          
          Listing volumes referenced in accumulo.root tablets section
                  Volume : hdfs://namenode.example.com:8020/accumulo
                  Volume : hdfs://nameservice1/accumulo
          Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time)
          
          Listing volumes referenced in accumulo.metadata tablets section
                  Volume : hdfs://namenode.example.com:8020/accumulo
                  Volume : hdfs://nameservice1/accumulo
          Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time)
          

          8) You may still see some erroneous GC failure messages as things transition.

          I can successfully ingest data both before and after failover using these steps

          Other things to clean up

          • The manual instructions around instance.volumes incorrectly lists the command as ```accumulo init --add-volume```
          • The init --add-volumes doesn't say that it has done anything
          • I get errors from the GC and when trying to start the shell after failover, looks like they're trying to use the defunct Volume
          • How do I remove the defunct Volume so that my configuration only refers to the HA nameservice?
          Show
          Sean Busbey added a comment - - edited Work in progress steps to change from non-HA to HA: assumptions, based on starting with a clean 1.6.0 install on top of non-HA HDFS: original HDFS FS URL is hdfs://namenode.example.com:8020 new HA nameservice URL is hdfs://nameservice1 you didn't have instance.volumes configured you didn't change the legacy instance dir stuff from /accumulo 1) use ```$ACCUMULO_HOME/bin/accumulo admin volumes``` to confirm that only the expected volume is present Listing volumes referenced in zookeeper Volume : hdfs://namenode.example.com:8020/accumulo Listing volumes referenced in accumulo.root tablets section Volume : hdfs://namenode.example.com:8020/accumulo Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time) Listing volumes referenced in accumulo.metadata tablets section Volume : hdfs://namenode.example.com:8020/accumulo Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time) 2) shut down the Accumulo cluster 3) Transition to HDFS HA with given nameservice above 4) Edit $ACCUMULO_CONF_DIR/accumulo-site.xml <property> <name> instance.volumes </name> <value> hdfs://namenode.example.com:8020/accumulo,hdfs://nameservice1/accumulo </value> </property> <property> <name> instance.volumes.replacements </name> <value> hdfs://namenode.example.com:8020/accumulo hdfs://nameservice1/accumulo </value> </property> 5) run $ACCUMULO_HOME/bin/accumulo init --add-volumes 6) start the Accumulo cluster 7) Verify that the new volume now shows up Listing volumes referenced in zookeeper Volume : hdfs://namenode.example.com:8020/accumulo Volume : hdfs://nameservice1/accumulo Listing volumes referenced in accumulo.root tablets section Volume : hdfs://namenode.example.com:8020/accumulo Volume : hdfs://nameservice1/accumulo Listing volumes referenced in accumulo.root deletes section (volume replacement occurrs at deletion time) Listing volumes referenced in accumulo.metadata tablets section Volume : hdfs://namenode.example.com:8020/accumulo Volume : hdfs://nameservice1/accumulo Listing volumes referenced in accumulo.metadata deletes section (volume replacement occurrs at deletion time) 8) You may still see some erroneous GC failure messages as things transition. I can successfully ingest data both before and after failover using these steps Other things to clean up The manual instructions around instance.volumes incorrectly lists the command as ```accumulo init --add-volume``` The init --add-volumes doesn't say that it has done anything I get errors from the GC and when trying to start the shell after failover, looks like they're trying to use the defunct Volume How do I remove the defunct Volume so that my configuration only refers to the HA nameservice?
          Hide
          Keith Turner added a comment -

          In step 4 why is the defunct volume still present in instance.volumes?

          Show
          Keith Turner added a comment - In step 4 why is the defunct volume still present in instance.volumes?
          Hide
          Sean Busbey added a comment -

          I thought it would be needed in order to use it in the replacements field (since the first error I got was that hte new volume was not present in the volumes list). I figured I'd have to wait for things to update before I could remove it.

          So I can just never list the defunct volume in the instance.volumes property?

          Show
          Sean Busbey added a comment - I thought it would be needed in order to use it in the replacements field (since the first error I got was that hte new volume was not present in the volumes list). I figured I'd have to wait for things to update before I could remove it. So I can just never list the defunct volume in the instance.volumes property?
          Hide
          Keith Turner added a comment -

          So I can just never list the defunct volume in the instance.volumes property?

          Yeah, should remove it from instance.volumes when setting the replacement. I checked w/ VolumeIT to ensure this scenario was tested, and it is. Leaving it would probably cause new files to be placed in the defunct volume.

          I took a look at ServerConstants.getVolumeReplacements() It does require that the new volume is present like you mentioned.

          Show
          Keith Turner added a comment - So I can just never list the defunct volume in the instance.volumes property? Yeah, should remove it from instance.volumes when setting the replacement. I checked w/ VolumeIT to ensure this scenario was tested, and it is. Leaving it would probably cause new files to be placed in the defunct volume. I took a look at ServerConstants.getVolumeReplacements() It does require that the new volume is present like you mentioned.
          Hide
          Keith Turner added a comment -

          The init --add-volumes doesn't say that it has done anything

          So it worked, but gave no confirmation?

          Show
          Keith Turner added a comment - The init --add-volumes doesn't say that it has done anything So it worked, but gave no confirmation?
          Hide
          Sean Busbey added a comment -

          The init --add-volumes doesn't say that it has done anything

          So it worked, but gave no confirmation?

          yes, precisely.

          Show
          Sean Busbey added a comment - The init --add-volumes doesn't say that it has done anything So it worked, but gave no confirmation? yes, precisely.
          Hide
          Keith Turner added a comment -

          Some sort of confirmation on add-volumes would be nice.

          Maybe the scenario where volumes being replaced still exist in instance.volumes should cause a warning to be logged, or even failure.

          Show
          Keith Turner added a comment - Some sort of confirmation on add-volumes would be nice. Maybe the scenario where volumes being replaced still exist in instance.volumes should cause a warning to be logged, or even failure.
          Hide
          Sean Busbey added a comment -

          Yes, I think a log at ERROR would be good. I don't know about actually failing.

          Show
          Sean Busbey added a comment - Yes, I think a log at ERROR would be good. I don't know about actually failing.
          Hide
          Mike Drob added a comment -

          Are the instructions given by Sean Busbey in his earlier comment accurate? I see there is some discussion about them afterwards and it is hard to decipher if anything needs to be tweaked or not.

          Show
          Mike Drob added a comment - Are the instructions given by Sean Busbey in his earlier comment accurate? I see there is some discussion about them afterwards and it is hard to decipher if anything needs to be tweaked or not.
          Hide
          Keith Turner added a comment -

          I think hdfs://namenode.example.com:8020/accumulo should be removed from the value of instance.volumes. Also may be worth add the following comment to the site snippet.

              <!-- instance.dfs.uri and instance.dfs.dir should not be set-->
              <property>
                <name>instance.volumes</name>
                <value>hdfs://nameservice1/accumulo</value>
              </property>
              <property>
                <name>instance.volumes.replacements</name>
                <value>hdfs://namenode.example.com:8020/accumulo hdfs://nameservice1/accumulo</value>
              </property>
          
          Show
          Keith Turner added a comment - I think hdfs://namenode.example.com:8020/accumulo should be removed from the value of instance.volumes . Also may be worth add the following comment to the site snippet. <!-- instance.dfs.uri and instance.dfs.dir should not be set--> <property> <name> instance.volumes </name> <value> hdfs://nameservice1/accumulo </value> </property> <property> <name> instance.volumes.replacements </name> <value> hdfs://namenode.example.com:8020/accumulo hdfs://nameservice1/accumulo </value> </property>
          Hide
          Josh Elser added a comment -

          Sean Busbey, do you have cycles to get this into 1.6.2?

          Show
          Josh Elser added a comment - Sean Busbey , do you have cycles to get this into 1.6.2?
          Hide
          Christopher Tubbs added a comment -

          If this is a blocker, it should be a blocker for 1.6.2, not 1.6.3 (we shouldn't be bumping blockers, by definition of "blocker"). So, either the "blocker" keyword needs to change to something that indicates less urgency, or the fixVersion needs to be set back to 1.6.2.

          That said, I don't think this should be a blocker, but it'd be a very nice to have.

          Show
          Christopher Tubbs added a comment - If this is a blocker, it should be a blocker for 1.6.2, not 1.6.3 (we shouldn't be bumping blockers, by definition of "blocker"). So, either the "blocker" keyword needs to change to something that indicates less urgency, or the fixVersion needs to be set back to 1.6.2. That said, I don't think this should be a blocker, but it'd be a very nice to have.
          Hide
          Sean Busbey added a comment -

          FWIW, this was originally a blocker on 1.6.1. My reasoning was that without this kind of documentation operations users would suffer because of the features added in 1.6.0, even if they didn't directly use the feature.

          Since it has been pushed out twice now, I think it's safe to infer other folks don't feel the same. Presuming no one speaks up, it's probably safe to downgrade the priority. Personally, I'd still consider it critical.

          Show
          Sean Busbey added a comment - FWIW, this was originally a blocker on 1.6.1. My reasoning was that without this kind of documentation operations users would suffer because of the features added in 1.6.0, even if they didn't directly use the feature. Since it has been pushed out twice now, I think it's safe to infer other folks don't feel the same. Presuming no one speaks up, it's probably safe to downgrade the priority. Personally, I'd still consider it critical.
          Hide
          Christopher Tubbs added a comment -

          It's a shame it got bumped past 1.6.1 since it was marked as a blocker (apparently without a review?). We need to do a better job of reviewing issues (especially blockers) before releasing. We still have time to fix it for 1.6.2 if somebody wants to submit a patch.

          Show
          Christopher Tubbs added a comment - It's a shame it got bumped past 1.6.1 since it was marked as a blocker (apparently without a review?). We need to do a better job of reviewing issues (especially blockers) before releasing. We still have time to fix it for 1.6.2 if somebody wants to submit a patch.
          Hide
          Corey J. Nolet added a comment -

          I should probably have reduced the priority when I bumped it. I reached out to Sean Busbey offline about this. I'll get a patch together for this if you feel strongly that it should be included in the 1.6.2 release.

          Show
          Corey J. Nolet added a comment - I should probably have reduced the priority when I bumped it. I reached out to Sean Busbey offline about this. I'll get a patch together for this if you feel strongly that it should be included in the 1.6.2 release.
          Hide
          Corey J. Nolet added a comment -

          Do these instructions really fit in the troubleshooting section? It seems like they would fit better in a section geared towards administration of the cluster than a section geared towards trying to solve problems with the cluster. At least the first place I would look would be in the administration section.

          Show
          Corey J. Nolet added a comment - Do these instructions really fit in the troubleshooting section? It seems like they would fit better in a section geared towards administration of the cluster than a section geared towards trying to solve problems with the cluster. At least the first place I would look would be in the administration section.
          Hide
          Josh Elser added a comment -

          If you're taking the time to write it up, wherever you put it in the manual will be fine!

          Show
          Josh Elser added a comment - If you're taking the time to write it up, wherever you put it in the manual will be fine!
          Hide
          Corey J. Nolet added a comment -
          Show
          Corey J. Nolet added a comment - Review posted at https://reviews.apache.org/r/29959/
          Hide
          Corey J. Nolet added a comment -

          Going to close this for now. If we determine the current docs aren't clear enough for the user, we can reopen.

          Show
          Corey J. Nolet added a comment - Going to close this for now. If we determine the current docs aren't clear enough for the user, we can reopen.

            People

            • Assignee:
              Corey J. Nolet
              Reporter:
              Sean Busbey
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m

                  Development