Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: master, tserver
    • Labels:
      None

      Description

      instance.dfs.uri and instance.dfs.dir are no longer needed with the instance.volumes property.

      Together, these two fields are needed for upgrades from relative paths, but full URIs for volumes should be specified in the instance.volumes set.

      Instead of appending instance.dfs.dir to every volume, which is a bit confusing, they should be specified explicitly in the instance.volumes.

      Example:

       <property>
          <name>instance.volumes</name>
          <value>hdfs://nn1/accumulo</value>
        </property>
      

      should be equivalent to

       <property>
          <name>instance.dfs.uri</name>
          <value>hdfs://nn1</value>
        </property>
       <property>
          <name>instance.dfs.dir</name>
          <value>/accumulo</value>
        </property>
      

      This change simplifies the semantics of configuring volumes for Accumulo to use for storage, and is a bit more obvious that we're logically configuring filesystem volumes, not "namenode URIs".

        Issue Links

          Activity

          Christopher Tubbs created issue -
          Christopher Tubbs made changes -
          Field Original Value New Value
          Component/s master [ 12316202 ]
          Component/s tserver [ 12316201 ]
          Hide
          Josh Elser added a comment - - edited

          I do want to note that you are increasing the the configuration complexity for "common" users now. Currently, instance.dfs.uri doesn't need to be specified (and will be pulled out of hdfs-site.xml if on the classpath) which lets me make a new installation by only specifying the HDFS directory I want to use with instance.dfs.dir.

          Edit: Can I specify a path only in instance.volumes instead of URI which would default to the configuration from hdfs-site (defaultFS, is it?) to maintain backwards compatibility with configurations?

          Show
          Josh Elser added a comment - - edited I do want to note that you are increasing the the configuration complexity for "common" users now. Currently, instance.dfs.uri doesn't need to be specified (and will be pulled out of hdfs-site.xml if on the classpath) which lets me make a new installation by only specifying the HDFS directory I want to use with instance.dfs.dir. Edit: Can I specify a path only in instance.volumes instead of URI which would default to the configuration from hdfs-site (defaultFS, is it?) to maintain backwards compatibility with configurations?
          Hide
          Christopher Tubbs added a comment -

          I think you can specify (or at least, you should be able to) an absolute path only, instead of a full URI, and it should resolve using the default hdfs-configured filesystem.

          Show
          Christopher Tubbs added a comment - I think you can specify (or at least, you should be able to) an absolute path only, instead of a full URI, and it should resolve using the default hdfs-configured filesystem.
          Hide
          Christopher Tubbs added a comment -

          And, given the previous comment I made, "/accumulo" is a reasonable default value for this field.

          Show
          Christopher Tubbs added a comment - And, given the previous comment I made, "/accumulo" is a reasonable default value for this field.
          Hide
          Christopher Tubbs added a comment - - edited

          After some discussion with Keith Turner, I realize it may need to be clarified that the default behavior should be to fall back on whatever the old fields did, if instance.volumes is not specified. Semantically, a default value of "/accumulo" for the new property is the same behavior as the old properties' defaults. If a user doesn't specify anything in the new property, it should default to reading the old properties, but if they actually specified anything here, to include "/accumulo", that needs to comply with the new behavior, and ignore the old properties (except for interpreting relative paths for upgrade).

          (Note: this might require that we be able to tell the difference between a default value of "/accumulo" and a user-specified value of "/accumulo").

          Show
          Christopher Tubbs added a comment - - edited After some discussion with Keith Turner , I realize it may need to be clarified that the default behavior should be to fall back on whatever the old fields did, if instance.volumes is not specified. Semantically, a default value of "/accumulo" for the new property is the same behavior as the old properties' defaults. If a user doesn't specify anything in the new property, it should default to reading the old properties, but if they actually specified anything here, to include "/accumulo", that needs to comply with the new behavior, and ignore the old properties (except for interpreting relative paths for upgrade). (Note: this might require that we be able to tell the difference between a default value of "/accumulo" and a user-specified value of "/accumulo").
          Hide
          Keith Turner added a comment -

          Yeah, it would be confusing if the old props were only used to resolve relative paths, but new files always went into /accumulo if the user does not specify instance.volumes.

          Show
          Keith Turner added a comment - Yeah, it would be confusing if the old props were only used to resolve relative paths, but new files always went into /accumulo if the user does not specify instance.volumes .
          Hide
          Josh Elser added a comment -

          Cool, I think that sounds good.

          Show
          Josh Elser added a comment - Cool, I think that sounds good.
          Hide
          Keith Turner added a comment -

          For 1.6.0, I think we can only deprecate instance.dfs.dir if instance.volumes is changed to not be defined using instance.dfs.dir as suggested in the ticket description. We may be able to deprecate instance.dfs.uri now even if the current behavior in 1.6.0-SNAPSHOT is not changed.

          One other thing to consider is that these properties will be needed as long as the user has relative paths. If a user upgrades to 1.7.0 and still have relative paths, then the props may still be needed.

          Show
          Keith Turner added a comment - For 1.6.0, I think we can only deprecate instance.dfs.dir if instance.volumes is changed to not be defined using instance.dfs.dir as suggested in the ticket description. We may be able to deprecate instance.dfs.uri now even if the current behavior in 1.6.0-SNAPSHOT is not changed. One other thing to consider is that these properties will be needed as long as the user has relative paths. If a user upgrades to 1.7.0 and still have relative paths, then the props may still be needed.
          Hide
          Christopher Tubbs added a comment -

          It might be a good idea to force resolution of relative paths on upgrade, so we don't run into problems handling it in inconsistent ways throughout the code later.

          Show
          Christopher Tubbs added a comment - It might be a good idea to force resolution of relative paths on upgrade, so we don't run into problems handling it in inconsistent ways throughout the code later.
          Hide
          Josh Elser added a comment -

          Just so I'm fully understanding this: by "relative paths", you both are referring to the only the path portion of the URI without the scheme and hostname(:port)? Re-reading this was confusing to me for that reason.

          e.g. Absolute = hdfs://localhost:8020/foobar, Relative = /foobar ?

          Show
          Josh Elser added a comment - Just so I'm fully understanding this: by "relative paths", you both are referring to the only the path portion of the URI without the scheme and hostname(:port)? Re-reading this was confusing to me for that reason. e.g. Absolute = hdfs://localhost:8020/foobar, Relative = /foobar ?
          Hide
          Keith Turner added a comment -

          Josh Elser I am usually thinking of a suffix of the path portion when I mention relative paths. If a file for a tablet is stored in hdfs://foo1:9000/accumulo_test/tables/9/t-003/foo.bar, then prior to 1.6 a tablet would keep track of t-003/foo.bar (or ../9/t-003/foo.bar if were a cloned table). This is what I usually mean by relative path. Prior to 1.6 we would take t-003/foo.bar and put that together with fs uri (from hadoop or accumulo config), accumulo dir (from accumulo config), the table dir constant, and table id (from tablet metadata) to produce a location. Actually the I am using relative path a little more generally, because it also refers to the process needed to resolve walogs paths. For tablets and walogs we switched from late binding to early binding for new paths. Paths created in older releases are still resolved using late binding.

          Show
          Keith Turner added a comment - Josh Elser I am usually thinking of a suffix of the path portion when I mention relative paths. If a file for a tablet is stored in hdfs://foo1:9000/accumulo_test/tables/9/t-003/foo.bar , then prior to 1.6 a tablet would keep track of t-003/foo.bar (or ../9/t-003/foo.bar if were a cloned table). This is what I usually mean by relative path. Prior to 1.6 we would take t-003/foo.bar and put that together with fs uri (from hadoop or accumulo config), accumulo dir (from accumulo config), the table dir constant, and table id (from tablet metadata) to produce a location. Actually the I am using relative path a little more generally, because it also refers to the process needed to resolve walogs paths. For tablets and walogs we switched from late binding to early binding for new paths. Paths created in older releases are still resolved using late binding.
          Hide
          Josh Elser added a comment -

          Ok, that's what I typically expected when I hear relative paths in the context of 1.6, but when dealing with the configuration of these parameters, it wasn't clear to me.

          So what exactly needs to be decided here? Is deprecating instance.dfs.dir and instance.dfs.uri definitely what we want to do for 1.6? It looks like instance.volumes defaults to "" presently which doesn't jive with what Christopher said earlier about defaulting to "/accumulo" – should this be updated?

          As of now, it looks like the "default" volume is still pulled only from instance.dfs.dir (will error if you try it through instance.volumes) and instance.volumes is used for additional volume definitions.

          Show
          Josh Elser added a comment - Ok, that's what I typically expected when I hear relative paths in the context of 1.6, but when dealing with the configuration of these parameters, it wasn't clear to me. So what exactly needs to be decided here? Is deprecating instance.dfs.dir and instance.dfs.uri definitely what we want to do for 1.6? It looks like instance.volumes defaults to "" presently which doesn't jive with what Christopher said earlier about defaulting to "/accumulo" – should this be updated? As of now, it looks like the "default" volume is still pulled only from instance.dfs.dir (will error if you try it through instance.volumes) and instance.volumes is used for additional volume definitions.
          Hide
          Christopher Tubbs added a comment -

          Josh Elser wrote:

          So what exactly needs to be decided here?

          1. instance.dfs.uri is certainly unnecessary outside of upgrades, only needed for resolving the previous relative paths. We can deprecate its use now... no deciding necessary.
          2. instance.dfs.dir is needed to resolve the previous relative paths, but also (currently) as a (in my view) confusing extra element for each volume in instance.volumes... we need to decide if we want it to only be used for resolving the previous relative paths.

          Do we agree that the extra element adds little value as a separate property and is really only needed to resolve relative paths from previous versions?.... OR... do we think that it's useful to configure a common subdirectory with a separate property and that this warrants keeping the current behavior of joining two properties to make absolute paths?

          Personally, if we think there's any value in configuring a common subdirectory, i think we can support that feature by adding property interpolation in configuration parsing in the future. Example:

          subdir = myApplication.accumulo-files
          instance.volumes = hdfs://nn1/${subdir},hdfs://nn2/${subdir}
          

          However, while interpolation is a useful feature... I don't think it's a prerequisite for solidifying the behavior of absolute paths with a single property rather than two.

          Show
          Christopher Tubbs added a comment - Josh Elser wrote: So what exactly needs to be decided here? instance.dfs.uri is certainly unnecessary outside of upgrades, only needed for resolving the previous relative paths. We can deprecate its use now... no deciding necessary. instance.dfs.dir is needed to resolve the previous relative paths, but also (currently) as a (in my view) confusing extra element for each volume in instance.volumes... we need to decide if we want it to only be used for resolving the previous relative paths. Do we agree that the extra element adds little value as a separate property and is really only needed to resolve relative paths from previous versions?.... OR... do we think that it's useful to configure a common subdirectory with a separate property and that this warrants keeping the current behavior of joining two properties to make absolute paths? Personally, if we think there's any value in configuring a common subdirectory, i think we can support that feature by adding property interpolation in configuration parsing in the future. Example: subdir = myApplication.accumulo-files instance.volumes = hdfs: //nn1/${subdir},hdfs://nn2/${subdir} However, while interpolation is a useful feature... I don't think it's a prerequisite for solidifying the behavior of absolute paths with a single property rather than two.
          Hide
          John Vines added a comment -

          It feels weird deprecating something which is required for an upgrade, A.
          B. I think doing a rewrite of all paths from the relative to the absolute will makes for a great simplification, and something that should be done sooner than later. So it should be done as part of the upgrade.

          Show
          John Vines added a comment - It feels weird deprecating something which is required for an upgrade, A. B. I think doing a rewrite of all paths from the relative to the absolute will makes for a great simplification, and something that should be done sooner than later. So it should be done as part of the upgrade.
          Hide
          Christopher Tubbs added a comment -

          John Vines:

          Re A: We've done it before (local walogs directory). That's not to say it was necessarily a great idea, but there is precedent.

          Re B: I agree. I think when the tables come online, they should only see absolute paths. I've voiced this opinion to Keith Turner, who has been spending a lot of time on 118. He seemed to indicate to me that it was easiest to do it when the tablets load and that doing this portion of the upgrade before things are fully online would add a lot of complexity. I'm not sure if he'd like to weigh in on the impact of doing that.

          Show
          Christopher Tubbs added a comment - John Vines : Re A: We've done it before (local walogs directory). That's not to say it was necessarily a great idea, but there is precedent. Re B: I agree. I think when the tables come online, they should only see absolute paths. I've voiced this opinion to Keith Turner , who has been spending a lot of time on 118. He seemed to indicate to me that it was easiest to do it when the tablets load and that doing this portion of the upgrade before things are fully online would add a lot of complexity. I'm not sure if he'd like to weigh in on the impact of doing that.
          Josh Elser made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Hide
          Keith Turner added a comment -

          Anyone opposed to punting on this for 1.6.0? The current behavior may be odd, but it is documented.

          Show
          Keith Turner added a comment - Anyone opposed to punting on this for 1.6.0? The current behavior may be odd, but it is documented.
          Hide
          Mike Drob added a comment -

          Anyone opposed to punting on this for 1.6.0? The current behavior may be odd, but it is documented.

          +1

          Show
          Mike Drob added a comment - Anyone opposed to punting on this for 1.6.0? The current behavior may be odd, but it is documented. +1
          Hide
          John Vines added a comment -

          I'm still thinking that a full rewrite of paths will greatly minimize code complexity down the road for 1.7. Especially if in that release we're going to have refined metadata cleaning utilities, because dealing with two different versions of metadata layout is pretty sucky.

          Show
          John Vines added a comment - I'm still thinking that a full rewrite of paths will greatly minimize code complexity down the road for 1.7. Especially if in that release we're going to have refined metadata cleaning utilities, because dealing with two different versions of metadata layout is pretty sucky.
          Hide
          Josh Elser added a comment -

          I would rather see this as part of 1.6.0. It seems like if we let this ride, it'd just be this wart for those trying to use it.

          I had it in my head that I was going to try to start looking into this today to see how bad it would actually be.

          Show
          Josh Elser added a comment - I would rather see this as part of 1.6.0. It seems like if we let this ride, it'd just be this wart for those trying to use it. I had it in my head that I was going to try to start looking into this today to see how bad it would actually be.
          Hide
          Bill Havanki added a comment -

          +1 to punt.

          Show
          Bill Havanki added a comment - +1 to punt.
          Josh Elser made changes -
          Assignee Josh Elser [ elserj ]
          Hide
          Christopher Tubbs added a comment -

          I think this issue gets to the heart of how users fundamentally understand how to interact with this very important new feature. We will not be able to fundamentally alter this interaction in the future. If we "punt" and do not do this for this release, it will be a significant burden to do it later. I really think it should be done for 1.6.0. If not for 1.6.0, then it probably shouldn't change at all.

          Show
          Christopher Tubbs added a comment - I think this issue gets to the heart of how users fundamentally understand how to interact with this very important new feature. We will not be able to fundamentally alter this interaction in the future. If we "punt" and do not do this for this release, it will be a significant burden to do it later. I really think it should be done for 1.6.0. If not for 1.6.0, then it probably shouldn't change at all.
          Hide
          Josh Elser added a comment -

          Christopher Tubbs that's precisely my concern as well.

          In talking to Keith Turner, I realized that despite the desire to deprecate the two Property}}s, we're stuck keeping {{instance.dfs.dir around because, until we can be absolutely positive that there are no relative paths anywhere, we have to know what the "default" path for a FileSystem is. It doesn't make sense to introduce an extra Property just to do this, so we're stuck using something internally that we really want to avoid.

          To be clear on the self-assignment of this issue, I've started working through some of this, primarily to avoid instance.volumes using instance.dfs.dir when determining where to write in a FileSystem (instead using the fully configured Path). I'll be offline for most of the next week, but I'll try to push up what I have done tonight and will pick it back up then.

          Show
          Josh Elser added a comment - Christopher Tubbs that's precisely my concern as well. In talking to Keith Turner , I realized that despite the desire to deprecate the two Property}}s, we're stuck keeping {{instance.dfs.dir around because, until we can be absolutely positive that there are no relative paths anywhere, we have to know what the "default" path for a FileSystem is. It doesn't make sense to introduce an extra Property just to do this, so we're stuck using something internally that we really want to avoid. To be clear on the self-assignment of this issue, I've started working through some of this, primarily to avoid instance.volumes using instance.dfs.dir when determining where to write in a FileSystem (instead using the fully configured Path). I'll be offline for most of the next week, but I'll try to push up what I have done tonight and will pick it back up then.
          Hide
          ASF subversion and git services added a comment -

          Commit 6f0528e5a63851d8de6b014d780a381ab9ed6d62 in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=6f0528e ]

          ACCUMULO-2061 Re-sync up the changes from upstream. Start fixing things where it is fit.

          Avoiding re-creating the old getFileSystem methods to force us to evaluate
          each usage to be aware of whether or not the base path matters.

          Show
          ASF subversion and git services added a comment - Commit 6f0528e5a63851d8de6b014d780a381ab9ed6d62 in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=6f0528e ] ACCUMULO-2061 Re-sync up the changes from upstream. Start fixing things where it is fit. Avoiding re-creating the old getFileSystem methods to force us to evaluate each usage to be aware of whether or not the base path matters.
          Hide
          ASF subversion and git services added a comment -

          Commit e8a978ca91ae07cf97e4f327462c8ae106a0e675 in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=e8a978c ]

          ACCUMULO-2061 Re-sync up the changes from upstream. Start fixing things where it is fit.

          Avoiding re-creating the old getFileSystem methods to force us to evaluate
          each usage to be aware of whether or not the base path matters.

          Show
          ASF subversion and git services added a comment - Commit e8a978ca91ae07cf97e4f327462c8ae106a0e675 in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=e8a978c ] ACCUMULO-2061 Re-sync up the changes from upstream. Start fixing things where it is fit. Avoiding re-creating the old getFileSystem methods to force us to evaluate each usage to be aware of whether or not the base path matters.
          Hide
          ASF subversion and git services added a comment -

          Commit ce32e3ed8e1801eb282ff0977e413d48b633f31a in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=ce32e3e ]

          ACCUMULO-2061 Get VolumeManagerImpl compiling again

          Show
          ASF subversion and git services added a comment - Commit ce32e3ed8e1801eb282ff0977e413d48b633f31a in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=ce32e3e ] ACCUMULO-2061 Get VolumeManagerImpl compiling again
          Hide
          ASF subversion and git services added a comment -

          Commit 43a7ebda2d02e7482efe1717cabddf2317c2d7b0 in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=43a7ebd ]

          ACCUMULO-2061 Core module builds now

          Show
          ASF subversion and git services added a comment - Commit 43a7ebda2d02e7482efe1717cabddf2317c2d7b0 in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=43a7ebd ] ACCUMULO-2061 Core module builds now
          Josh Elser made changes -
          Link This issue is blocked by ACCUMULO-2442 [ ACCUMULO-2442 ]
          Hide
          ASF subversion and git services added a comment -

          Commit 3eb1593a450e89851a518aa5dccd454753442824 in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=3eb1593 ]

          Merge remote-tracking branch 'origin/1.6.0-SNAPSHOT' into ACCUMULO-2061

          Show
          ASF subversion and git services added a comment - Commit 3eb1593a450e89851a518aa5dccd454753442824 in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=3eb1593 ] Merge remote-tracking branch 'origin/1.6.0-SNAPSHOT' into ACCUMULO-2061
          Hide
          ASF subversion and git services added a comment -

          Commit c3214e0e9433fc35cc822671ba66286e15d38f22 in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c3214e0 ]

          ACCUMULO-2061 Don't store the defaultVolume in with the rest but expect it in the constructor.

          Show
          ASF subversion and git services added a comment - Commit c3214e0e9433fc35cc822671ba66286e15d38f22 in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c3214e0 ] ACCUMULO-2061 Don't store the defaultVolume in with the rest but expect it in the constructor.
          Hide
          ASF subversion and git services added a comment -

          Commit 039989ad44bb9b46d620a3e302e15c32b3b3840a in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=039989a ]

          ACCUMULO-2061 Remove implicit assumptions about '/accumulo' directory that all volumes would have.

          Added some extra logging at trace to help if this needs to be revisited.

          Show
          ASF subversion and git services added a comment - Commit 039989ad44bb9b46d620a3e302e15c32b3b3840a in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=039989a ] ACCUMULO-2061 Remove implicit assumptions about '/accumulo' directory that all volumes would have. Added some extra logging at trace to help if this needs to be revisited.
          Hide
          ASF subversion and git services added a comment -

          Commit 9060185c500b2ab17b99b09c1e1c3cd1791b1c6e in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=9060185 ]

          ACCUMULO-2061 Don't use the default volume, use one of the "real" volumes.

          Show
          ASF subversion and git services added a comment - Commit 9060185c500b2ab17b99b09c1e1c3cd1791b1c6e in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=9060185 ] ACCUMULO-2061 Don't use the default volume, use one of the "real" volumes.
          Hide
          ASF subversion and git services added a comment -

          Commit 2da21b6a450c474115ad8d9c94747116dccf4017 in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=2da21b6 ]

          ACCUMULO-2061 Update FileTypeTest to assert that volumes don't need to have an 'accumulo' directory

          Show
          ASF subversion and git services added a comment - Commit 2da21b6a450c474115ad8d9c94747116dccf4017 in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=2da21b6 ] ACCUMULO-2061 Update FileTypeTest to assert that volumes don't need to have an 'accumulo' directory
          Hide
          ASF subversion and git services added a comment -

          Commit 243aafa494d61c8ad539b6cdef0c688423f4db42 in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=243aafa ]

          ACCUMULO-2061 Update now invalid assertions in VolumeIT

          Show
          ASF subversion and git services added a comment - Commit 243aafa494d61c8ad539b6cdef0c688423f4db42 in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=243aafa ] ACCUMULO-2061 Update now invalid assertions in VolumeIT
          Hide
          Josh Elser added a comment -

          Alright, I have all of the tests running again and have done some poking around locally. Tested the following:

          1. That volume replacement still works (accumulo init --add-volumes and set instance.volumes.replacements)
          2. I can start with instance.dfs.* configuration and switch it to instance.volumes on restart
          3. Tested upgrade case. Adding a volume after upgrade and verified all volumes used
          4. Observed relative paths becoming absolute after compaction

          I need to poke through the final set of changes, but expect a RB tmrw.

          Show
          Josh Elser added a comment - Alright, I have all of the tests running again and have done some poking around locally. Tested the following: That volume replacement still works (accumulo init --add-volumes and set instance.volumes.replacements) I can start with instance.dfs.* configuration and switch it to instance.volumes on restart Tested upgrade case. Adding a volume after upgrade and verified all volumes used Observed relative paths becoming absolute after compaction I need to poke through the final set of changes, but expect a RB tmrw.
          Hide
          ASF subversion and git services added a comment -

          Commit bf5da4423bd8f54491e5b67176cd639ff8273903 in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=bf5da44 ]

          ACCUMULO-2061 Add another test which doesn't start with /accumulo

          Make sure that the garbage collector is expecting anything special like
          /accumulo in the path.

          Show
          ASF subversion and git services added a comment - Commit bf5da4423bd8f54491e5b67176cd639ff8273903 in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=bf5da44 ] ACCUMULO-2061 Add another test which doesn't start with /accumulo Make sure that the garbage collector is expecting anything special like /accumulo in the path.
          Hide
          ASF subversion and git services added a comment -

          Commit 85b5caf418e2693f1b2e554dbbbf671d8f15ab6f in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=85b5caf ]

          ACCUMULO-2061 Excessive logging

          Show
          ASF subversion and git services added a comment - Commit 85b5caf418e2693f1b2e554dbbbf671d8f15ab6f in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=85b5caf ] ACCUMULO-2061 Excessive logging
          Hide
          ASF subversion and git services added a comment -

          Commit db76dd75161c9a6ad94a6a4cc5e1f5656d80e21c in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=db76dd7 ]

          Merge remote-tracking branch 'origin/1.6.0-SNAPSHOT' into ACCUMULO-2061

          Show
          ASF subversion and git services added a comment - Commit db76dd75161c9a6ad94a6a4cc5e1f5656d80e21c in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=db76dd7 ] Merge remote-tracking branch 'origin/1.6.0-SNAPSHOT' into ACCUMULO-2061
          Hide
          ASF subversion and git services added a comment -

          Commit 0b4d50358b3edfa0e42ecd9902c99ce2cdc8bedf in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=0b4d503 ]

          Merge branch '1.6.0-SNAPSHOT' into ACCUMULO-2061

          Show
          ASF subversion and git services added a comment - Commit 0b4d50358b3edfa0e42ecd9902c99ce2cdc8bedf in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=0b4d503 ] Merge branch '1.6.0-SNAPSHOT' into ACCUMULO-2061
          Hide
          ASF subversion and git services added a comment -

          Commit de5c29bc8f029f85f6abafa903b5dd7a9f84e27a in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=de5c29b ]

          ACCUMULO-2061 Use URI instead of FileSystem as the key to find correct Volumes and ensure that absolute URIs
          are still valid even after they are not configured.

          This will help ensure that FileSystem implementations' hashCode and equals don't
          have the potential to collide but still provide unique access back to the Volumes
          contained in the FileSystem. Added tests for the NonConfiguredVolume and also
          for the no-longer-configured volumes.

          Show
          ASF subversion and git services added a comment - Commit de5c29bc8f029f85f6abafa903b5dd7a9f84e27a in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=de5c29b ] ACCUMULO-2061 Use URI instead of FileSystem as the key to find correct Volumes and ensure that absolute URIs are still valid even after they are not configured. This will help ensure that FileSystem implementations' hashCode and equals don't have the potential to collide but still provide unique access back to the Volumes contained in the FileSystem. Added tests for the NonConfiguredVolume and also for the no-longer-configured volumes.
          Hide
          ASF subversion and git services added a comment -

          Commit 45aec91e12744adf0299aee902184a562f111b5b in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=45aec91 ]

          Merge branch '1.6.0-SNAPSHOT' into ACCUMULO-2061

          Show
          ASF subversion and git services added a comment - Commit 45aec91e12744adf0299aee902184a562f111b5b in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=45aec91 ] Merge branch '1.6.0-SNAPSHOT' into ACCUMULO-2061
          Hide
          ASF subversion and git services added a comment -

          Commit 8b8d565b79a83ad6789356a18027cc2c0417b7f0 in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=8b8d565 ]

          ACCUMULO-2061 A few more minor fixes from reviewboard.

          Fixed a PrintInfo comment. Expand the VolumeImpl.isValidPath javadoc
          and implementation to be more encompassing. Suppress warnings in VolumeIT.
          Remove implementation of isValidPath from NonConfiguredVolume as it
          could be misleading.

          Show
          ASF subversion and git services added a comment - Commit 8b8d565b79a83ad6789356a18027cc2c0417b7f0 in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=8b8d565 ] ACCUMULO-2061 A few more minor fixes from reviewboard. Fixed a PrintInfo comment. Expand the VolumeImpl.isValidPath javadoc and implementation to be more encompassing. Suppress warnings in VolumeIT. Remove implementation of isValidPath from NonConfiguredVolume as it could be misleading.
          Hide
          ASF subversion and git services added a comment -

          Commit 3487f8a80810a73fb742382a6ecab609cb518b0b in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=3487f8a ]

          ACCUMULO-2061 Co-locate volumes with the MAC dir

          Losing the volumes on test failure sucks because you can't validate
          if we misplaced a file on the wrong volume in the first place.

          Show
          ASF subversion and git services added a comment - Commit 3487f8a80810a73fb742382a6ecab609cb518b0b in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=3487f8a ] ACCUMULO-2061 Co-locate volumes with the MAC dir Losing the volumes on test failure sucks because you can't validate if we misplaced a file on the wrong volume in the first place.
          Sean Busbey made changes -
          Remote Link This issue links to "reviewboard (Web Link)" [ 14636 ]
          Hide
          Josh Elser added a comment -

          Update, trying to get this wrapped up today, but I get an intermittent test error that Keith is helping me track down.

          Show
          Josh Elser added a comment - Update, trying to get this wrapped up today, but I get an intermittent test error that Keith is helping me track down.
          Hide
          ASF subversion and git services added a comment -

          Commit f5ba4d5ba51697597077d9851a7bde09532758f6 in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f5ba4d5 ]

          ACCUMULO-2061 Fixing whitespace nits

          Show
          ASF subversion and git services added a comment - Commit f5ba4d5ba51697597077d9851a7bde09532758f6 in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f5ba4d5 ] ACCUMULO-2061 Fixing whitespace nits
          Hide
          ASF subversion and git services added a comment -

          Commit e796ffc7ae2195ba8feb877bded675f4c4346f30 in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=e796ffc ]

          ACCUMULO-2061 Make sure we sleep to allow the processes to die before fs rename

          If we don't wait a little bit, it's possible that the MAC processes will
          recreate a file in the directory we just renamed which will cause the test
          to fail despite Accumulo doing the right things.

          Show
          ASF subversion and git services added a comment - Commit e796ffc7ae2195ba8feb877bded675f4c4346f30 in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=e796ffc ] ACCUMULO-2061 Make sure we sleep to allow the processes to die before fs rename If we don't wait a little bit, it's possible that the MAC processes will recreate a file in the directory we just renamed which will cause the test to fail despite Accumulo doing the right things.
          Hide
          ASF subversion and git services added a comment -

          Commit 05553e8fa5e0ab047cd4c8e1d442c1efbaa4bf8b in accumulo's branch refs/heads/ACCUMULO-2061 from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=05553e8 ]

          ACCUMULO-2061 Add a utility method to pull a Path with the instance_id

          Consolidates a little bit of code in one place.

          Show
          ASF subversion and git services added a comment - Commit 05553e8fa5e0ab047cd4c8e1d442c1efbaa4bf8b in accumulo's branch refs/heads/ ACCUMULO-2061 from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=05553e8 ] ACCUMULO-2061 Add a utility method to pull a Path with the instance_id Consolidates a little bit of code in one place.
          Hide
          ASF subversion and git services added a comment -

          Commit 9172f52bc74de2590603c5282c19264447e33da6 in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=9172f52 ]

          ACCUMULO-2061 A few more minor fixes from reviewboard.

          Fixed a PrintInfo comment. Expand the VolumeImpl.isValidPath javadoc
          and implementation to be more encompassing. Suppress warnings in VolumeIT.
          Remove implementation of isValidPath from NonConfiguredVolume as it
          could be misleading.

          Show
          ASF subversion and git services added a comment - Commit 9172f52bc74de2590603c5282c19264447e33da6 in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=9172f52 ] ACCUMULO-2061 A few more minor fixes from reviewboard. Fixed a PrintInfo comment. Expand the VolumeImpl.isValidPath javadoc and implementation to be more encompassing. Suppress warnings in VolumeIT. Remove implementation of isValidPath from NonConfiguredVolume as it could be misleading.
          Hide
          ASF subversion and git services added a comment -

          Commit 53b9f25db94d812f78fd468e88e05da33fdd5190 in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=53b9f25 ]

          ACCUMULO-2061 Co-locate volumes with the MAC dir

          Losing the volumes on test failure sucks because you can't validate
          if we misplaced a file on the wrong volume in the first place.

          Show
          ASF subversion and git services added a comment - Commit 53b9f25db94d812f78fd468e88e05da33fdd5190 in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=53b9f25 ] ACCUMULO-2061 Co-locate volumes with the MAC dir Losing the volumes on test failure sucks because you can't validate if we misplaced a file on the wrong volume in the first place.
          Hide
          ASF subversion and git services added a comment -

          Commit 03baf91611120dfdd36eaa5a824b99144b9eb36a in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=03baf91 ]

          ACCUMULO-2061 Fixing whitespace nits

          Show
          ASF subversion and git services added a comment - Commit 03baf91611120dfdd36eaa5a824b99144b9eb36a in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=03baf91 ] ACCUMULO-2061 Fixing whitespace nits
          Hide
          ASF subversion and git services added a comment -

          Commit c25a41fe905cda4aa914f232db5cc26b3254f4a0 in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c25a41f ]

          ACCUMULO-2061 Add a utility method to pull a Path with the instance_id

          Consolidates a little bit of code in one place.

          Show
          ASF subversion and git services added a comment - Commit c25a41fe905cda4aa914f232db5cc26b3254f4a0 in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c25a41f ] ACCUMULO-2061 Add a utility method to pull a Path with the instance_id Consolidates a little bit of code in one place.
          Hide
          ASF subversion and git services added a comment -

          Commit 53b9f25db94d812f78fd468e88e05da33fdd5190 in accumulo's branch refs/heads/master from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=53b9f25 ]

          ACCUMULO-2061 Co-locate volumes with the MAC dir

          Losing the volumes on test failure sucks because you can't validate
          if we misplaced a file on the wrong volume in the first place.

          Show
          ASF subversion and git services added a comment - Commit 53b9f25db94d812f78fd468e88e05da33fdd5190 in accumulo's branch refs/heads/master from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=53b9f25 ] ACCUMULO-2061 Co-locate volumes with the MAC dir Losing the volumes on test failure sucks because you can't validate if we misplaced a file on the wrong volume in the first place.
          Hide
          ASF subversion and git services added a comment -

          Commit 03baf91611120dfdd36eaa5a824b99144b9eb36a in accumulo's branch refs/heads/master from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=03baf91 ]

          ACCUMULO-2061 Fixing whitespace nits

          Show
          ASF subversion and git services added a comment - Commit 03baf91611120dfdd36eaa5a824b99144b9eb36a in accumulo's branch refs/heads/master from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=03baf91 ] ACCUMULO-2061 Fixing whitespace nits
          Hide
          ASF subversion and git services added a comment -

          Commit c25a41fe905cda4aa914f232db5cc26b3254f4a0 in accumulo's branch refs/heads/master from Josh Elser
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c25a41f ]

          ACCUMULO-2061 Add a utility method to pull a Path with the instance_id

          Consolidates a little bit of code in one place.

          Show
          ASF subversion and git services added a comment - Commit c25a41fe905cda4aa914f232db5cc26b3254f4a0 in accumulo's branch refs/heads/master from Josh Elser [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c25a41f ] ACCUMULO-2061 Add a utility method to pull a Path with the instance_id Consolidates a little bit of code in one place.
          Josh Elser made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Josh Elser made changes -
          Link This issue is blocked by ACCUMULO-2442 [ ACCUMULO-2442 ]
          Hide
          Christopher Tubbs added a comment -

          The previous patch introduced a ton of new deprecation warnings for using the newly deprecated properties. Those warnings need to be dealt with, preferably by isolating the code which accesses them to a utility method, with a using a single suppression.

          If necessary, we could make a new ticket. I was going to make a sub-task, rather than reopen this one, but this is already a sub-task.

          Show
          Christopher Tubbs added a comment - The previous patch introduced a ton of new deprecation warnings for using the newly deprecated properties. Those warnings need to be dealt with, preferably by isolating the code which accesses them to a utility method, with a using a single suppression. If necessary, we could make a new ticket. I was going to make a sub-task, rather than reopen this one, but this is already a sub-task.
          Christopher Tubbs made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          Josh Elser added a comment -

          I'd prefer a new ticket for handling the deprecation warnings. Deprecating was the point of this ticket. I'll make a new one and link it to this.

          Show
          Josh Elser added a comment - I'd prefer a new ticket for handling the deprecation warnings. Deprecating was the point of this ticket. I'll make a new one and link it to this.
          Josh Elser made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Josh Elser made changes -
          Link This issue is related to ACCUMULO-2527 [ ACCUMULO-2527 ]
          Hide
          ASF subversion and git services added a comment -

          Commit 59db6fedd3aea9615be156d1f7a1be08e0de0a86 in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Christopher Tubbs
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=59db6fe ]

          ACCUMULO-2061 Remove useless javadoc (tag without description)

          Show
          ASF subversion and git services added a comment - Commit 59db6fedd3aea9615be156d1f7a1be08e0de0a86 in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Christopher Tubbs [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=59db6fe ] ACCUMULO-2061 Remove useless javadoc (tag without description)
          Hide
          ASF subversion and git services added a comment -

          Commit 59db6fedd3aea9615be156d1f7a1be08e0de0a86 in accumulo's branch refs/heads/master from Christopher Tubbs
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=59db6fe ]

          ACCUMULO-2061 Remove useless javadoc (tag without description)

          Show
          ASF subversion and git services added a comment - Commit 59db6fedd3aea9615be156d1f7a1be08e0de0a86 in accumulo's branch refs/heads/master from Christopher Tubbs [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=59db6fe ] ACCUMULO-2061 Remove useless javadoc (tag without description)

            People

            • Assignee:
              Josh Elser
              Reporter:
              Christopher Tubbs
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development