Accumulo
  1. Accumulo
  2. ACCUMULO-118

accumulo could work across HDFS instances, which would help it to scale past a single namenode

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: master, tserver
    • Labels:
      None

      Description

      Consider using full path names to files, which would allow the servers to access the files on any HDFS file system.

      Work may exist elsewhere to run HDFS using a number of NameNode instances to break up the namespace.

      We may need a pluggable strategy to determine namespace for new files.

      1. ACCUMULO-118-02.txt
        6 kB
        Eric Newton
      2. ACCUMULO-118-01.txt
        5 kB
        Eric Newton

        Issue Links

        1.
        create a utility for converting the !METADATA table entry to a full path Sub-task Resolved Unassigned  
         
        2.
        ServerConstants.getBaseDirs() use fs default name, not instance.dfs.uri Sub-task Resolved Keith Turner  
         
        3.
        Get disk usage will not work across namenodes Sub-task Resolved Eric Newton  
         
        4.
        Offline map reduce will fail if tablet spans multiple namenodes Sub-task Resolved Keith Turner  
         
        5.
        RFile printinfo command does not handle multiple namenodes Sub-task Resolved Keith Turner  
         
        6.
        Monitor collects disk usage from single namenode Sub-task Resolved Josh Elser  
         
        7.
        Bulk import does sanity checks on client side using a single filesystem Sub-task Resolved Keith Turner  
         
        8.
        Changing Accumulo config can prevent locating root table files. Sub-task Resolved Keith Turner  
         
        9.
        Analyze all usages of instance.dfs.uri and instance.volumes Sub-task Resolved Keith Turner  
         
        10.
        Instance id and version info only stored on one hdfs instance Sub-task Resolved Keith Turner  
         
        11.
        Need utility to decommission dfs uris Sub-task Resolved Keith Turner  
         
        12.
        Garbage collector may delete referenced files after upgrade Sub-task Resolved Keith Turner  
         
        13.
        Write ahead logs from upgrade prematurely GCed Sub-task Resolved Eric Newton  
         
        14.
        Create utility for rewriting uris Sub-task Resolved Keith Turner

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
         
        15.
        WALog entries not properly deleted after recovery during upgrade 1.5 -> 1.6 Sub-task Resolved Eric Newton  
         
        16.
        Accumulo fails when instance.volumes and fs.default.name are disjoint Sub-task Resolved Keith Turner  
         
        17.
        Recommend using viewfs:// or HA Namenode Sub-task Resolved Keith Turner  
         
        18.
        Deprecate instance.dfs.uri and instance.dfs.dir Sub-task Resolved Josh Elser  
         
        19.
        general.volume.chooser has no environment Sub-task Resolved Eric Newton  
         
        20.
        Log recovery is converting paths to relative Sub-task Resolved Keith Turner  
         
        21.
        updateAccumuloVersion only updates the data version on a single configured volume Sub-task Resolved Josh Elser  
         
        22.
        Copying failed bulk imports seems broken Sub-task Resolved Eric Newton  
         
        23.
        Suppress expected deprecation warnings from instance.dfs.{dir,uri} Sub-task Resolved Josh Elser  
         
        24.
        FileUtil expects instance.dfs.dir in tmpDir path Sub-task Resolved Josh Elser  
         
        25.
        Update ClientOpts to read from volumes or instance dir Sub-task Resolved Josh Elser  
         

          Activity

          Eric Newton created issue -
          Eric Newton made changes -
          Field Original Value New Value
          Original Estimate 0.05h [ 180 ] 2,016h [ 7257600 ]
          Remaining Estimate 0.05h [ 180 ] 2,016h [ 7257600 ]
          Gavin made changes -
          Workflow no-reopen-closed, patch-avail [ 12641050 ] patch-available, re-open possible [ 12671517 ]
          Eric Newton made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Christopher Tubbs made changes -
          Affects Version/s 1.5.0 [ 12318645 ]
          Josh Elser made changes -
          Fix Version/s 1.6.0 [ 12322468 ]
          Eric Newton made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Eric Newton made changes -
          Priority Critical [ 2 ] Blocker [ 1 ]
          Eric Newton made changes -
          Attachment ACCUMULO-118-01.txt [ 12584591 ]
          Eric Newton made changes -
          Attachment ACCUMULO-118-02.txt [ 12586164 ]
          Christopher Tubbs made changes -
          Link This issue is related to ACCUMULO-1524 [ ACCUMULO-1524 ]
          Eric Newton made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Eric Newton made changes -
          Link This issue relates to ACCUMULO-1685 [ ACCUMULO-1685 ]
          Josh Elser made changes -
          Link This issue relates to ACCUMULO-1747 [ ACCUMULO-1747 ]
          Keith Turner made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Eric Newton made changes -
          Status Reopened [ 4 ] Reopened [ 4 ]
          Keith Turner made changes -
          Link This issue depends upon ACCUMULO-1771 [ ACCUMULO-1771 ]
          Keith Turner made changes -
          Link This issue depends upon ACCUMULO-1771 [ ACCUMULO-1771 ]
          Keith Turner made changes -
          Comment [ Need to fix ACCUMULO-1771 before fixing this ]
          John Vines made changes -
          Assignee Eric Newton [ ecn ] John Vines [ vines ]
          John Vines made changes -
          Assignee John Vines [ vines ] Eric Newton [ ecn ]
          Sean Busbey made changes -
          Link This issue is related to ACCUMULO-2274 [ ACCUMULO-2274 ]
          Sean Busbey made changes -
          Link This issue is related to ACCUMULO-1902 [ ACCUMULO-1902 ]
          Josh Elser made changes -
          Link This issue is related to ACCUMULO-2552 [ ACCUMULO-2552 ]
          Sean Busbey made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Eric Newton made changes -
          Link This issue relates to ACCUMULO-722 [ ACCUMULO-722 ]
          John Vines made changes -
          Link This issue is related to ACCUMULO-2974 [ ACCUMULO-2974 ]
          Christopher Tubbs made changes -
          Link This issue is related to ACCUMULO-3535 [ ACCUMULO-3535 ]
          Christopher Tubbs made changes -
          Link This issue is related to ACCUMULO-2530 [ ACCUMULO-2530 ]

            People

            • Assignee:
              Eric Newton
              Reporter:
              Eric Newton
            • Votes:
              3 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 2,016h Original Estimate - 2,016h
                2,016h
                Remaining:
                Remaining Estimate - 2,016h
                2,016h
                Logged:
                Remaining Estimate - 2,016h
                0.5h

                  Development