[ACCUMULO-118] accumulo could work across HDFS instances, which would help it to scale past a single namenode

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: master, tserver
• Assignee: Eric C. Newton
• Reporter: Eric C. Newton
• Created: 04/Nov/11 16:09
• Updated: 06/Feb/15 17:42
• Resolved: 27/Mar/14 17:45

Description

Consider using full path names to files, which would allow the servers to access the files on any HDFS file system.

Work may exist elsewhere to run HDFS using a number of NameNode instances to break up the namespace.

We may need a pluggable strategy to determine namespace for new files.

Issue Links

• is related to: ACCUMULO-2274 AddFilesWithMissingEntries needs to support multiple volumes
• is related to: ACCUMULO-2974 Unable to assign single tablet table migrated to 1.6.0
• is related to: ACCUMULO-1902 DefaultSecretKeyEncryptionStrategy will not work w/ multiple namenodes
• is related to: ACCUMULO-2530 Update crypto properties and encryption strategies to be volume-aware
• is related to: ACCUMULO-1524 Abstract file pointers in metadata table
• is related to: ACCUMULO-2552 Check for usage of FileSystem.get(Configuration)
• is related to: ACCUMULO-3535 Transition plan: remove instance.dfs.{uri,dir} to instance.volumes
• relates to: ACCUMULO-1747 ServerConstants.getBaseDirs() use fs default name, not instance.dfs.uri
• relates to: ACCUMULO-1685 bench testing shows that the NN loses the WAL
• relates to: ACCUMULO-722 Accumulo using Accumulo as its own NameNode

Sub-Tasks

• create a utility for converting the !METADATA table entry to a full path: Resolved
• ServerConstants.getBaseDirs() use fs default name, not instance.dfs.uri: Resolved
• Get disk usage will not work across namenodes: Resolved
• Offline map reduce will fail if tablet spans multiple namenodes: Resolved
• RFile printinfo command does not handle multiple namenodes: Resolved
• Monitor collects disk usage from single namenode: Resolved
• Bulk import does sanity checks on client side using a single filesystem: Resolved
• Changing Accumulo config can prevent locating root table files.: Resolved
• Analyze all usages of instance.dfs.uri and instance.volumes: Resolved
• Instance id and version info only stored on one hdfs instance: Resolved
• Need utility to decommission dfs uris: Resolved
• Garbage collector may delete referenced files after upgrade: Resolved
• Write ahead logs from upgrade prematurely GCed: Resolved
• Create utility for rewriting uris: Resolved
• WALog entries not properly deleted after recovery during upgrade 1.5 -> 1.6: Resolved
• Accumulo fails when instance.volumes and fs.default.name are disjoint: Resolved
• Recommend using viewfs:// or HA Namenode: Resolved
• Deprecate instance.dfs.uri and instance.dfs.dir: Resolved
• general.volume.chooser has no environment: Resolved
• Log recovery is converting paths to relative: Resolved
• updateAccumuloVersion only updates the data version on a single configured volume: Resolved
• Copying failed bulk imports seems broken: Resolved
• Suppress expected deprecation warnings from instance.dfs.{dir,uri}: Resolved
• FileUtil expects instance.dfs.dir in tmpDir path: Resolved
• Update ClientOpts to read from volumes or instance dir: Resolved