Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2600

regarding HDFS-RAID

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Invalid
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: contrib/raid
    • Labels:

      Description

      I've tried to do HDFS-RAID over hadoop-0.21.0 ...

      But, It's failed to raid when I've tried to start raidnode (start-raidnode.sh)

      What do I have to correct it?

      This is the information. Would you tell me how to correct it?

      Namenode : xxx.23.129.40:9006
      Datanode: xxx.23.129.39, 38
      Raidnode : xxx.23.129.40 ( I want to seperate raidnoe from namenode, how can I do?)

      • hdfs-site.xml
        <name>hdfs.raid.locations</name>
        <value>hdfs://xxx.23.129.40:9006/raid</value>
        => (What's the meaning of this, which address do I have to write down?)
        <description>The location for parity files. If this is
        is not defined, then defaults to /raid.
        </description>
        </property>
      • raid.xml
        <configuration>
        <srcPath prefix="hdfs://xxx.23.129.40:9006/TEST">
        => (What's the meaning of this, which address do I have to write down?)
        <policy name = "TEST"> => (What's this? )
        <property>
        <name>srcReplication</name>
        <value>3</value>
        <description> pick files for RAID only if their replication factor is
        greater than or equal to this value.
        </description>
        </property>
        <property>
        <name>targetReplication</name>
        <value>2</value>
        <description> after RAIDing, decrease the replication factor of a file to
        this value.
        </description>
        </property>
        <property>
        <name>metaReplication</name>
        <value>2</value>
        <description> the replication factor of the RAID meta file
        </description>
        </property>
        <property>
        <name>modTimePeriod</name>
        <value>100</value>
        <description> time (milliseconds) after a file is modified to make it a
        candidate for RAIDing
        </description>
        </property>

      *log: when start-raidnode.sh

      /************************************************************
      SHUTDOWN_MSG: Shutting down RaidNode at test40/xxx.23.129.40
      ************************************************************/
      2011-11-25 23:59:13,417 INFO org.apache.hadoop.raid.RaidNode: STARTUP_MSG:
      /************************************************************
      STARTUP_MSG: Starting RaidNode
      STARTUP_MSG: host = test40/xxx.23.129.40
      STARTUP_MSG: args = []
      STARTUP_MSG: version = 0.21.0
      ************************************************************/
      2011-11-25 23:59:13,518 ERROR org.apache.hadoop.raid.ConfigManager: Reloading config

      file /KTBHOME/hadoop/hadoop/conf/raid.xml
      2011-11-25 23:59:13,611 INFO org.apache.hadoop.security.Groups: Group mapping

      impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
      2011-11-25 23:59:13,647 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is

      deprecated. Instead, use mapreduce.task.attempt.id
      2011-11-25 23:59:13,714 INFO org.apache.hadoop.raid.ConfigManager:

      TEST.srcReplication = 3
      2011-11-25 23:59:13,714 INFO org.apache.hadoop.raid.ConfigManager:

      TEST.targetReplication = 2
      2011-11-25 23:59:13,715 INFO org.apache.hadoop.raid.ConfigManager:

      TEST.metaReplication = 2
      2011-11-25 23:59:13,715 INFO org.apache.hadoop.raid.ConfigManager:

      TEST.modTimePeriod = 100
      2011-11-25 23:59:13,716 INFO org.apache.hadoop.raid.ConfigManager:

      k_table1.targetReplication = 1
      2011-11-25 23:59:13,716 INFO org.apache.hadoop.raid.ConfigManager:

      k_table1.metaReplication = 2
      2011-11-25 23:59:13,716 INFO org.apache.hadoop.raid.ConfigManager:

      k_table1.modTimePeriod = 100
      2011-11-25 23:59:13,728 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
      2011-11-25 23:59:13,751 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing

      RPC Metrics with hostName=RaidNode, port=60000
      2011-11-25 23:59:13,810 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics:

      Initializing RPC Metrics with hostName=RaidNode, port=60000
      2011-11-25 23:59:13,812 INFO org.apache.hadoop.raid.RaidNode: RaidNode up at:

      /127.0.0.1:60000
      2011-11-25 23:59:13,813 INFO org.apache.hadoop.ipc.Server: IPC Server Responder:

      starting
      2011-11-25 23:59:13,813 INFO org.apache.hadoop.ipc.Server: IPC Server listener on

      60000: starting
      2011-11-25 23:59:13,814 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on

      60000: starting
      2011-11-25 23:59:13,814 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on

      60000: starting
      2011-11-25 23:59:13,815 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on

      60000: starting
      2011-11-25 23:59:13,815 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on

      60000: starting
      2011-11-25 23:59:13,815 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on

      60000: starting
      2011-11-25 23:59:13,815 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on

      60000: starting
      2011-11-25 23:59:13,815 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on

      60000: starting
      2011-11-25 23:59:13,816 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on

      60000: starting
      2011-11-25 23:59:13,816 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on

      60000: starting
      2011-11-25 23:59:13,816 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on

      60000: starting
      2011-11-25 23:59:13,818 INFO org.apache.hadoop.raid.RaidNode: Triggering Policy

      Filter TEST hdfs://xxx.23.129.40:9006/TEST
      2011-11-25 23:59:13,819 INFO org.apache.hadoop.raid.RaidNode: Started archive scan
      2011-11-25 23:59:13,828 INFO org.apache.hadoop.raid.RaidNode: No filtered paths for

      policy TEST
      2011-11-25 23:59:13,828 INFO org.apache.hadoop.raid.RaidNode: Triggering Policy
      Filter k_table1 hdfs://xxx.23.129.40:9006
      2011-11-25 23:59:13,829 INFO org.apache.hadoop.raid.RaidNode: No filtered paths for
      policy k_table1

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              leiental kiupbyun
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 120h
                120h
                Remaining:
                Remaining Estimate - 120h
                120h
                Logged:
                Time Spent - Not Specified
                Not Specified