Hadoop Common
  1. Hadoop Common
  2. HADOOP-5670

Hadoop configurations should be read from a distributed system

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: conf
    • Labels:
      None

      Description

      Rather than distributing the hadoop configuration files to every data node, compute node, etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper, whatever.

        Issue Links

          Activity

          Hide
          Jakob Homan added a comment -

          Odd. I replied as a comment but jira didn't seem to pick it up...

          Agreed. It would still be necessary to support local configuration for single-node clusters and testing though. The configuration should be able to take its values either from a distributed system, or from local values. ZooKeeper would be a reasonable system for retrieving/setting the key-value pairs of the configuration file.

          Show
          Jakob Homan added a comment - Odd. I replied as a comment but jira didn't seem to pick it up... Agreed. It would still be necessary to support local configuration for single-node clusters and testing though. The configuration should be able to take its values either from a distributed system, or from local values. ZooKeeper would be a reasonable system for retrieving/setting the key-value pairs of the configuration file.
          Hide
          dhruba borthakur added a comment -

          One aproach would be to wrap LDAP, Zookeeper, etc via an implementation of org.apache.hadoop.FileSystem.

          Show
          dhruba borthakur added a comment - One aproach would be to wrap LDAP, Zookeeper, etc via an implementation of org.apache.hadoop.FileSystem.
          Hide
          Allen Wittenauer added a comment -

          I think it goes without saying that a pluggable system would be great here. LDAP and ZK are just two examples of many that could be used to provide this sort of functionality. Heck, no reason it couldn't be as simple as a HTTP GET.

          [I'll admit i'm slightly partial to LDAP here due to the requirement to provide a schema. It provides a rudimentary way to enforce some basic rules on the data prior to Hadoop even seeing it.]

          Show
          Allen Wittenauer added a comment - I think it goes without saying that a pluggable system would be great here. LDAP and ZK are just two examples of many that could be used to provide this sort of functionality. Heck, no reason it couldn't be as simple as a HTTP GET. [I'll admit i'm slightly partial to LDAP here due to the requirement to provide a schema. It provides a rudimentary way to enforce some basic rules on the data prior to Hadoop even seeing it.]
          Hide
          Konstantin Shvachko added a comment -

          Do you think LDAP will scale if 2000 data-nodes will start reading their conf at once?

          Show
          Konstantin Shvachko added a comment - Do you think LDAP will scale if 2000 data-nodes will start reading their conf at once?
          Hide
          Allen Wittenauer added a comment -

          Like Hadoop, ZK, webservices, etc, LDAP's scale is implementation dependent.

          Show
          Allen Wittenauer added a comment - Like Hadoop, ZK, webservices, etc, LDAP's scale is implementation dependent.
          Hide
          Philip Zeyliger added a comment -

          I'd love to be able to specify a configuration URL, instead of distributing hadoop-site.xml all over. Presumably you would specify the URL as "http://host/getConfig" and Hadoop would add some parameters, like "http://host/getConfig?host=xyz&name=foo".

          Show
          Philip Zeyliger added a comment - I'd love to be able to specify a configuration URL, instead of distributing hadoop-site.xml all over. Presumably you would specify the URL as "http://host/getConfig" and Hadoop would add some parameters, like "http://host/getConfig?host=xyz&name=foo".
          Hide
          Jakob Homan added a comment -

          Wrapping filesystem around ldap/zk/etc seems a bit heavy. Another option would be to extend Configuration and create ZooKeeperConfiguration, LDAPConfiguration, HTTPConfiguration, etc. Each node would still require a small file to tell it where/which resource to locate. A builder could read this file and construct the appropriate configuration and hand it off to the node. This would be very pluggable and transparent to the nodes.

          Show
          Jakob Homan added a comment - Wrapping filesystem around ldap/zk/etc seems a bit heavy. Another option would be to extend Configuration and create ZooKeeperConfiguration, LDAPConfiguration, HTTPConfiguration, etc. Each node would still require a small file to tell it where/which resource to locate. A builder could read this file and construct the appropriate configuration and hand it off to the node. This would be very pluggable and transparent to the nodes.
          Hide
          Konstantin Shvachko added a comment -

          Centralized configuration facility adds one more single point of failure. If LDAP server fails - everything fails.
          I am not arguing against centralized configurators in general. Just pointing out possible issues.

          Configuration may be just the first step. Then we may want to build centralize hadoop distribution repositories from which cluster nodes can pick required jars, etc.

          Show
          Konstantin Shvachko added a comment - Centralized configuration facility adds one more single point of failure. If LDAP server fails - everything fails. I am not arguing against centralized configurators in general. Just pointing out possible issues. Configuration may be just the first step. Then we may want to build centralize hadoop distribution repositories from which cluster nodes can pick required jars, etc.
          Hide
          Konstantin Shvachko added a comment -

          > Each node would still require a small file to tell it where/which resource to locate.

          Yes and that makes me think that something is not right here: in order obtain data-node configuration I need to specify the location of the configuration server and still distribute this configuration file to 2000 nodes.

          Show
          Konstantin Shvachko added a comment - > Each node would still require a small file to tell it where/which resource to locate. Yes and that makes me think that something is not right here: in order obtain data-node configuration I need to specify the location of the configuration server and still distribute this configuration file to 2000 nodes.
          Hide
          Allen Wittenauer added a comment -

          The amount of churn in a configuration file that only points to where to read the real configuration data is significantly lower than changes in Hadoop's configuration in my experience. For example, our system level ldap.conf has changed maybe three times in 1 year, and most of those were growing pains of a new infrastructure/data center. I have lost track of how many times we have needed to change and bounce the jobtracker ACLs in the past week.

          Additionally, HTTP, LDAP, etc, have real, proven HA solutions. They are much less likely to go down than the Hadoop grids they would be supporting....

          Show
          Allen Wittenauer added a comment - The amount of churn in a configuration file that only points to where to read the real configuration data is significantly lower than changes in Hadoop's configuration in my experience. For example, our system level ldap.conf has changed maybe three times in 1 year, and most of those were growing pains of a new infrastructure/data center. I have lost track of how many times we have needed to change and bounce the jobtracker ACLs in the past week . Additionally, HTTP, LDAP, etc, have real, proven HA solutions. They are much less likely to go down than the Hadoop grids they would be supporting....
          Hide
          Todd Lipcon added a comment -

          Another nice thing about this is that you can use DNS for a super-simple HA/metaconfig system. Simply point all the hadoop nodes at http://confmaster/config, then set up CNAME or A records on the local domain network to the right machines. If one confmaster is down, it'll just pull from another.

          Show
          Todd Lipcon added a comment - Another nice thing about this is that you can use DNS for a super-simple HA/metaconfig system. Simply point all the hadoop nodes at http://confmaster/config , then set up CNAME or A records on the local domain network to the right machines. If one confmaster is down, it'll just pull from another.
          Hide
          Mahadev konar added a comment -

          making a case for zookeeper , to write a configuration layer on top of zookeeper would be just 20-30 lines of code and we can handle around >20K writes per second (with 3 servers, which I dont think would be necessary but other apps also could use zookeeper) and >50K reads/sec. Also zookeeper has a pretty strong HA story.

          Show
          Mahadev konar added a comment - making a case for zookeeper , to write a configuration layer on top of zookeeper would be just 20-30 lines of code and we can handle around >20K writes per second (with 3 servers, which I dont think would be necessary but other apps also could use zookeeper) and >50K reads/sec. Also zookeeper has a pretty strong HA story.
          Hide
          dhruba borthakur added a comment -

          Also, this could sow the seeds of a true federation of hdfs clusters!

          Show
          dhruba borthakur added a comment - Also, this could sow the seeds of a true federation of hdfs clusters!
          Hide
          Amr Awadallah added a comment -

          I like the zookeeper option, especially the HA part.

          – amr

          Show
          Amr Awadallah added a comment - I like the zookeeper option, especially the HA part. – amr
          Hide
          Andrey Kuzmin added a comment -

          Couple of suggestions (assuming zookeeper option) based on past experience with similar issue.

          1. Splitting configuration into two parts - default and site-specific overrides (with further specialization down the line) - would simplify hadoop upgrade by minimizing upgrade impact on locally overridden options.

          2. If zookeeper does not provide for node-local persistent caching, adding this option specifically for configuration data could support disconnected node operations if need be (say, for sanity checks on start-up).

          Show
          Andrey Kuzmin added a comment - Couple of suggestions (assuming zookeeper option) based on past experience with similar issue. 1. Splitting configuration into two parts - default and site-specific overrides (with further specialization down the line) - would simplify hadoop upgrade by minimizing upgrade impact on locally overridden options. 2. If zookeeper does not provide for node-local persistent caching, adding this option specifically for configuration data could support disconnected node operations if need be (say, for sanity checks on start-up).
          Hide
          Allen Wittenauer added a comment -

          #1 could be done no matter what the source. It just depends upon how smart the plug in framework and the actual plug in is. For example, assuming an HTTP plug in: it would just fetch two files and do the merge just like Hadoop configures things today.

          #2 should be done no matter what the source. The question is whether it should be handled inside or outside the plug in.

          The follow up to this bug is really: how do you build a registry of configs and have the client smart enough to know which entry it needs to follow. So that might need to be part of the design here. [For example, if I have a client machine that needs to submit jobs to two different grids, how can it automagically pull the proper configuration information for those two grids? ]

          Show
          Allen Wittenauer added a comment - #1 could be done no matter what the source. It just depends upon how smart the plug in framework and the actual plug in is. For example, assuming an HTTP plug in: it would just fetch two files and do the merge just like Hadoop configures things today. #2 should be done no matter what the source. The question is whether it should be handled inside or outside the plug in. The follow up to this bug is really: how do you build a registry of configs and have the client smart enough to know which entry it needs to follow. So that might need to be part of the design here. [For example, if I have a client machine that needs to submit jobs to two different grids, how can it automagically pull the proper configuration information for those two grids? ]
          Hide
          Andrey Kuzmin added a comment -

          > For example, assuming an HTTP plug in: it would just fetch two files and do the merge
          > just like Hadoop configures things today.
          Yes, this is exactly what a simplistic solution would do. I'd rather not limit myself to two files, though: why not get ready for federated grids in advance. Further, one has to take some care with option semantics: for instance, override of some default options may turn undesirable or even should be prohibited. There some nits here, to summarize.

          > [For example, if I have a client machine that needs to submit jobs to two different grids,
          > how can it automagically pull the proper configuration information for those two grids? ]
          I didn't actually consider clients. If ZK supports client connections (meaning outside-world readers and may be even writers), not sure with this - just started reading docs/code last week, - this should be fairly straightforward. The only thing one then needs at the client to answer your concern would be a simple "AvailableGrids" config listing respective ZK URLs to connect to.

          Show
          Andrey Kuzmin added a comment - > For example, assuming an HTTP plug in: it would just fetch two files and do the merge > just like Hadoop configures things today. Yes, this is exactly what a simplistic solution would do. I'd rather not limit myself to two files, though: why not get ready for federated grids in advance. Further, one has to take some care with option semantics: for instance, override of some default options may turn undesirable or even should be prohibited. There some nits here, to summarize. > [For example, if I have a client machine that needs to submit jobs to two different grids, > how can it automagically pull the proper configuration information for those two grids? ] I didn't actually consider clients. If ZK supports client connections (meaning outside-world readers and may be even writers), not sure with this - just started reading docs/code last week, - this should be fairly straightforward. The only thing one then needs at the client to answer your concern would be a simple "AvailableGrids" config listing respective ZK URLs to connect to.
          Hide
          Philip Zeyliger added a comment -

          HADOOP-4944 contains a patch to use "xinclude" in configuration files. That seems functionally similar to the goals of this ticket.

          Thoughts?

          Show
          Philip Zeyliger added a comment - HADOOP-4944 contains a patch to use "xinclude" in configuration files. That seems functionally similar to the goals of this ticket. Thoughts?
          Hide
          Tom White added a comment -

          XInclude from a remote filesystem? I'm worried that XML parsers won't
          handle errors well. Perhaps it could be made to work.

          There are two pieces to this ticket anyway. 1. read config files from
          an arbitrary filesystem. 2. get daemons to re-read config more
          dynamically - most of them just do so once at startup. I would have
          separate issues for these.

          I had some conversations with Doug on this in the summer, but I never
          got round to even opening a ticket . The idea was to expose ZK as a
          Hadoop FileSystem then change Configuration to load from an arbitrary
          FS. There is a bootstraping issue to get round too.

          Tom

          Show
          Tom White added a comment - XInclude from a remote filesystem? I'm worried that XML parsers won't handle errors well. Perhaps it could be made to work. There are two pieces to this ticket anyway. 1. read config files from an arbitrary filesystem. 2. get daemons to re-read config more dynamically - most of them just do so once at startup. I would have separate issues for these. I had some conversations with Doug on this in the summer, but I never got round to even opening a ticket . The idea was to expose ZK as a Hadoop FileSystem then change Configuration to load from an arbitrary FS. There is a bootstraping issue to get round too. Tom
          Hide
          Allen Wittenauer added a comment -

          Given ZK's status as a subproject, where does ZK read its configuration data from? Do we have a chicken and egg problem?

          Show
          Allen Wittenauer added a comment - Given ZK's status as a subproject, where does ZK read its configuration data from? Do we have a chicken and egg problem?
          Hide
          Tom White added a comment -

          ZK uses Java properties files, so it has no dependencies on Hadoop Core configuration.

          The bootstrapping problem is that FileSystem relies on a Configuration object having loaded its resources. So to get a Configuration to load from a FileSystem won't work, the way things stand (see Configuration#loadResource). (It's possible that the reason for this code has gone away, or at least lessened, following the recent split of configurations into core, hdfs and mapred - it might be possible to bootstrap from the core config.) There's no inherent reason, though, why a Configuration couldn't load its resources from HDFS or a ZK-backed FileSystem. We just need to avoid a system loading resources from itself.

          Show
          Tom White added a comment - ZK uses Java properties files, so it has no dependencies on Hadoop Core configuration. The bootstrapping problem is that FileSystem relies on a Configuration object having loaded its resources. So to get a Configuration to load from a FileSystem won't work, the way things stand (see Configuration#loadResource). (It's possible that the reason for this code has gone away, or at least lessened, following the recent split of configurations into core, hdfs and mapred - it might be possible to bootstrap from the core config.) There's no inherent reason, though, why a Configuration couldn't load its resources from HDFS or a ZK-backed FileSystem. We just need to avoid a system loading resources from itself.
          Hide
          Mahadev konar added a comment -

          There are two pieces to this ticket anyway. 1. read config files from
          an arbitrary filesystem. 2. get daemons to re-read config more
          dynamically - most of them just do so once at startup. I would have
          separate issues for these.

          +1

          Show
          Mahadev konar added a comment - There are two pieces to this ticket anyway. 1. read config files from an arbitrary filesystem. 2. get daemons to re-read config more dynamically - most of them just do so once at startup. I would have separate issues for these. +1
          Hide
          dhruba borthakur added a comment -

          > The idea was to expose ZK as a Hadoop FileSystem

          Like I mentioned earlier in this JIRA, I really like this idea. I have an app that currently stores data on HDFS, but I would love to make it run on zookeeper without changing my application code.

          Show
          dhruba borthakur added a comment - > The idea was to expose ZK as a Hadoop FileSystem Like I mentioned earlier in this JIRA, I really like this idea. I have an app that currently stores data on HDFS, but I would love to make it run on zookeeper without changing my application code.
          Hide
          steve_l added a comment -
          1. Apache Directory implements an NFS filesystem front end to LDAP data, so you could today use it to provide all of the configuration data in a single point of failure/HA directory service
          2. I do subclass Configuration for my work, with a ManagedConfiguration being read from SmartFrog, rather than the filesystem. I have to pass an instance of this down when any service gets created.
          3. These Configurations get serialised and sent around with Job submissions, so they soon become static snapshots of configuration when jobs were queued, not when they were executed. You can't be dynamic without changing this behaviour.
          4. There are 350+ places in the Hadoop codebase where something calls the {{new Configuration()} operation, to create a config purely from the registered files. This is bad; some kind of factory approach would be better, with the factory settable on a per-JVM or per-thread basis.
          5. HADOOP-3582 collects other requirements for configuration.

          I am willing to get involved in this, as config is part of management, and I probably have the most dynamic configuration setup to date. But I'd rather wait until after the service lifecycle stuff is in, so change can be handled more gradually. If any configuration changes were to go into 0.21, I'd opt for a factory mechanism for creating new Configuration instances, so that people exploring configuration options can control things better.

          Show
          steve_l added a comment - Apache Directory implements an NFS filesystem front end to LDAP data, so you could today use it to provide all of the configuration data in a single point of failure/HA directory service I do subclass Configuration for my work, with a ManagedConfiguration being read from SmartFrog, rather than the filesystem. I have to pass an instance of this down when any service gets created. These Configurations get serialised and sent around with Job submissions, so they soon become static snapshots of configuration when jobs were queued, not when they were executed. You can't be dynamic without changing this behaviour. There are 350+ places in the Hadoop codebase where something calls the {{new Configuration()} operation, to create a config purely from the registered files. This is bad; some kind of factory approach would be better, with the factory settable on a per-JVM or per-thread basis. HADOOP-3582 collects other requirements for configuration. I am willing to get involved in this, as config is part of management, and I probably have the most dynamic configuration setup to date. But I'd rather wait until after the service lifecycle stuff is in, so change can be handled more gradually. If any configuration changes were to go into 0.21, I'd opt for a factory mechanism for creating new Configuration instances, so that people exploring configuration options can control things better.
          Hide
          Andrey Kuzmin added a comment -

          > until after the service lifecycle stuff is in
          Could you please post a link to discussion/ticket, I'd like to take a look.

          Thanks,
          Andrey

          Show
          Andrey Kuzmin added a comment - > until after the service lifecycle stuff is in Could you please post a link to discussion/ticket, I'd like to take a look. Thanks, Andrey
          Hide
          steve_l added a comment -

          Looking at my subclassed configuration code http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/hadoop/src/org/smartfrog/services/hadoop/conf/ManagedConfiguration.java?view=markup I am reminded of some things

          1. It subclasses JobConf, as some things expect to be handed a JobConf
          2. In the past, I did have it live, bound to the SmartFrog configuration graph, so (until serialized) every change in the SF configuration was reflected in the result of the get() method.
          3. That was pretty complex, I had to subclass a lot of methods that assumed that all state was stored statically in two properties files.
          4. It proved easier to comment the overrides out and move to grabbing all the values once, and no longer being live.

          Some people have mentioned that they'd like daemons to re-read their config more dynamically. That can get very confusing, as you may have some config values (ports, network cards, temp directories) whose changes only get picked up by a full restart. There are two tactics we've used in the past for this

          1. Have your service support a "reload" operation which loads in all values and resets thing
          2. Just restart the service
            Provided the cost of restarting is low, restarting services is way, way easier.
          Show
          steve_l added a comment - Looking at my subclassed configuration code http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/hadoop/src/org/smartfrog/services/hadoop/conf/ManagedConfiguration.java?view=markup I am reminded of some things It subclasses JobConf , as some things expect to be handed a JobConf In the past, I did have it live , bound to the SmartFrog configuration graph, so (until serialized) every change in the SF configuration was reflected in the result of the get() method. That was pretty complex, I had to subclass a lot of methods that assumed that all state was stored statically in two properties files. It proved easier to comment the overrides out and move to grabbing all the values once, and no longer being live. Some people have mentioned that they'd like daemons to re-read their config more dynamically. That can get very confusing, as you may have some config values (ports, network cards, temp directories) whose changes only get picked up by a full restart. There are two tactics we've used in the past for this Have your service support a "reload" operation which loads in all values and resets thing Just restart the service Provided the cost of restarting is low, restarting services is way, way easier.
          Hide
          steve_l added a comment -

          @Andrey, I was referring to HADOOP-3628. the changes are orthogonal, its just that I want to get the service stuff done before I spend any of my time worrying about what we can do to make configuration easier. With the subclassing working, I am happy short term, though I think it could be done more cleanly.

          Show
          steve_l added a comment - @Andrey, I was referring to HADOOP-3628 . the changes are orthogonal, its just that I want to get the service stuff done before I spend any of my time worrying about what we can do to make configuration easier. With the subclassing working, I am happy short term, though I think it could be done more cleanly.
          Hide
          Arun C Murthy added a comment - - edited

          One important and related feature are the necessary access-control mechanisms for reloading configurations, should we track it separately? Maybe HADOOP-5772 ?

          Show
          Arun C Murthy added a comment - - edited One important and related feature are the necessary access-control mechanisms for reloading configurations, should we track it separately? Maybe HADOOP-5772 ?
          Hide
          Edward Capriolo added a comment -

          I like the way glusterfs handles this.

          http://www.gluster.com/community/documentation/index.php/Client_Installation_and_Configuration#Manually_Mounting_a_Volume

          On the glusterfs server it holds all the client configurations in a configuration file. The cluster client can either be started up with a local configuration file or be given an IP/PORT and it will download its configuration from the server.

          Datanode should probably pull it its configuration from the namenode.
          Tasktracker should pull its configuration from jobtracker

          On the practicality side, large numbers of server will likely share a configuration. We should have a concept of a default configuration, and then per hosts overrides, and a way of saying groups of servers all share the same configuration file.

          I think with reloads a deamon like a DataNode could theoretically check the configuration and then make its own decision on if could apply the change, or if it should restart because of the change.

          Show
          Edward Capriolo added a comment - I like the way glusterfs handles this. http://www.gluster.com/community/documentation/index.php/Client_Installation_and_Configuration#Manually_Mounting_a_Volume On the glusterfs server it holds all the client configurations in a configuration file. The cluster client can either be started up with a local configuration file or be given an IP/PORT and it will download its configuration from the server. Datanode should probably pull it its configuration from the namenode. Tasktracker should pull its configuration from jobtracker On the practicality side, large numbers of server will likely share a configuration. We should have a concept of a default configuration, and then per hosts overrides, and a way of saying groups of servers all share the same configuration file. I think with reloads a deamon like a DataNode could theoretically check the configuration and then make its own decision on if could apply the change, or if it should restart because of the change.
          Hide
          Allen Wittenauer added a comment -

          I wish I had pictures of the whiteboard talk Owen and I had around this feature.

          One the big things I want people to keep in mind is that a client of the grid needs a lot of this information too. [e.g., something like default block size]

          There is also the issue of multiple JobTrackers per file system.

          You can do lovely things with a hierarchy of configuration information.

          Show
          Allen Wittenauer added a comment - I wish I had pictures of the whiteboard talk Owen and I had around this feature. One the big things I want people to keep in mind is that a client of the grid needs a lot of this information too. [e.g., something like default block size] There is also the issue of multiple JobTrackers per file system. You can do lovely things with a hierarchy of configuration information.
          Hide
          Tomer Shiran added a comment -

          Today, many users put the configuration files on an NFS server (e.g., NetApp) and all daemons and clients read the configuration from there. I can see two goals for this JIRA:

          1. Remove the external dependency. That is, allow people to deploy Hadoop without an NFS server.
          2. Remove the single point of failure. The NFS server might not have HA, in which case there is a single point of failure.

          Are there other issues with the current NFS-based architecture? What are we trying to solve here?

          Show
          Tomer Shiran added a comment - Today, many users put the configuration files on an NFS server (e.g., NetApp) and all daemons and clients read the configuration from there. I can see two goals for this JIRA: Remove the external dependency. That is, allow people to deploy Hadoop without an NFS server. Remove the single point of failure. The NFS server might not have HA, in which case there is a single point of failure. Are there other issues with the current NFS-based architecture? What are we trying to solve here?
          Hide
          Allen Wittenauer added a comment -

          Small (I'd say tiny) grids that can get away with using NFS is not the target audience here.

          Besides being a SPOF, for large installations having your entire grid and your clients mounting the same fs is likely impossible due to firewall restrictions. Using a distributed system such as LDAP or ZK or whatever means you also have a higher chance of replicating configs across firewall boundaries in a secure fashion.

          Also, if Hadoop is deployed to optimize certain features, chances are very very high that a site is going to have multiple configs for the same grid. For example, removing certain non-client related options out of mapred-site.xml or specifically tuning the HDFS audit log.

          Show
          Allen Wittenauer added a comment - Small (I'd say tiny) grids that can get away with using NFS is not the target audience here. Besides being a SPOF, for large installations having your entire grid and your clients mounting the same fs is likely impossible due to firewall restrictions. Using a distributed system such as LDAP or ZK or whatever means you also have a higher chance of replicating configs across firewall boundaries in a secure fashion. Also, if Hadoop is deployed to optimize certain features, chances are very very high that a site is going to have multiple configs for the same grid. For example, removing certain non-client related options out of mapred-site.xml or specifically tuning the HDFS audit log.
          Hide
          Edward Capriolo added a comment -

          I know this is not relevant to the ticket to dynamically/remotely read configuration files but I wanted some people interested in hadoop configuration to take a look at:

          http://www.jointhegrid.com/acod/

          The concept is to give users a IOS like interface to configure hadoop. The configurations are generated locally and then pushed out over SSH/SCP.

          The concept I am using: In most cases large portions of the configuration are shared. Their are very few exceptions to this. (IE one node has 9 disks not 8). These exceptions are done with an override list per host.

          node6.hdfs-site.xml = shared.hdfs-site.xml + override.get(node6).hdfs-site.xml
          

          Getting the effective configuration for a host is as simple as taking the shared list, and replacing the overridden variables for that host. I am using a few simple abstractions, and Java XML beans serialization for the configuration. Thus, the entire configuration is easily dumped to a single XML file, so one File object represents the state of all of the clusters configurations.

          Show
          Edward Capriolo added a comment - I know this is not relevant to the ticket to dynamically/remotely read configuration files but I wanted some people interested in hadoop configuration to take a look at: http://www.jointhegrid.com/acod/ The concept is to give users a IOS like interface to configure hadoop. The configurations are generated locally and then pushed out over SSH/SCP. The concept I am using: In most cases large portions of the configuration are shared. Their are very few exceptions to this. (IE one node has 9 disks not 8). These exceptions are done with an override list per host. node6.hdfs-site.xml = shared.hdfs-site.xml + override.get(node6).hdfs-site.xml Getting the effective configuration for a host is as simple as taking the shared list, and replacing the overridden variables for that host. I am using a few simple abstractions, and Java XML beans serialization for the configuration. Thus, the entire configuration is easily dumped to a single XML file, so one File object represents the state of all of the clusters configurations.
          Hide
          Alex Newman added a comment -

          Howdy,

          This is really exciting as I feel currently that configs are a hard problem. Currently it can be difficult to get transparency and flexibility depending on the configs scope. Specifically, where do I audit if a user sets a particular job paramater, or an admin sets a paramater for a particular system. It seems that this jira is about setting up scope on a cluster wide basis. LDAP and Zookeeper could allow configs across clusters.

          This is something that gasp vms logicals got right IMO in that it is one interface to solve all of these problems. I think one giant key value store is a good first step, but realistically you need different permissions based on if you are making a change intra cluster, inter cluster, for one box, for one user or for one job. Being able to go back and see what the hierarchy of those values, and having them being centrally managed would be really neat. Just my two cents, I am just spoiled from my slow big machines.

          Show
          Alex Newman added a comment - Howdy, This is really exciting as I feel currently that configs are a hard problem. Currently it can be difficult to get transparency and flexibility depending on the configs scope. Specifically, where do I audit if a user sets a particular job paramater, or an admin sets a paramater for a particular system. It seems that this jira is about setting up scope on a cluster wide basis. LDAP and Zookeeper could allow configs across clusters. This is something that gasp vms logicals got right IMO in that it is one interface to solve all of these problems. I think one giant key value store is a good first step, but realistically you need different permissions based on if you are making a change intra cluster, inter cluster, for one box, for one user or for one job. Being able to go back and see what the hierarchy of those values, and having them being centrally managed would be really neat. Just my two cents, I am just spoiled from my slow big machines.
          Hide
          Patrick Hunt added a comment -

          Here's a really interesting paper from Akamai related to Quorum use for configuration:
          "ACMS: The Akamai Configuration Management System"
          http://www.usenix.org/event/nsdi05/tech/full_papers/sherman/sherman_html/index.html

          ZooKeeper has a REST based interface that might work well if client's (hadoop) primary interest is to poll for configuration
          during bootstrap. Not that one has to use REST, if the ZK client interface were used directly then one could register
          for dynamic changes via ZK watches. Additionally the hadoop nodes could take advantage of other ZK features to implement
          group membership,(nodes register themselves, and their state, with ZK, allowing simple monitoring
          to be implemented), Leader election, etc...

          Show
          Patrick Hunt added a comment - Here's a really interesting paper from Akamai related to Quorum use for configuration: "ACMS: The Akamai Configuration Management System" http://www.usenix.org/event/nsdi05/tech/full_papers/sherman/sherman_html/index.html ZooKeeper has a REST based interface that might work well if client's (hadoop) primary interest is to poll for configuration during bootstrap. Not that one has to use REST, if the ZK client interface were used directly then one could register for dynamic changes via ZK watches. Additionally the hadoop nodes could take advantage of other ZK features to implement group membership,(nodes register themselves, and their state, with ZK, allowing simple monitoring to be implemented), Leader election, etc...
          Hide
          Andrey Kuzmin added a comment -

          > nodes register themselves, and their state, with ZK, allowing simple monitoring to be implemented
          I would add (dynamic) membership-aware scheduling. It's typically far from straightforward for a scheduler to account for nodes (re)joining group unless there is an explicit group registration mechanism, with join/leave events dispatched to scheduler and other interested parties.

          Show
          Andrey Kuzmin added a comment - > nodes register themselves, and their state, with ZK, allowing simple monitoring to be implemented I would add (dynamic) membership-aware scheduling. It's typically far from straightforward for a scheduler to account for nodes (re)joining group unless there is an explicit group registration mechanism, with join/leave events dispatched to scheduler and other interested parties.
          Hide
          Allen Wittenauer added a comment -

          Considering that the task tracker talks to the job tracker (where the scheduler runs), the dynamic add/join is already there.

          I want to re-iterate something here: Think about the client, not just the server! Chances are good the clients are NOT owned by the same entity that runs the Hadoop grid. Building something that is strictly usable by the server is not going to be that much in terms of bang-for-the-buck, considering any reasonable configuration management tool could do that.

          ... and given the various private convos I've had about this, I'm still not convinced ZK is a reasonable answer. Sure you can probably do more with it, but that comes at a cost, especially on the operations side where there is (highly likely) already a DS in place. It just isn't practical to require ZK for this.

          Show
          Allen Wittenauer added a comment - Considering that the task tracker talks to the job tracker (where the scheduler runs), the dynamic add/join is already there. I want to re-iterate something here: Think about the client, not just the server! Chances are good the clients are NOT owned by the same entity that runs the Hadoop grid. Building something that is strictly usable by the server is not going to be that much in terms of bang-for-the-buck, considering any reasonable configuration management tool could do that. ... and given the various private convos I've had about this, I'm still not convinced ZK is a reasonable answer. Sure you can probably do more with it, but that comes at a cost, especially on the operations side where there is (highly likely) already a DS in place. It just isn't practical to require ZK for this.
          Hide
          Patrick Hunt added a comment -

          Allen and I had a conversation about this a while ago and he made some very good points for storing certain data
          in a DS vs ZK. In particular user data makes alot of sense imo to be stored in DS. Data. Keep in mind that
          ZK is all about coordination, not "data storage". We don't support search for example, which is a significant
          feature in most DSs. Also integration with legacy systems (your existing user database) is also a feature
          of most DSs that ZK does not have. While ZK could do these things, a typical
          DS will do them for you out of the box, and make your admin's lives easier in the sense that they already
          have experience with this.

          At the same time things like coordination are best served by ZK. Keeping track of which nodes are allocated
          to which functions, the status of processes and coordinating operations between them, the load and activity
          of processes (nodes), Leader election within a highly reliable/available service, distributed locks and work
          queues, etc...

          Take a look at LinkedIn's Norbert for an example of one instantiation of something like this: http://bit.ly/6OQhwe

          Show
          Patrick Hunt added a comment - Allen and I had a conversation about this a while ago and he made some very good points for storing certain data in a DS vs ZK. In particular user data makes alot of sense imo to be stored in DS. Data. Keep in mind that ZK is all about coordination, not "data storage". We don't support search for example, which is a significant feature in most DSs. Also integration with legacy systems (your existing user database) is also a feature of most DSs that ZK does not have. While ZK could do these things, a typical DS will do them for you out of the box, and make your admin's lives easier in the sense that they already have experience with this. At the same time things like coordination are best served by ZK. Keeping track of which nodes are allocated to which functions, the status of processes and coordinating operations between them, the load and activity of processes (nodes), Leader election within a highly reliable/available service, distributed locks and work queues, etc... Take a look at LinkedIn's Norbert for an example of one instantiation of something like this: http://bit.ly/6OQhwe
          Hide
          Alex Newman added a comment -

          So I think their exists a couple of different conversations going on here. I suppose if all we wish is to replace basically the -D options passed into hadoop, then I am very openminded to a ldap based scheme as a plugin. I would be very happy if it included some scoping so that configuration could be provided on a intra cluster, inter cluster, host ,user or job level. Guidelines should be provided as how not to disrupt existing ldap users.

          However their exists numerous other things which fall into the cluster configuration in which ldap may not be the best choice.

          In regards to operations, I think deployment complexity is an argument against ldap. Even if an organization were willing to store such configuration in their existing ldap infrastructure, which I am skeptical of in many cases, it would require organizations not using ldap to run additional infrastructure to solve a problem which could be solved by something already provided by the hadoop infrastructure. I would feel as though if we saw many users deploying a isolated ldap installation just for hadoop that we would have made people's life harder not easier.

          Allen, sorry to be a pain , but could you be a bit more pedantic about what you mean by configurations ? Do you think a hybrid system may make sense here?

          Show
          Alex Newman added a comment - So I think their exists a couple of different conversations going on here. I suppose if all we wish is to replace basically the -D options passed into hadoop, then I am very openminded to a ldap based scheme as a plugin. I would be very happy if it included some scoping so that configuration could be provided on a intra cluster, inter cluster, host ,user or job level. Guidelines should be provided as how not to disrupt existing ldap users. However their exists numerous other things which fall into the cluster configuration in which ldap may not be the best choice. In regards to operations, I think deployment complexity is an argument against ldap. Even if an organization were willing to store such configuration in their existing ldap infrastructure, which I am skeptical of in many cases, it would require organizations not using ldap to run additional infrastructure to solve a problem which could be solved by something already provided by the hadoop infrastructure. I would feel as though if we saw many users deploying a isolated ldap installation just for hadoop that we would have made people's life harder not easier. Allen, sorry to be a pain , but could you be a bit more pedantic about what you mean by configurations ? Do you think a hybrid system may make sense here?
          Hide
          Allen Wittenauer added a comment -

          it would require organizations not using ldap to run additional infrastructure to solve a problem which could be solved by something already provided by the hadoop infrastructure. I would feel as though if we saw many users deploying a isolated ldap installation just for hadoop that we would have made people's life harder not easier.

          If you s,LDAP,ZK,g above, you'll find it is a better fit. The cold, hard reality is that LDAP is everywhere, ZK is not. There are some key features in ZK that are/were missing in order for it to fit here (with the good news being that those gaps are slowly closing). But the truth of the matter is that LDAP is a well understood technology by most IT departments and ZK is not. (Sidenote: it would be interesting to know how well used the security components of ZK are...)

          Also, I don't think you should think of ZK as part of the Hadoop infrastructure. It is a sub project (and therefore part of the ecosystem), but you can run Hadoop without using ZK and many many many people do, including Yahoo! for years.

          Allen, sorry to be a pain , but could you be a bit more pedantic about what you mean by configurations ? Do you think a hybrid system may make sense here?

          I'll try and write/diagram something up with a concrete proposal as to how I think this should be done, based on conversations I've had with Owen and others over the years. You'll find I'm thinking way beyond just a simple 10 node grid.

          Show
          Allen Wittenauer added a comment - it would require organizations not using ldap to run additional infrastructure to solve a problem which could be solved by something already provided by the hadoop infrastructure. I would feel as though if we saw many users deploying a isolated ldap installation just for hadoop that we would have made people's life harder not easier. If you s,LDAP,ZK,g above, you'll find it is a better fit. The cold, hard reality is that LDAP is everywhere, ZK is not. There are some key features in ZK that are/were missing in order for it to fit here (with the good news being that those gaps are slowly closing). But the truth of the matter is that LDAP is a well understood technology by most IT departments and ZK is not. (Sidenote: it would be interesting to know how well used the security components of ZK are...) Also, I don't think you should think of ZK as part of the Hadoop infrastructure. It is a sub project (and therefore part of the ecosystem), but you can run Hadoop without using ZK and many many many people do, including Yahoo! for years. Allen, sorry to be a pain , but could you be a bit more pedantic about what you mean by configurations ? Do you think a hybrid system may make sense here? I'll try and write/diagram something up with a concrete proposal as to how I think this should be done, based on conversations I've had with Owen and others over the years. You'll find I'm thinking way beyond just a simple 10 node grid.
          Hide
          Alex Newman added a comment -

          Allen, A more concrete proposal would be great, I am just trying to understand which part of the problem we are trying to solve. It's good to see people putting some real elbow great in making hadoop more palatable to ops/system as they are a very important stakeholder. I do think that requiring sysadmins to understand the details of zk, just to do configuration stuff would be kindof a show stopper for acceptance. Also having cluster and host wide information in ldap definitely seems like a win in the portability department. The only real worry is that we end up increasing the deployment complexity or develope things in such a way that people have to deploy an array of ldap servers within each cluster. Considering I have never worked at any large enterprise which uses directory services besides notes or active directory, I think that this could seriously be a burden if we do it wrong.

          Show
          Alex Newman added a comment - Allen, A more concrete proposal would be great, I am just trying to understand which part of the problem we are trying to solve. It's good to see people putting some real elbow great in making hadoop more palatable to ops/system as they are a very important stakeholder. I do think that requiring sysadmins to understand the details of zk, just to do configuration stuff would be kindof a show stopper for acceptance. Also having cluster and host wide information in ldap definitely seems like a win in the portability department. The only real worry is that we end up increasing the deployment complexity or develope things in such a way that people have to deploy an array of ldap servers within each cluster. Considering I have never worked at any large enterprise which uses directory services besides notes or active directory, I think that this could seriously be a burden if we do it wrong.
          Hide
          Patrick Hunt added a comment -

          would be interesting to know how well used the security components of ZK are

          ZK has plugable auth with acls on the znodes. Yahoo uses this internally for multi-tenant ZK clusters, the auth
          in this case is Yahoo specific implementation (uses certificates). We have met with the Yahoo Hadoop Security
          team, we don't have a kerberos plugin for ZK auth yet but we have discussed it.

          you can run Hadoop without using ZK

          that's true for MR/HDFS today, it may not be so in future.

          HBase requires ZK today, so if you're using HBase you're already using ZK.

          LDAP is a well understood technology by most IT departments and ZK is not

          I agree with this, in general most IT departments are going to be much more familiar with technologies such
          as LDAP, MySQL, Exchange, etc... that have been around for a while. This is a big plus. ZK is still very new
          relative to these mature technologies. (although the same could be said for Hadoop itself).

          Show
          Patrick Hunt added a comment - would be interesting to know how well used the security components of ZK are ZK has plugable auth with acls on the znodes. Yahoo uses this internally for multi-tenant ZK clusters, the auth in this case is Yahoo specific implementation (uses certificates). We have met with the Yahoo Hadoop Security team, we don't have a kerberos plugin for ZK auth yet but we have discussed it. you can run Hadoop without using ZK that's true for MR/HDFS today, it may not be so in future. HBase requires ZK today, so if you're using HBase you're already using ZK. LDAP is a well understood technology by most IT departments and ZK is not I agree with this, in general most IT departments are going to be much more familiar with technologies such as LDAP, MySQL, Exchange, etc... that have been around for a while. This is a big plus. ZK is still very new relative to these mature technologies. (although the same could be said for Hadoop itself).
          Hide
          Mahadev konar added a comment -

          Just my thought - I think ZooKeeper might become a part of Hadoop Deployment in somewhere near term (for High Availability) or some other part of Hadoop. With HBase already deploying ZooKeeper as part of there stack, it might need a little more thinking in just outrightly rejecting the idea of ZooKeeper because of unfamiliarity to ops.

          Show
          Mahadev konar added a comment - Just my thought - I think ZooKeeper might become a part of Hadoop Deployment in somewhere near term (for High Availability) or some other part of Hadoop. With HBase already deploying ZooKeeper as part of there stack, it might need a little more thinking in just outrightly rejecting the idea of ZooKeeper because of unfamiliarity to ops.
          Hide
          Alex Newman added a comment -

          I would say it this way:

          • What's the right fit
          • What role could ldap play
          • What are the specific goals we are trying to solve
          • How do we make ops/syseng guys comfortable with our decisions.
          • How do we reduce operational complexity
          • How do we reduce deployment complexity
          Show
          Alex Newman added a comment - I would say it this way: What's the right fit What role could ldap play What are the specific goals we are trying to solve How do we make ops/syseng guys comfortable with our decisions. How do we reduce operational complexity How do we reduce deployment complexity
          Hide
          Alex Newman added a comment -

          Allen, any updates here. I know y'all have release some of your stuff.

          Show
          Alex Newman added a comment - Allen, any updates here. I know y'all have release some of your stuff.
          Hide
          Edward Capriolo added a comment -

          I have been really intested in this ticket for a while. I think we can go in circles for a long time. No solution will fit everyonones needs.

          The best general solution in my optionion is LDAP. why?

          • if you have an argument against ldap but are making a case for zookeeper every argument you make against ldap could be made against zookeeper
          • LDAP is easy to scale single master+thousands of slaves or multi master configurations
          • LDAP is the defacto standard for the configuration of network servers
          • JAVA provides built in directory clients
          • Current hadoop configuration with overrides per host can be easily accomplished with LDAP
          • yes nodes will have to bootstrap with a hostname or lookup a default host
          • people that dont want to run LDAP for hadoop probably will not want to run alternative XYZ either. For those people their are configuration files.

          If someone seconds the 'I like LDAP idea' I will start working on it and submit a patch for review.

          Show
          Edward Capriolo added a comment - I have been really intested in this ticket for a while. I think we can go in circles for a long time. No solution will fit everyonones needs. The best general solution in my optionion is LDAP. why? if you have an argument against ldap but are making a case for zookeeper every argument you make against ldap could be made against zookeeper LDAP is easy to scale single master+thousands of slaves or multi master configurations LDAP is the defacto standard for the configuration of network servers JAVA provides built in directory clients Current hadoop configuration with overrides per host can be easily accomplished with LDAP yes nodes will have to bootstrap with a hostname or lookup a default host people that dont want to run LDAP for hadoop probably will not want to run alternative XYZ either. For those people their are configuration files. If someone seconds the 'I like LDAP idea' I will start working on it and submit a patch for review.
          Hide
          Alex Newman added a comment -

          I think getting a patch and ldap support is a great idea, although i disagree with the sentiment "if you have an argument against ldap but are making a case for zookeeper every argument you make against ldap could be made against zookeeper"

          Show
          Alex Newman added a comment - I think getting a patch and ldap support is a great idea, although i disagree with the sentiment "if you have an argument against ldap but are making a case for zookeeper every argument you make against ldap could be made against zookeeper"
          Hide
          Jeff Hammerbacher added a comment -

          LDAP is a protocol, not a storage system. Which implementation did you have in mind?

          Show
          Jeff Hammerbacher added a comment - LDAP is a protocol, not a storage system. Which implementation did you have in mind?
          Hide
          Sharad Agarwal added a comment -

          I think we don't need to debate hard on ldap versus zookeeper versus etc on this specific issue. I assume that we would have the abstraction here so that folks can try out different systems - ldap, zookeeper etc. and can configure one as per their environment needs.

          Show
          Sharad Agarwal added a comment - I think we don't need to debate hard on ldap versus zookeeper versus etc on this specific issue. I assume that we would have the abstraction here so that folks can try out different systems - ldap, zookeeper etc. and can configure one as per their environment needs.
          Hide
          Edward Capriolo added a comment -

          Jeff the implementation I want to go for was described above:

          I suppose if all we wish is to replace basically the -D options passed into hadoop, then I am very openminded to a ldap based scheme as a plugin. I would be very happy if it included some scoping so that configuration could be provided on a intra cluster, inter cluster, host ,user or job level. Guidelines should be provided as how not to disrupt existing ldap users.

          And meet this requirement:

          I think we don't need to debate hard on ldap versus zookeeper versus etc on this specific issue. I assume that we would have the abstraction here so that folks can try out different systems - ldap, zookeeper etc. and can configure one as per their environment needs.

          Also I wanted to add that since the new hadoop security is kerberos. Kerberos is normally backended by LDAP, so a good point of users will have an ldap server at their disposal.

          Show
          Edward Capriolo added a comment - Jeff the implementation I want to go for was described above: I suppose if all we wish is to replace basically the -D options passed into hadoop, then I am very openminded to a ldap based scheme as a plugin. I would be very happy if it included some scoping so that configuration could be provided on a intra cluster, inter cluster, host ,user or job level. Guidelines should be provided as how not to disrupt existing ldap users. And meet this requirement: I think we don't need to debate hard on ldap versus zookeeper versus etc on this specific issue. I assume that we would have the abstraction here so that folks can try out different systems - ldap, zookeeper etc. and can configure one as per their environment needs. Also I wanted to add that since the new hadoop security is kerberos. Kerberos is normally backended by LDAP, so a good point of users will have an ldap server at their disposal.
          Hide
          Allen Wittenauer added a comment -

          +1 on abstraction

          > LDAP is a protocol, not a storage system. Which implementation did you have in mind?

          LDAP is a well understood, well documented, and RFC standardized protocol. The implementation on the back-end is almost completely irrelevant as long as we follow the RFCs (and few offer extensions anyway). The only place where the back-end matters is in the raw schema definition file, as some implementations expect slightly different text formatting. Converting from, say, OpenLDAP's format to Sun/Fedora ... is trivial and all LDAP admins are used to it.

          > Kerberos is normally backended by LDAP, so a good point of users will have an ldap server at their disposal.

          This is a claim I'm not sure I'd make. By far, the biggest LDAP installations also happen to be Kerberos installations (thanks Active Directory!), but I suspect many places treat these as separate systems (MIT KDCs just got LDAP backend support in the past few years).

          In other news, I wrote up some random notes (implementation ideas? requirements? lots of stuff buried in there) at:

          http://wiki.apache.org/hadoop/IdeasOnLdapConfiguration

          It also includes a very quick 'n dirty LDAP tutorial to help those unfamiliar with how useful the object search aspect is to making the configuration stuff full featured and more useful than just storing raw XML (which is what I suspect the ZK crowd is thinking).

          Show
          Allen Wittenauer added a comment - +1 on abstraction > LDAP is a protocol, not a storage system. Which implementation did you have in mind? LDAP is a well understood, well documented, and RFC standardized protocol. The implementation on the back-end is almost completely irrelevant as long as we follow the RFCs (and few offer extensions anyway). The only place where the back-end matters is in the raw schema definition file, as some implementations expect slightly different text formatting. Converting from, say, OpenLDAP's format to Sun/Fedora ... is trivial and all LDAP admins are used to it. > Kerberos is normally backended by LDAP, so a good point of users will have an ldap server at their disposal. This is a claim I'm not sure I'd make. By far, the biggest LDAP installations also happen to be Kerberos installations (thanks Active Directory!), but I suspect many places treat these as separate systems (MIT KDCs just got LDAP backend support in the past few years). In other news, I wrote up some random notes (implementation ideas? requirements? lots of stuff buried in there) at: http://wiki.apache.org/hadoop/IdeasOnLdapConfiguration It also includes a very quick 'n dirty LDAP tutorial to help those unfamiliar with how useful the object search aspect is to making the configuration stuff full featured and more useful than just storing raw XML (which is what I suspect the ZK crowd is thinking).
          Hide
          Edward Capriolo added a comment -

          I think we are thinking along the same lines. I and I understand why you did and it that way but I wanted to chat about it.

          hadoopTaskTracker
          commonname: string
          hostname: multi-string
          mapred.job.tracker: string
          mapred.local.dir: multi-string
          mapred.tasktracker.map.tasks.maximum: integer
          mapred.tasktracker.reduce.tasks.maximum: integer
          

          Doing this will cause you to have your LDAP schema generated from your hadoop configuration. This is great because we can ensure type checking of the parameters but bad in a couple ways.

          • we can not deal with non hadoop parameters (which should not be in the configuration but could be)
          • it will be a very intricate and large and we have to track new changes

          Have your considered an alternate like this?

          commonname: string
          hostname: multi-string
          HadoopPropertyName: string
          hadoopPropertyValue: multi-string
          

          Yes, this is overly generic way is bad for several reasons, but I live it. Mostly because right now the current configuration is an XML file. It has no XSD schema, hadoop does the validity checking. What do you think?

          Show
          Edward Capriolo added a comment - I think we are thinking along the same lines. I and I understand why you did and it that way but I wanted to chat about it. hadoopTaskTracker commonname: string hostname: multi-string mapred.job.tracker: string mapred.local.dir: multi-string mapred.tasktracker.map.tasks.maximum: integer mapred.tasktracker.reduce.tasks.maximum: integer Doing this will cause you to have your LDAP schema generated from your hadoop configuration. This is great because we can ensure type checking of the parameters but bad in a couple ways. we can not deal with non hadoop parameters (which should not be in the configuration but could be) it will be a very intricate and large and we have to track new changes Have your considered an alternate like this? commonname: string hostname: multi-string HadoopPropertyName: string hadoopPropertyValue: multi-string Yes, this is overly generic way is bad for several reasons, but I live it. Mostly because right now the current configuration is an XML file. It has no XSD schema, hadoop does the validity checking. What do you think?
          Hide
          Allen Wittenauer added a comment -

          [FWIW, I'm not tied to that schema layout--it really was just a quick and dirty example.]

          One of the things Owen and I talked about last year was that you should be able to use files as a local override. I forgot to throw that in the mix here. heh. Anyway, non-hadoop configs that there is no schema definition for could be done that way. It also solves the one-off problems. [Custom -D overrides for cron'd jobs. Think block size or replication factor for example.]

          While it seems very intricate, I don't think it is any worse that what we have with XML flat files today. Simpler configs could have a very simple layout as well, potentially in one object if we lay out the schema correctly. Chances are good that more and more folks will have more complex config requirements and the LDAP tree layout is actually MUCH easier to deal with than XML. [In my case, I have 3 sets of configs, one for JT/NN, one for client, and one for compute nodes. That's ignoring the one-off configs for specialized processes... ]

          It is also worth pointing out that this is really intended for something beyond a simple 'play' grid. These types of folks already have a scale and use case for needing this level of support.

          As to using generic properties...

          I don't think it will work. Instead of 1 fetch per object class, you end up with 1 fetch per HadoopPropertyName. If you make HadoopPropertyName multi-valued so you can have more than one per object class, now we need to determine which value matches which name.

          Show
          Allen Wittenauer added a comment - [FWIW, I'm not tied to that schema layout--it really was just a quick and dirty example.] One of the things Owen and I talked about last year was that you should be able to use files as a local override. I forgot to throw that in the mix here. heh. Anyway, non-hadoop configs that there is no schema definition for could be done that way. It also solves the one-off problems. [Custom -D overrides for cron'd jobs. Think block size or replication factor for example.] While it seems very intricate, I don't think it is any worse that what we have with XML flat files today. Simpler configs could have a very simple layout as well, potentially in one object if we lay out the schema correctly. Chances are good that more and more folks will have more complex config requirements and the LDAP tree layout is actually MUCH easier to deal with than XML. [In my case, I have 3 sets of configs, one for JT/NN, one for client, and one for compute nodes. That's ignoring the one-off configs for specialized processes... ] It is also worth pointing out that this is really intended for something beyond a simple 'play' grid. These types of folks already have a scale and use case for needing this level of support. As to using generic properties... I don't think it will work. Instead of 1 fetch per object class, you end up with 1 fetch per HadoopPropertyName. If you make HadoopPropertyName multi-valued so you can have more than one per object class, now we need to determine which value matches which name.
          Hide
          Steve Loughran added a comment -

          I should give some live demo of what we're up to; the current slides are online

          1. ask the infrastructure for the VMs, get some whose name is unknown
          2. bring up the NN/JT with a datanode on the same master host, ensures the JT doesn't block waiting for the filesystem to have >1 DN and so be live.
          3. provided that master node is live, ask for more workers. If the master doesn't come up, release that machine instance and ask for a new one.
          4. I also serve up the JT's config by having the machine manager front end have a URL to the hadoop config, one that 302s over to the config of the JT. You get what it is really running, and as it has a static URL you can get it whenever the cluster is live (you get a 404 if there is no vm, connection-refused if the 302 fails). I can use this in a build with <get> and deploy client code against the cluster no need for fanciness.

          I did try with the nodes picking ports dynamically, there's no easy way of getting that info or the actual live hostnames back into the configurations. future work. We need the services to tell the base class what (host,port) they are using for each action, and to dynamically generate a config file from that.

          As an aside, I am not a fan of XSD, so don't miss its absence. XSD's type model is fundamentally different from that of programming languages, and is far to complex for people, be they authors of xml schema files or XSD-aware XML parsers, go look at xsd:any, whether the default namespace is in the ##other namespace and you too will conclude that it is wrong.

          Show
          Steve Loughran added a comment - I should give some live demo of what we're up to; the current slides are online ask the infrastructure for the VMs, get some whose name is unknown bring up the NN/JT with a datanode on the same master host, ensures the JT doesn't block waiting for the filesystem to have >1 DN and so be live. provided that master node is live, ask for more workers. If the master doesn't come up, release that machine instance and ask for a new one. I also serve up the JT's config by having the machine manager front end have a URL to the hadoop config, one that 302s over to the config of the JT. You get what it is really running, and as it has a static URL you can get it whenever the cluster is live (you get a 404 if there is no vm, connection-refused if the 302 fails). I can use this in a build with <get> and deploy client code against the cluster no need for fanciness. I did try with the nodes picking ports dynamically, there's no easy way of getting that info or the actual live hostnames back into the configurations. future work. We need the services to tell the base class what (host,port) they are using for each action, and to dynamically generate a config file from that. As an aside, I am not a fan of XSD, so don't miss its absence. XSD's type model is fundamentally different from that of programming languages, and is far to complex for people, be they authors of xml schema files or XSD-aware XML parsers, go look at xsd:any, whether the default namespace is in the ##other namespace and you too will conclude that it is wrong.

            People

            • Assignee:
              Unassigned
              Reporter:
              Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              43 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development