|
Norval Hope made changes - 12/Oct/06 02:37 PM
[
Permlink
| « Hide
]
Norval Hope added a comment - 12/Oct/06 02:42 PM
I should also mention from the testing I've done the difference in performance between loading the schemas dynamically at startup versus running the Maven generated classes is pretty minimal, I've loaded eight schemas where one is large (689 attribute types and 58 object classes) in less then a second. The overhead is really just the cost of parsing the .schema files as the generated code ultimately creates the same classes that my convertors do.
Thanks for the patch !
Having "dynamic" schema is something we want to include in the server, the latest discussion showed that the current solution via a maven plugin is only a temporary solution. I haven't read your code thouroughly, but I already have a few questions : 1) are you generating .class files from the schema (like the one we are generating with the maven plugin) 2) If so, which tool do you use to compile the files ? (assuming that we can't distribute tools.jar, this can be an issue) 3) what about dependencies? Are they deducted from the references to objects/attributes and from the file where they are declared ? I do think that the time needed to parse the schema is totally irrelevant, as we don't launch the server every 10 seconds, so even if parsing cost 1 minutes, this is always better than any other static solution. Last point, I think that generating classes is definitively not the best solution. Here, we want to have a totally dynamic system, and schema just rely on few static classes : matchingRules classes. It should be possible to replace hardwired Objects/Attributes with data structures associated with Abstract class implementing the needed features. We have to think about it, because this will help us to implement future extensions, like per partition schemas, or stuff like that. I think the subject is wide open, and we can discuss all the pros and cons in this JIRA. Thanks again, Norval/Norbet :) PS: your post wasn't forgotten, we just haven't had time/energy to address it lately, due to ApacheCon US. No problem, always trying to help.
Here are the answers to your questions: 1) no classes are generated, I just parse the .schema file and create the same objects that the Maven plugin-generated code would when run. 2) N/A 3) Dependencies are handled by simple ordering (ala Unix init tasks) rather then explicit statement of dependencies. Hence a core schema needs to appear before one that relies on it. Two methods are currently supported: 1) reading list of files explicitly named in server.xml or 2) read alpha-sorted list of files matching a regex in a named directory (in which case file names need to be chosen carefully so dependency ordering is respected). The per-partition schema concept is one of my target use-cases and I included some extra code which can contribute schema information for an external directory server via standard LDAP querying it. The problem I had when running it was that contributing content to the global registries after startup had no effect as the it seems the bootstrap registries were always used. In the past I had a hack which got around this problem but is stopped working three months or so ago and I haven't bothered revisting it. Two points remain that I'm not clear on: a) Whether my proposed patch needs further thought in the "embedded" ADS use-case b) Whether syntax defs and matching rule defs are as dynamic as attribute types and object classes. OpenLdapSchemaParser doesn't have any methods returning them as it does for attr types/object classes so I presume they are simply deduced via "create on reference" or something like that. My current code ignores them (assuming they have all been defined via the Maven generated code), but will be easily updated once I know what to do. Thanks for the PS too. I completely understand, but at the same time was expecting at least a glimmer of interest ;-) Regarding dependencies : there is no way to handle them simply but building a kind of association between Objects/attributes (O/A) and the file where we found them. At the end, either we have created a full tree, or there are lacking elements, and we must discard one or more file, generating an error log.
Regarding your patch : 1) We need to read it thoroughly, and this is my intention for the next couple of days. 2) It would be very cool if patches contains javadoc, comments and use the ADS coding convention (which, obviously, is pretty much an oral tradition than a written rule, so I guess nobody by current committers now about :) It helps a lot to understand the design and intention (I must admit that I don't really know this part of the server) 3) As this is really a very important part of the server, deep tests should be run, and I think it may deserve to be applied in a branch, until we pass successfully all the tests, including VSLDAP tests (the one about Open Group certification) The matching rules might be a little bit more static, because I don't know anybody who added one in any server. This is pretty much something that is hardwired in the server atm. But we can really include tham as plugins, via spring configuration (or osgi in the near future). Thought in progress... PS2 : yeah, I understand that you felt a little bit disapointed... However, you also have to understand that we are volunteer, and after having spent one week in Austin, working, discussing, (drinking), we were a little bit burnt out when coming back home. I personnally had to deal with a damn jet lag and more than 2000 mails, plus many importants other things to do (ya know, familly, friends, job etc). Nothing get lost when using JIRA, it's pretty much a question of time and energy to read them and respond to them :) Norval,
I have created a branche (apacheds-schema) to testyour patch, I have applied it - the pom.xml should also be atched if we want the project to compile under eclipse - but I need a sample server.xml to test it, or at least some more doco. Could we figure out an IM/skype session on the next few days in order to fix this point ? Thanks ! We have a problem : the initialization process is loading some schemas, based on some generated class (CoreSchema, CosineSchema, ...). If we decide not to load those classes, but to use parsed files, then those loaded classes should be removed. In fact, the initialization process must be modified to only be based on parsed files (I Don't think that using generated classes is really usefull, even when embedding the server. Or should we default to those classes, being delivered within the jars?)
server.xml fragment was included in initial comment for this issue.
I'm happy to adjust code to match ADS coding conventions, bit hard to see exactly what they are as your previous doco references an Eclipse .xml file without giving any code examples (I'm using IDEA). I can just scout other ADS code to get a feeling, are there some packages which are particularly compliant that I should use as my reference points? Javadoc is in pretty good shape already, but happy to improve it if there are any areas you feel need further attention. As for initialisation, I'll try doing a run with only .schema files and see if everything works. I wasn't expecting there'd be immediate interest in retiring the Maven plugin so wasn't thinking so big, but logically I can't see a problem if syntax and matching rules are being handled statically already... Will post my results shortly. Ok, I understand how the objects other then attribute type and object class are handled now. The Maven plug-in generates stub classes for 13 different aspects of a schema (see ProducerTypeEnum), but only fills in the contents for attribute types and object classes automatically.
Hence three of the existing schemas (system, nis, inetorgperson) need extra manually maintained java classes to be sucked in, in addition to their .schema files. Hence I need I'm thinking of removing the existing ???Schema.java classes (which implement BootstrapSchema) and replace them with an optional placemarker, which can be used to suck in manually maintained content if its is found on the classpath at the time its related ???.schema file is loaded. These new optional ???Schema classes will implement a method: BootstrapProducer getProducer(ProducerTypeEnum) and will return a non-null producer for any cases where manual additions are required. For schemas where no manual additions are required no ??Schema class will be required at all. Hi Norval,
I have added some doco in cwiki : http://cwiki.apache.org/confluence/display/DIRxSRVx11/Apache+DS+initialization It explains the way ADS is initialized, and I have started a page about Schema Loading, but is very empty atm :) I don't know if you have write access to the wiki, but if this is not the case (usually, it takes time to get full kerma when you have been elected as a committer ...), then I will be pleased to "inject" whatever text you want into it for you. Schema parser so far just handle ObjectClasses and Attributes, not matchingRules, matchingRuleUses, DITContentRuleDescription, DITStructureRuleDescription or anything described in RFC 4125. Further more, many of the base classes seems to be hand crafted, and loaded during the initialisation phase, but this is still something I have to study. This would be very good if we can write down some specs/doco about what should be done in order to determine how long it could take to implement dynamic schema managment, considering that we want to create a GUI to do that (implying that we will need a clear API to manipulate the schemas). Have got a solution working (nothing checked in yet) using the approach I outlined above, with an optional ???SchemaAdditions class which can contribute manual content, and have the server starting up happily.
Still need to consider whether things can be done more explicitly (ideally a separate Spring .xml matching .schema files requiring manual content to be registered) and what parts of the existing code can be retired/tidied up. Will post update shortly. I have updated the schema documentation in the new wiki :
http://cwiki.apache.org/confluence/display/DIRxSRVx11/Schema+loading Hope it helps ... Here is an initial patch of a working server (at least given the testing I've done) against https://svn.apache.org/repos/asf/directory/branches/apacheds-schema , where no output from the Maven schema plugin is used in the server.xml file (I ran ./apache/server_main/apacheds.bat).
This is obviously a work in progress to show the job can be done, and one possible implementational approach. There are a number of significant questions that still need to be addressed by some developers more knowledgeable about ApacheDS then myself: 1. Maven support: currently I ran by looking up the .schema files in ./apache/server_main/conf/*.schema (configured via ./apache/server_main/server.xml, which I also copied into the installers dir). I copied these files manually to test, but presume they should be copied from apache/core/src/main/schema to somewhere else for packaging in all supported release packages. Even if the standard .schema files are packaged in .jars, I think it is important that a dir outside of jars can be easily configured to accept extra .schema files not included in the standard build. a. Currently extra content (not attribute types or object classes) that is associated with a .schema file, but which must be manually maintained is found by looking for a matching ???SchemaAdditions class (eg SystemSchemaAdditions associated with "00_system.schema") using reflection. A more explicit approach would be better but simply using Spring is problematic as the major variable in the manual configurations is the code (normalizer/comparator etc) for each item, and having a separate class for each them is most probably not a viable approach. 2. Tidy up: I haven't removed any code made defunct by this patch, but there is some (eg BootstrapSchemaLoader). There was some ThreadLocal support in this class which I don't understand the motivation for, so I may have missed something. a. I don't understand the reasoning behind the BootstrapRegistries versus the GlobalRegistries, and it seems to me that ultimately a single Registries flavour should be supported, and it should support schema being contributed to the server dynamically after it has started up (for instance on a custom partition being dynamically activated). b. Some refactoring of project modules will most probably be required if my implementation approach is accepted. In particular the apache/core-plugin module should perhaps be deleted and the Antlr support in it moved to apache/core. 3. Once more knowledgeable peeps then I have weighed in so that the details are clearer, I'm happy to document the details.
Norval Hope made changes - 07/Nov/06 10:57 AM
To bring some thiughts to you questions :
1) I agree with you. Schema should be available somwhere alese than in jars. What we can also do is to preload them into a partition. I'm dreaming of a Ldap server which use itself to manage schema... Of course, you will still have schema files, but you will just parse them at start only if they have been 'touched'. Does it sounds crazy ? 1.a) I think that the spring approach seems better, from my point of view. This can be discussed. For normalizer and comparator, we should consider that we will have System Normalizer/Comparator, and User Defined N/C. Only those last classes should be put in Spring file, otherwise, we will end with a monster configuration file :) 2) Yes, a lot of dead code will be removed. We can do that in the branch. Man, I hope you will get karam quickly :) 3) We badly need some documentation :( I'm trying to improve it, but it takes a *lot* of time, as I don't have a deep knowledge of all the code, and to be sure that I don't write down stupid tgings. And because english is not my favorite language, too :) I will apply your patch in 2 hours. I didn't commit the patch, because the code differs a little bit too much from our coding standard (http://cwiki.apache.org/confluence/display/DIRxDEV/Coding+standards).
Is it possible for you to modify the code accordingly to what we suggest ? If you have any question regarding those standard, feel free to ask ! (btw, we will use the schema loader with Java 5, so use generics as much as you can, it's really good to have them !) This is the same as the last patch except for updates to coding style and some added use of JDK 1.5 features
Norval Hope made changes - 09/Nov/06 01:05 AM
After submitting a tidier patch, here are some answers/comment son Emmanuel's questions/notes:
> To bring some thiughts to you questions : > 1) I agree with you. Schema should be available somwhere alese than in jars. What we can also do is to preload them into a partition. I'm > dreaming of a Ldap server which use itself to manage schema... Of course, you will still have schema files, but you will just parse them at start > only if they have been 'touched'. Does it sounds crazy ? If by this you mean persisting whats read from the .schema files into LDAP representation in the JDBM backend, then I can see that's possible but don't really see the motivation. The schema import process at start-up adds very little overhead, so runtime impact is not a big motivator. My concern with persisting the schemas imported is simply that it makes them harder to manage, instead of commenting out a line in a server.xml to hide a schema I'd need to delete a bunch of LDAP objects. While there is a single global schema this is quite problematic as objects from all the .schemas are mixed together. My primary use-case if for schema information to be contributed dynamically after start-up, for instance just prior to activating a custom partition requiring additional schema support. > 1.a) I think that the spring approach seems better, from my point of view. This can be discussed. For normalizer and comparator, we should > consider that we will have System Normalizer/Comparator, and User Defined N/C. Only those last classes should be put in Spring file, otherwise, > we will end with a monster configuration file :) I prefer explicit solutions too but see problems with a Spring approach because the most important part of manually added schema items in the pluggable code that actually implements added behaviour (other then that there are mainly just basic attributes like names and OIDS). As far as I'm aware Spring can't help too much with the need for custom code except for allowing you to create named classes, and creating constructor chains of named classes. That's why I said in a previous comment that I thought an extra class for every manually added schema item wouldn't be the best path. > 2) Yes, a lot of dead code will be removed. We can do that in the branch. Man, I hope you will get karam quickly :) I'm afraid I don't know Maven very well so I may need to ask for help in the tidy-up, when we get to this phase. For instance with the conf/*.schema files I'm currently getting the uber-jar to read, I don't know how i'd get Maven to copy them around and/or where would be the best place to copy them to, so that installer/embedded/uberjar release flavours can all find them. So karma isn't necessarily a complete answer. > 3) We badly need some documentation :( I'm trying to improve it, but it takes a *lot* of time, as I don't have a deep knowledge of all the code, and to be sure that I don't write down stupid tgings. And because english is not my favorite language, too :) I'm happy to help as time permits, once the details have been reviewed so that there is a sufficient level of confidence to start documenting. As an aside, who is the guru of the existing schema code? Is it Alex? Is there any chance we can get this person to cast their eye quickly over my patch (once it's committed on the schema branch) and get any feedback on some of my questions in the comment trail for this issue? The idea beneath Ldap stored schema is to be able to change the schema on a live ADS, withour restarting the server. Sometime, you create an AttributeType, but you may want to modify an element of it (like the Matching Rule or whatever), without having the possibility to stop and start the server again - production servers, for instance -. (I mean, you don't *do* this kind of manipulations on production servers, but you may want to inject the tested and validated schema on a production server withour stopping it). This will allow cool administration operations.
Ok, this is not really urgent... Let's postpone this "idea" 1.a) Spring/maven There is not so much "behavior" we need to add to schema objects. Matching rules are pretty much stable, and I never heard about anyone who wanted to add a matching rule. But obviously, tthis is something we might want to do, as ADS coders. I f we have to deal with a mechanism which will load a jar, then we will have to create our own classloader for that, I think. Don't know at this point. We should discuss this point, maybe in another thread. ATM, isolating core behavior in the server in a sub-project could be an idea. 2) Not a pb. We will handle this smoothly. Note the patches have been applied to branch from the 1.1 trunks which Emmanuel created called https://svn.apache.org/repos/asf/directory/branches/apacheds-schema for review.
ok. I also applied it yesturday, but I think I did it with the previous one (not jdk 1.5). I work from home and office, and I think I committed the one from office; My bad. I hope that everything is fine on the branch :)
Just a little point : I may be picky abot code convention, but this is just to be sure we follow commons rules, so don't take it personnaly. I'm sure that after a few commits I will stop... For instance, you commited this code in apacheds/server-main/src/main/java/org/apache/directory/server/UberjarMain.java : + if (log4jProps == null) + log4jProps = "log4j.properties"; It would be good to add enclosing { and }. Something we may want tod discuss is the use of the 'final' keyword. I personnally don't use it a lot for many reasons, but essentially because I find its semantic not clear enough in Java. Let me explain : - final for class and method makes sense, and should be use from time to time. - final for fields are acceptable for constants, associated with static , like in public static final int ERROR_LEVEL = -1; Otherwise, I don't really think this is usefull (even for guarantee a unique instanciation in constructor), because the semantic of final is quite different in this case - final for inner variables does not make sense except if those variables are used in inner classes. If you have to declare a variable final, then one can ask the question : shouldn't it be a constant ? Again, it's more or less a question of semantic - final for parameters makes sense only if you want to insure invariance. That's a pity that java does not use the const keyword... This is sometime usefull especially for public API. This is IMHO, of course. I just find it a pity that this keyword has so many different semantics, and I also find it a little bit overuse and tools like PMD are trying to make you think that you should use it almost everywhere. It hurts my eyes to see final everywhere :) I don't want to start a religious war. May be we can discuss the use of final in the ML, because there are quite good arguments in favor of its use (we don't use it a lot inj ADS, and this may be bad). May be we should also try to use it more, in order to have a little bit more defensive code... wdyt ? Re having {}s around a single statement; no problem I just missed that line in my tidy-up (as the code existed before the knew exactly what the coding conventions were).
Re "final", again I don't have strong feeling about it either. I use "final" by default because the IDEA settings used in my day job enforce its use, with the only real benefit being just to be absolutely sure that the compiler (and readers of the code) know that the value isn't going to change. Before the final merge into trunks/ (assuming the schema reworking can be brought to an acceptable state) I'll de-finalise files as required. Don't spend time on de-finalizing too much :)
We have some problem with the way schema are dynamically loaded. It seems to work for a standard server, but integration tests are failing because they assume that schema have already been generated and try to load the classes. We have to fix that otherwise we wont be able to integrate the schema loading system. FYI, when we think that we have a correct version of the server, we check it by launching the following command : mvn -Dintegration test It launches a bunch of tests which are used to check that the embedable server still works. Here we have a massive failure. Not a big deal, though, but it need to be fix :) This is some of the information I knew I didn't know :-)
I suspect Maven skills will need to come into play here, as currently I have "svn cp"d the .schema files from apacheds/core/ to server-main/conf/ (with new names having appropriate numeric prefixes) simply so the UberJarMain case could be demonstrated and the general approach reviewed. As outlined previously in this (admittedly long) issue thread, I have questions to which I don't know the answers: 1. Where should the numerically prefixed .schema files be copied to by Maven so that they can be used in all release packages (uberjar/embedded/installers etc)? I know so little about Maven that I wouldn't know how to arrange the copying of the files even if this destination directory were decided. 2. Where should the numerically prefixed .schema files end up in the deployed images for each of the release packages (uberjar/embedded/installers etc)? I noticed the InstallationLayout.getConfigurationDirectory() method which looks promising; is this method used/usable for all release packages? Now that we have a 1.5.0 (almost) version in which the schema is dynamic, should we close this issue ?
My motivation for the original issue still exists even with the new dynamic schema system: registering schema dynamically with the server at runtime (hence some varieties of SchemaLoaders taking schema content from various sources).
However, I will need to upgrade my code to use the schema subsystem in the near term so the attached patches are redundant. Most probably makes most sense for me to post new patches to this issue when they are ready (or commit to my sandbox or something) assuming they are interest, rather then closing it. WDYT? ok, let's keep this issue open.
I think that with the new schema system, you will be able to register new schema on the fly, but this has to be tested (not 100% sure). The main problem is that removing an element from an existing schema may lead in considerable impact on the data : if you remove, say, and ObjectClass XXX, you will need to check *all* entries and remove all of them which contains this ObjectClass. Our intention was to use LdapStudio to offer such a tool, so that you can anticipate the impact of such modifications. However, if yoiu have millions of entries into the server, this kind of operation will be very costly. In your case, you have to know that now, the schema is just considered as standard Ldap entries, so you can edit it with any ldapBrowser, assuming you know its structure (not really well documented atm :) Sorry for this dumb question, but could someone explain how what's in this patch relates to
Sorry, David, I might be totally dumb, but I don't see any relation between
I don't really understand what this issue is about, but it looks to me as if both this one and 834 are about having the set of schemas you want present after the server starts, rather than the set of schemas present in the default server installation. I'm hoping someone can explain when you would use this system and when you'd use the 834 solution. I think that 834 asks you to preprocess your schemas into a apacheds friendly jar format which then gets installed as the schemas on the first startup, whereas this is more dynamic, but I'd like someone who understands this better to confirm or deny this.
Hmmm... I quickly parsed the two issues again, and it seems they are quite releated, or at least there is a strong link.
The thing is that the whole picture may have changed in 1.5.0, as the schema wont be anymore files, but data directly injected into ADS : you will be able to inject a new schema using a ldif file, or with LdapStudio. However, I agree with David, this is not really cristal clear. We need to double-check all that and add some doco. I'm not sure we can handle this for 2.0, but let's try anyway.
Emmanuel Lecharny made changes - 09/Jul/08 12:48 PM
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
New Feature
Open
Major
reading .schema files at server start-up
