|
[
Permlink
| « Hide
]
Doug Cutting added a comment - 17/Jul/07 07:30 PM
This can easily be implemented by packing the user jars together into a single jar, then submitting the job with that jar, right? I don't think the mapred kernel needs to deal with more than a single jar per job, but making the user API accept multiple jars is fine.
If the user's jar does not have the main function (when the user uses Aggregate, Streaming, DataJoin, etc.), his/her jar file(s) will not be picked up. Plus, in many cases, the user's job may also depend on some third party jars. I don't disagree with any of your statements in the previous message: currently we encourage the main to be in the top-level jar specified, which can be awkward; and, yes, it would be more convenient to let users list multiple jars when submitting jobs.
I'm suggesting that JobClient should jar things together. This would change the way that the job jar is determined, and thus the relationship between the main() and user jar files can be altered at the same time. Users should be able to submit jobs specifying a set of jars. That's the crux of this issue, and I agree we ought to support it. But I suggest that the way we ought to implement this is to change JobClient to pack together the user's jars into a single jar, and submit this. Few if any changes should be required to the JobTracker or TaskTracker. Does that make sense? Do you have an alternate implementation proposal? I think your proposal of letting JobClient to jar multiple jars into a single one is reasonable. > I'd also suggest to change bin/hadoop or the underliying java code to accept options like --additionalJars
Adding such generic options can easily be achieved with the proposed implementation in Adds the ability to load multiple jar files for a single job. All jar files are merged into a single master job jar file that is then submitted as the job.jar to hadoop.
The multipleJobJars.patch file is available for review. While this patch does add the ability to have multiple jars it does not change any script options.
One enhancement will be to unjar the jars in the reversed order of the jars in the option spec. For Nutch development at least (I don't know about others), it is more useful if classes in later jars overwrite classes in earlier jars. This will enable someone to do Nutch development, overriding or reworking core classes, without touching the main Nutch source code base. For many of the Nutch programs, a NutchJob is created that automatically sets the job jar file, which would now be the first jar file. We wanted to be able to override that when necessary.
This is looking good. A few comments:
1. Just because most of the public methods in JobConf don't have javadoc is no excuse not to add javadoc to new methods there. 2. The logic added to JobClient is big enough that it should go in a new private method. 3. The temporary directory should probably be created under hadoop.tmp.dir and should have a unique name. It should be removed either in a 'finally' clause or in a shutdown hook, preferably a 'finally' clause. Would it be helpful to make changes to JobClient to be able to support the command line options, such as -additionalJars, at the same time we make introduce the new functionality. Also what do we need to do to support these options within the command line scripts?
> it is more useful if classes in later jars overwrite classes in earlier jars
That's inconsistent with normal CLASSPATH behavior, no? And, in any case, shouldn't an application (like Nutch) be able to easily order jars according to whatever convention we implement? > command line options Yes, it would be gppd to add command-line options in this patch. No changes should be required to the scripts, but rather just to JobClient#run(String[]) and/or RunJar#main(String[]). I hadn't thought about it that way and yes it does go against java classloader semantics. I will make the changes to make it consistent with normal classpath behavior . I will also make the other changes mentioned above.
My intention here was really to allow custom development without changing standard jars, but yes that can be handled by semanics that we define. What would be a good way to do this, something like a addPriorityJar() method in JobConf? Currently setJar will take priority over addJar and addJars. My intuition is that addJar() should add things onto the front of the classpath, overriding what's there.
Just confirm this for me. Each application (i.e. Injector, Generator, etc.) would be responsible for dealing with the command line options in their own way, even though they extend ToolBase. If I modify JobClient#run(String[]) then that only sets additional jars for the JobClient. I could add the command line parsing to ToolBase#processGeneralOptions, and then it affects any child of ToolBase. Just wondering where to put the command line parsing and how it would affect applications like Injector.
+0, new Findbugs warnings
http://issues.apache.org/jira/secure/attachment/12362196/multipleJobJars.patch New Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/445/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html >Just wondering where to put the command line parsing and how it would affect applications like Injector.
At the risk of repeating my previous comment, i should say : Adding such generic options can easily be achieved with the proposed implementation in Dennis, for this issue, i think you could just implement command line parsing in ToolBase#processGeneralOptions, and ToolBase#buildGeneralOptions. You can add a method to get the additional jars. >then it affects any child of ToolBase For the patch, IMO having two functions JobConf#getJar() and JobConf#getJars() is a bit confusing. Could we just deprecate getJar(), and return an array of length one in getJars. And it would be good to make a new util class JarUtils, and refactor RunJar#jar(), RunJar#unjar and RunJar#getToBeJared().
> Could we just deprecate getJar(), and return an array of length one in getJars()?
+1 We should deprecate both setJar() and getJar(), replacing them with addJar() and getJars(). Remember that multiple jars is really just a front end merge before a single merged job jar file is submitted to the MR system. We still have to set a single job jar (currently through setJar) and the various job runners and task trackers need to get that single jar (currently through getJar). So if we are going to deprecate getJar and setJar, then we will need to change how we set and get the final job jar throughout the system.
The other changes and recommendations have been made and I am doing final testing. If we can get a consensus on the get and set jar methods I can have a new candidate patch uploaded today. > So if we are going to deprecate getJar and setJar, then we will need to change how we set and get the final job jar
Good point. Maybe we can make these (or equivalent new methods) package-private instead of removing them, only used by the core? I got to thinking, always a dangerous thing, and I thought if we are extending this for multiple jar file, why not other resources like jars on the classpath, jars that contain a given class, and directories. Let's say that we could specify one or more directories as a resource to be included in the job jar, then when we do the merge we would copy all resources from that directory into the job jar. This would allow us to do thing like deploy executables, resource files, or multiple jar files across the cluster to be used in the jobs. So say you have a custom executable you need to call in your MR job, you just drop it in a directory, include the directory as a job resource and that executable would get deployed out onto the cluster and would be available for that single job.
I went back and refactored the code to allow job resources as opposed to just jar files. A resource would be either an absolute path to a jar file, a jar file on the classpath, a directory, or the name of a class that is contained in a jar on the classpath. As an added bonus getJars and addJar now become getJobResources and addJobResource (we may need to come up with a different name as this might be too easily confused with default and final resouces in configuration), and we can keep getJar and setJar as they now apply only to the final job jar file. I am doing final testing of this code right now and will have a patch up in just a little while. The multipleJobResources.patch has all of the jar code refactored and tested. It as functionality to allow multiple type of job resources including jar files, classpath jars and classes, files, and most importantly directories. All resources are merged into a single job.jar before the job file is submitted. Options were added to ToolBase to support multiple job resources. Unit tests for jar utilities are included. Current patch passed all unit tests and worked successfully with a job run.
+1 for the patch. I have not tested it but the code seems good.
Is there anything else I need to do with this? If this patch works for the community I would love to get it included as it would help with some tutorials I am writing on doing custom nutch development (plugins, etc.). If I need to make any changes please suggest.
This patch includes the multiple job resources patch and fixes some issues we were seeing in a production environment with JarUtils NOT correctly copying some jar resources.
Which Hadoop release is this meant for? I would like to use to create an archive file for
I tried running latest patch and failed with below:
durruti:~/Documents/checkouts/hadoop-commit stack$ more build/test/TEST-org.apache.hadoop.util.TestJarUtils.txt
Testsuite: org.apache.hadoop.util.TestJarUtils
Tests run: 5, Failures: 4, Errors: 0, Time elapsed: 0.258 sec
------------- Standard Output ---------------
2007-08-08 08:00:01,272 INFO util.JarUtils (JarUtils.java:jarAll(311)) - Adding file2:102 to test-out2.jar.
2007-08-08 08:00:01,276 INFO util.JarUtils (JarUtils.java:jarAll(229)) - Adding dir1/dir1-1/dir1-2/file1:0 to test-out2.jar.
2007-08-08 08:00:01,279 INFO util.JarUtils (JarUtils.java:jarAll(229)) - Adding dir2/dir2-1/file2:0 to test-out2.jar.
2007-08-08 08:00:01,280 INFO util.JarUtils (JarUtils.java:jarAll(229)) - Adding dir3/file3:0 to test-out2.jar.
2007-08-08 08:00:01,281 INFO util.JarUtils (JarUtils.java:jarAll(257)) - Adding file4.txt:0 to test-out2.jar.
------------- ---------------- ---------------
Testcase: testGetJarPath took 0.175 sec
FAILED
expected:<.../dir1-2/...> but was:<...\dir1-2\...>
junit.framework.ComparisonFailure: expected:<.../dir1-2/...> but was:<...\dir1-2\...>
at org.apache.hadoop.util.TestJarUtils.testGetJarPath(TestJarUtils.java:88)
Testcase: testJar took 0.022 sec
FAILED
null
junit.framework.AssertionFailedError
at org.apache.hadoop.util.TestJarUtils.testJar(TestJarUtils.java:111)
Testcase: testIsJarOrZip took 0.015 sec
Testcase: testJarAll took 0.024 sec
FAILED
null
junit.framework.AssertionFailedError
at org.apache.hadoop.util.TestJarUtils.testJarAll(TestJarUtils.java:164)
Testcase: testCopyJarContents took 0.018 sec
FAILED
null
junit.framework.AssertionFailedError
at org.apache.hadoop.util.TestJarUtils.testCopyJarContents(TestJarUtils.java:218)
If you are going to make a new patch, here's a couple of other things you could fix: License missing on unit test. Creating files in tests, other folks seem to do something like following to keep them under configured test directory: System.getProperty("test.build.data","."). You might want to do the same. Next time, to save yourself a bit of typing, you could use the local file system in hdfs to do the recursive delete of a directory (I'm guessing thats why you individually remove each of the items in the teardown): FileSystem fs = FileSystem.getLocal(new Configuration()); fs.delete(new Path(f.toString())); In isJarOrZipFile, you could wrap your read of 5 bytes in a try/finally so the close always happens. In fact there are a bunch of places that could do w/ try/finally blocks (Its not critical in the usual case. The job jar will just error out w/o leaving hanging open files). This does not currently pass its own unit tests.
Sorry, I have been caught up in other development for the past couple of months. I will fix any problem with the current code, make sure it passes current unit tests, and upload a revised patch in the next couple of days.
Updated patch now passes all unit tests and is updated for curent 0.15 source. Build.xml has also been changed to allow tests for jarutils. Licenses have been added where applicable.
This patch makes it so there are two ways to specify resources, either with job.setJar(), or job.addResource(). This seems overly complicated. I wonder if we shouldn't rather deprecate getJar/setJar. Instead of removing them later, we might simply make these package-private, since they'll still be the means of passing the compound jar name to tasktrackers, but users should only have to know about addResource(), no?
I am good with deprecating them, just didn't want to break anything with the current patch. Let me know if you want me to build it into the patch, small change.
Any updates on this. Do we need to do anything to get this into Trunk? It has been running successfully for us in production (35 node hadoop cluster) for over a week now.
Here's a new version with setJar/getJar deprecated.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12367974/HADOOP-1622-5.patch  against trunk revision r586003. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs -1. The patch appears to introduce 3 new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/969/testReport/ This message is automatically generated. Findbugs issues need to be addressed.
Fixes for findbugs problems.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12368037/HADOOP-1622-6.patch against trunk revision r586264. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/972/testReport/ This message is automatically generated. Sorry, this patch doesn't apply anymore. Dennis, could you please regenerate the patch with the current trunk.
I've updated this to trunk and will commit it later today barring objections.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12368389/HADOOP-1622-7.patch against trunk revision r588300. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/982/testReport/ This message is automatically generated. I just committed this. Thanks, Dennis!
Integrated in Hadoop-Nightly #283 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/283/
) I had to revert the patch, because it broke
This patch brings the code up to the current trunk. It also fixes a bug in createJobJar in which we needed to using the context classloader to search for classes. This patch passes all unit tests and successfully runs the RandomWriter example.
Dennis, I'm sorry to come in late on this... a couple of comments:
1. Could you please remove the mention of 'final' and 'default' config resources from the javadoc for JobConf.{get|set}JobResources? They are no longer relevant vis-a-vis hadoop Configuration. Unrelated: Does it make sense to rename Configuration.addResource to Configuration.addConfigResource? I wonder how confusing these unrelated api names are, given JobConf is a Configuration too ... Integrated in Hadoop-Nightly #284 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/284/
) 1. Could you please remove the mention of 'final' and 'default' config resources from the javadoc for JobConf.{get|set}JobResources? They are no longer relevant vis-a-vis hadoop Configuration.
I have removed the mention of final and default resources. 2. Should we also have a JobConf.setJobResource along with JobConf.addJobResource, ala {{DistributedCache} apis? I had debated about set vs add resources. The current behavior is when you add a resource you are appending it to a list of resources as opposed to setting a resource which would clear anything previously added and add only that resource. Since many times jar resources are added by including the jar file which contains a given class, I thought it better to NOT allow clearing and resetting of job resources. 3. Should we move the private JobClient.createJobJar method to JarUtils to make it available as a useful utility? I debated about this too. JarUtils was generic jaring and unjaring utilities. But I don't see harm in putting createJobJar in and I think you are right we may need that somewhere else in the future. I have remvoed from JobClient and added to JarUtils. Unrelated: Does it make sense to rename Configuration.addResource to Configuration.addConfigResource? I wonder how confusing these unrelated api names are, given JobConf is a Configuration to Yeah, debated about this one too. In the end we weren't just adding jars but multiple things such as classes, exe, files. Couldn't find a better name for that then resource. I put it as jobResource to be a little less confusing. Changing Configuration over to configResource would be good I think, Although we should probably deprecate because a lot of things rely on that method. I am currently testing patch 9, will have it posted shortly. Updated to most recent trunk and added requested changes.
Integrated in Hadoop-Nightly #286 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/286/
) Owen & I talked a bit about this last week. We determined three commonly useful types of job resources:
All of these should be available from the command line, with -archive, -file and -jar respectively. Owen, does that capture our discussion? What would need to change in the current patch to be consistent with that proposal? Should we file another issue to improve command-line support for these, or should this be done as a part of this issue? I was thinking about this last night. Right now the command line only takes a single jar even though multiple resources are supported through the patch. We could add checking an system property for multiple comma separated resources when the job is submitted. The command line could behind the scenes set this variable. We could have a -resources or a -jobResources a,b,c switch. The current patch handles finding resources from multiple locations including full paths and jars/classes in classpath. We could add in a relative path structure. If this seems reasonable I will work up a patch for it ASAP.
Dennis, Did you get a chance to work on this after your last comment ? We would love to have this available to our users in 0.16.
I have only gotten a chance to design not to develop this as I have been launching the Search Wikia site. Here is what I have come up with in terms of a more generalized design after talking with both Doug and Owen about this enhancement:
1.A runjob utility. runjar is not affected as it is made to only run a single jar. 2.The options parser will be extended to to support resources, upload, classpath, noclasspath, compress, decompress, and cache.
3.Resources
4.Uploads
dennis, any updates on this bug?
No updates yet, but I should have time to start working on this again in the next couple of days, right after I finish some working on converting hadoop RPC to NIO.
great.... we also need this feature to get into 0.17. let me know if you need any hep getting this into 0.17...
Dennis,
Upon looking at this, I'm getting worried. This looks like a lot of special cases. What we really need is to support 3 kinds of files:
for each of these things, we would like them to be able to come from a URI and most convenient would be a default of a local file. So, I propose something like: -file foo,bar,hdfs:baz will upload foo and bar to an upload area and download foo, bar, and baz to the slave nodes as the tasks are run on them. -archive foo.zip,hdfs:baz.zip will download foo.zip and baz.zip and expand them. Finally, the -jar option would download them and put them on the class path. So, -jar myjar.jar,hadoop-0.16.1-streaming.jar would upload the files in the job client, download them to the slaves, and add them to the class path in the given order. I think I'd leave the rsync functionality out and just use hdfs:_upload/$jobid/... as transient storage and delete it when the job is done. If the user wants to save the bandwidth they can upload the files to hdfs themselves, in which case they don't need to be uploaded. marking this for 0.17 release.
am starting working on this ... dennis if you are already working on this please let me know..
I have not resumed working on this as of yet. Am currently neck deep in reworking NIO for hadoop RPC. I was planning on finishing on this as soon as I had completed the NIO code in the next 2-3 days. I would like to continue working on this if possible. When is 0.17 scheduled for release?
Owen, the first pass at this didn't distinguish between jar or regular files on the command line. Instead there was detection code that identified files as such. Also the first pass supported directories as well as files (don't know if you are including that in file). I think the ability to include directories for job input is extremely important. What were the special cases that you were seeing? The idea behind this code is much like streaming you could upload and cache files from any type of resource (file, directory, jar, etc.) from any file system. So, for instance people could store common jars or file resources on S3 and pull them down into a job. alos owen, what would the command line look like with your suggestions?
hadoop jar -file <files> -jar <jars> -archive <archives> ? Also, if that is the case then we could make it generic for streaming which uses its own options for -file , -archives and others .... though we do not need to do that in this patch... i like owens idea. its simple and gives the users the flexibility they need.
here is how I am implementing this – the hadoop command line will have the following options hadoop jar -file <comma seperated files> -jar <comma seperated jars> -archive <comma seperated archives> all of these can be comma seperated uri's – defaulting to local file system if not specified. jobclient uploads the files / jars / archives onto HDFS ..... or the filesystem used by mapreduce. ... under the job directory given that these files/jars/archives might have the same name and different uris.... To keep these files in different directories with the directory name as the uri would give us the ability to just use DistributedCache as it is. so we could say DistributedCache.addFiles(jobdir/file/file/file1, jobdir/hdfs_somehost_port/file1); so the job directory would like jobdir/jars/urischeme/<jarfiles> the one in jars will be added to the classpath of all the tasks in order they were mentioned. comments? Sounds good.
A couple comments: It seems weird to have jar and -jar as arguments/option When the job dir changes to jobdir/jars/urischeme/<jarfiles> will that break the current applications that assume their files loaded using -file and -archive options in the jobdir? ill leave the jar option to keep it backwards compatible. I dont want to break backwards compatibilty for users.
how about
hadoop jar -file <..> -libjar <comma sep jars> -archives<comma seperated> attaching a patch for this feature. It does not have unit tests included. I am still writing unit tests and will upload a patch by the end of the day.
this patch enhances the hadoop command line for job submission: so you can say:
attaching a patch with the unit test. passes the tests on my machine.
Do you think the dns resolution is going to be a big hit. I don't think so with dns caching in place, etc.
Can Pipes make use of this feature (this patch doesn't support pipes). I am ok with having a separate issue to address pipes if required. Otherwise, the patch looks fine. -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378428/HADOOP-1622_2.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 16 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to introduce 3 new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2030/testReport/ This message is automatically generated.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378510/HADOOP-1622_3.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 16 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to introduce 1 new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2040/testReport/ This message is automatically generated. got rid of the findbugs warning.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378527/HADOOP-1622_4.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 16 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2042/testReport/ This message is automatically generated. this is the patch implementing devaraj's coment about host resolution. I will add another jira for this feature to be used by pipes.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378580/HADOOP-1622_5.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 16 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2053/testReport/ This message is automatically generated. looks like the previous patch got stale with some commits yesterday. attaching a new patch.
I just committed this. Thanks Mahadev!
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378649/HADOOP-1622_6.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 16 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2067/testReport/ This message is automatically generated. Integrated in Hadoop-trunk #443 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/443/
) |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Improvement
Closed
Major
Hadoop should provide a way to allow the user to specify jar file(s) the user job depends on
..