Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
https://github.com/nathanmarz/storm/issues/229
When the resources are copied from the resources directory to something like /tmp/09c4e529-7e64-496e-8afa-4990b7aba206/supervisor/stormdist/word-count-1-1337616827/resources the permissions of the files are not preserved, but rather all set to 644. This causes a problem because then we cannot execute binary files as bolts and spouts.
----------
tobigue: I can second this issue. The workaround I ended up with for now is creating a shell script like this in the same folder:
chmod u+x ./relative/path/to/mybinary
./relative/path/to/mybinary
and calling
super("sh", "myscript.sh")
in the ShellSpout/ShellBolt instead of calling the binary directly.
----------
analogue: Tobigue,
I don't see how your workaround actually works because myscript.sh will not be executable after after being unzipped and will fail to run.
My workaround is as follows:
For a given spout or bolt:
public static class MySpout extends ShellSpout implements IRichSpout {
public MySpout()
{ super("bash", "-c", "chmod +x ./pystorm;./pystorm my_spout.py arg1 arg2 argN"); }And a shell script multilang/resources/pystorm:
#!/bin/sh
PYTHONPATH=`pwd` python $*
This also handles not being able to set environment variables 
I'm thinking that ShellSpout should have an additional constructor which takes a ProcessBuilder to provide maximum flexibility. There are downsides to this though. The author of the spout or bolt will then have access to the ProcessBuilder and could do something to screw things up like inadvertently reading/writing to the InputStream or OutputStream in an effort to debug things.
----------
tobigue: Hi!
I think for me it worked, because I had the permission to execute sh.
So this works:
sh myscript.sh
while this does not
./myscript.sh
as there are no execute permissions on the file itself.
----------
analogue: Ok, I see my problem. bash -c ./myscript.sh fails if the script is not executable but bash ./myscript.sh doesn't.
----------
alien2150: I am having the same problem with a C++ programm. The original permissions are:
ls -l target/classes/resources/xy
-rwxr-xr-x 1 tech tech 1080184 Jul 19 17:26 target/classes/resources/xy
But later it is (ls -l /tmp/448ce6ba-ccd2-4335-a119-903381aee09c/supervisor/stormdist/foo-1374248239/resources):
rw-rw-rw 1 tech tech 1080184 Jul 19 17:37 xy
Is there any solution for this?
----------
revans2: jar files are really just zip files and do not store any kind of permissions. There are some extensions to zip that do store permissions but jar does not use those. Because these resources are pushed to the nodes through a jar the permissions are being lost. Hadoop gets around this by doing the really ugly thing of changing all of the permissions to be user executable after unzipping the file. We could try something like this in the supervisor when it unzips the files but I would rather provide a different way of distributing files beyond a single jar file once the bit-torrent code is in place.
----------
patricklucas: We can reproduce this consistently by launching a topology to a "fresh" cluster. It appears that if the storm-local directory has to be created, unpacked files will not have the right permissions. If it is a subsequent run, and the storm-local directory already exists, permissions of unpacked files are correct.
Will this be fixed in 0.9.0?