[SPARK-39817] Missing sbin scripts in PySpark packages

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0, 3.2.1, 3.3.0, 3.2.2
• Component/s: PySpark
• Labels: easyfix
• Flags: Patch, Important

Language

python

• Assignee: Unassigned
• Reporter: F. H.
• Created: 19/Jul/22 13:02
• Updated: 12/Dec/22 18:10

Description

In the PySpark setup.py, only a subset of all scripts is included.
I'm in particular missing the `submit-all.sh` script:

        package_data={
            'pyspark.jars': ['*.jar'],
            'pyspark.bin': ['*'],
            'pyspark.sbin': ['spark-config.sh', 'spark-daemon.sh',
                             'start-history-server.sh',
                             'stop-history-server.sh', ],

            [...]
        },

 

The solution is super simple, just change 'pyspark.sbin' to:

'pyspark.sbin': ['*'],

 

I would happily submit a PR to github, but I have no clue on the organizational details.

This would be great to get backported for pyspark 3.2.x as well as 3.3.x soon.