Lucy
  1. Lucy
  2. LUCY-215

Support extensions written in C

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: Clownfish
    • Labels:
      None

      Description

      Currently, all Lucy extensions that contain C code must be shipped with the Lucy source tree. In order to support external C-based extensions, Clownfish must be able to compile .cfh files that extend classes from a separate source tree. As discussed on the mailing list, a solution could look like this:

      • Install the .cfh files together with Lucy.
      • When building the extension, rebuild the complete Clownfish hierarchy together with the new classes.
      • Generate C headers, regenerating the headers for Lucy's core classes.
      • Compile the extension.
      1. Build.PL
        0.8 kB
        Nick Wellnhofer
      2. 0019-LUCY-215-Add-autogen-source-directly-to-CF-build-sou.patch
        2 kB
        Nick Wellnhofer
      3. 0018-LUCY-215-Supply-some-build-params-via-Lucy-Build-con.patch
        4 kB
        Nick Wellnhofer
      4. 0017-LUCY-215-Add-C-C-P-B-cf_base_path.patch
        5 kB
        Nick Wellnhofer
      5. 0016-LUCY-215-Introduce-C-C-P-B-cf_copy_include.patch
        3 kB
        Nick Wellnhofer
      6. 0015-LUCY-215-Rework-installation-of-Clownfish-includes.patch
        4 kB
        Nick Wellnhofer
      7. 0014-LUCY-215-Add-build-param-for-core-source-dir.patch
        6 kB
        Nick Wellnhofer
      8. 0013-LUCY-215-Break-out-Clownfish-CFC-Perl-Build.patch
        30 kB
        Nick Wellnhofer
      9. 0012-LUCY-215-Add-build-param-for-autogen-header.patch
        1 kB
        Nick Wellnhofer
      10. 0011-LUCY-215-Add-build-param-for-Clownfish-include-dirs.patch
        2 kB
        Nick Wellnhofer
      11. 0010-LUCY-215-Add-build-param-for-extra-C-sources.patch
        2 kB
        Nick Wellnhofer
      12. 0009-LUCY-215-Custom-Module-Build-property-for-extra-buil.patch
        1 kB
        Nick Wellnhofer
      13. 0008-LUCY-215-Use-Module-Build-s-include_dirs.patch
        3 kB
        Nick Wellnhofer
      14. 0007-LUCY-215-Derive-build-file-names-from-module_name.patch
        6 kB
        Nick Wellnhofer
      15. 0006-LUCY-215-Use-Module-Build-s-dist_version.patch
        1 kB
        Nick Wellnhofer
      16. 0005-LUCY-215-Process-Binding-classes-from-all-namespaces.patch
        0.8 kB
        Nick Wellnhofer
      17. 0004-LUCY-215-Derive-parcel-and-boot_class-from-module_na.patch
        1 kB
        Nick Wellnhofer
      18. 0003-LUCY-215-Allow-filename-clash-between-source-and-inc.patch
        4 kB
        Nick Wellnhofer
      19. 0002-Implement-CFCFileSpec.patch
        35 kB
        Nick Wellnhofer
      20. 0001-Use-path_part-instead-of-source_class-in-CFC.patch
        31 kB
        Nick Wellnhofer

        Activity

        Hide
        Nick Wellnhofer added a comment -

        I created a new branch LUCY-215-cf-extensions. As a first step, I added support for multiple source directories to Clownfish.

        Show
        Nick Wellnhofer added a comment - I created a new branch LUCY-215 -cf-extensions. As a first step, I added support for multiple source directories to Clownfish.
        Hide
        Marvin Humphrey added a comment -

        Here's a 5-patch sequence eliminating the generation of Autobinding.pm.

        Show
        Marvin Humphrey added a comment - Here's a 5-patch sequence eliminating the generation of Autobinding.pm.
        Hide
        Marvin Humphrey added a comment -

        The commits on the branch have been replayed on trunk, and I've now committed
        my patch sequence. Once the branch is deleted, I think this issue can be
        resolved as "fixed".

        Show
        Marvin Humphrey added a comment - The commits on the branch have been replayed on trunk, and I've now committed my patch sequence. Once the branch is deleted, I think this issue can be resolved as "fixed".
        Hide
        Nick Wellnhofer added a comment -

        Here is a patch set that introduces a new CFCFileSpec class. This makes some of my previous changes cleaner IMO. On top of that I added the handling of filename clashes between source and include dirs.

        Show
        Nick Wellnhofer added a comment - Here is a patch set that introduces a new CFCFileSpec class. This makes some of my previous changes cleaner IMO. On top of that I added the handling of filename clashes between source and include dirs.
        Hide
        Nick Wellnhofer added a comment -

        As previously discussed on the mailing list, we should also provide parts of the code in buildlib/Lucy/Build.pm as a Clownfish module, so extensions can be built easily. To handle different build configuration variables like module name, additional C sources, etc., I would propose to create a BuildConfig class that provides all these variables via methods. Then, every extensions can optionally extend the class and create a BuildConfig object. This object can be stored as a "property" of the Module::Build object. Build configuration includes:

        • Main class name
        • Parcel
        • Additional C sources, include dirs and flags
        • Additional libraries and linker flags
        • Clownfish include directories
        • Other files to install in Clownfish/_include

        Open questions:

        • How to name the build module? Clownfish::CFC::Build?
        • Do we want provide an equivalent to Lucy::Build::CBuilder? Probably yes.
        • We might need a better way to find the Binding build packages.
        • Should we support parts of the Valgrind infrastructure?
        Show
        Nick Wellnhofer added a comment - As previously discussed on the mailing list, we should also provide parts of the code in buildlib/Lucy/Build.pm as a Clownfish module, so extensions can be built easily. To handle different build configuration variables like module name, additional C sources, etc., I would propose to create a BuildConfig class that provides all these variables via methods. Then, every extensions can optionally extend the class and create a BuildConfig object. This object can be stored as a "property" of the Module::Build object. Build configuration includes: Main class name Parcel Additional C sources, include dirs and flags Additional libraries and linker flags Clownfish include directories Other files to install in Clownfish/_include Open questions: How to name the build module? Clownfish::CFC::Build? Do we want provide an equivalent to Lucy::Build::CBuilder? Probably yes. We might need a better way to find the Binding build packages. Should we support parts of the Valgrind infrastructure?
        Hide
        Nick Wellnhofer added a comment -

        Regarding Lucy::Build::CBuilder: It's main use is to provide a working link_executable method for Windows. This has been fixed in ExtUtils::CBuilder 0.26_04, so Perl 5.12 and above shouldn't need this workaround. The Valgrind stuff in Lucy::Build::CBuilder can be moved to Lucy::Build, so I'd propose to get rid of Lucy::Build::CBuilder completely.

        Show
        Nick Wellnhofer added a comment - Regarding Lucy::Build::CBuilder: It's main use is to provide a working link_executable method for Windows. This has been fixed in ExtUtils::CBuilder 0.26_04, so Perl 5.12 and above shouldn't need this workaround. The Valgrind stuff in Lucy::Build::CBuilder can be moved to Lucy::Build, so I'd propose to get rid of Lucy::Build::CBuilder completely.
        Hide
        Marvin Humphrey added a comment -

        So, zapping Lucy::Build::CBuilder would have the effect of requiring
        ExtUtils::CBuilder 0.26_04 for MSVC builds on Windows. This definitely
        doesn't seem like a reason to bump the minimum required version of either Perl
        or ExtUtils::CBuilder for everyone.

        IMO this is OK so long as we throw a sensible error informing affected users
        that they need to upgrade.

            use ExtUtils::CBuilder;
        
            # ExtUtils::CBuilder#link_executable not implemented for Windows until
            # version 0.26_04.
            if ( $ExtUtils::CBuilder::Version < 0.2604 and $^O =~ /mswin/i ) {
                confess("Version 0.26_04 of ExtUtils::CBuilder is required for "
                      . "your system." );
            }
        
        Show
        Marvin Humphrey added a comment - So, zapping Lucy::Build::CBuilder would have the effect of requiring ExtUtils::CBuilder 0.26_04 for MSVC builds on Windows. This definitely doesn't seem like a reason to bump the minimum required version of either Perl or ExtUtils::CBuilder for everyone. IMO this is OK so long as we throw a sensible error informing affected users that they need to upgrade. use ExtUtils::CBuilder; # ExtUtils::CBuilder#link_executable not implemented for Windows until # version 0.26_04. if ( $ExtUtils::CBuilder::Version < 0.2604 and $^O =~ /mswin/i ) { confess("Version 0.26_04 of ExtUtils::CBuilder is required for " . "your system." ); }
        Hide
        Nick Wellnhofer added a comment -

        No, I wouldn't bump the version requirements for everyone. Just put in a warning like you proposed, or maybe monkey patch link_executable on incompatible platforms.

        Show
        Nick Wellnhofer added a comment - No, I wouldn't bump the version requirements for everyone. Just put in a warning like you proposed, or maybe monkey patch link_executable on incompatible platforms.
        Hide
        Marvin Humphrey added a comment -

        I just realized we don't need the warning either. We used to need
        link_executable() when we were building charmonize[.exe] within Lucy::Build.
        That's no longer the case, as it's built using charmonizer/Makefile.MSVC on
        the affected platform. (Grep for link_executable in our 0.1 branch for
        the relevant code.)

        +1 to zap Lucy::Build::CBuilder.

        Show
        Marvin Humphrey added a comment - I just realized we don't need the warning either. We used to need link_executable() when we were building charmonize [.exe] within Lucy::Build. That's no longer the case, as it's built using charmonizer/Makefile.MSVC on the affected platform. (Grep for link_executable in our 0.1 branch for the relevant code.) +1 to zap Lucy::Build::CBuilder.
        Hide
        Marvin Humphrey added a comment -

        The CFCFileSpec patch sequence looks great! I think there might be
        a little wackiness on windows where CFCUTIL_PATH_SEP is a backslash,
        but it will be easy to fix. +1 to commit!

        Show
        Marvin Humphrey added a comment - The CFCFileSpec patch sequence looks great! I think there might be a little wackiness on windows where CFCUTIL_PATH_SEP is a backslash, but it will be easy to fix. +1 to commit!
        Hide
        Nick Wellnhofer added a comment -

        Patches 0004 - 0008 contain some changes to Lucy::Build that are in preparation of breaking out parts of it into Clownfish::CFC::Perl::Build. Many build parameters can be handled using the Module::Build machinery.

        The following parameters still need work:

        Extra C sources

        Module::Build has a 'c_source' parameter that also supports a list of directories starting with version 0.3604. If we start using this parameter, I think we can even remove ACTION_compile_custom_xs completely and let M::B compile and link all .xs and .c files. If we don't want to bump the M::B version requirement, we have to find another way to pass extra C source dirs.

        Clownfish include dirs

        The directories with .cfh files to include. For now, a switch that turns the Clownfish/_include dirs in $Config

        {installsitearch}

        and $Config

        {installvendorarch}

        on or off should be enough.

        Autogen header

        Since the autogen header might contain a license, there should be a way for extensions to set their own header.

        Show
        Nick Wellnhofer added a comment - Patches 0004 - 0008 contain some changes to Lucy::Build that are in preparation of breaking out parts of it into Clownfish::CFC::Perl::Build. Many build parameters can be handled using the Module::Build machinery. The following parameters still need work: Extra C sources Module::Build has a 'c_source' parameter that also supports a list of directories starting with version 0.3604. If we start using this parameter, I think we can even remove ACTION_compile_custom_xs completely and let M::B compile and link all .xs and .c files. If we don't want to bump the M::B version requirement, we have to find another way to pass extra C source dirs. Clownfish include dirs The directories with .cfh files to include. For now, a switch that turns the Clownfish/_include dirs in $Config {installsitearch} and $Config {installvendorarch} on or off should be enough. Autogen header Since the autogen header might contain a license, there should be a way for extensions to set their own header.
        Hide
        Nick Wellnhofer added a comment -

        Patches 0009 - 0012 add a custom Module::Build hashref property named 'clownfish_params' with the following entries:

        • extra_c_sources
        • cf_include_dirs
        • autogen_header

        Then, patch 0013 breaks out Clownfish::CFC::Perl::Build from Lucy::Build.

        Show
        Nick Wellnhofer added a comment - Patches 0009 - 0012 add a custom Module::Build hashref property named 'clownfish_params' with the following entries: extra_c_sources cf_include_dirs autogen_header Then, patch 0013 breaks out Clownfish::CFC::Perl::Build from Lucy::Build.
        Hide
        Nick Wellnhofer added a comment -

        Patches 0014 and 0015 contain some additional cleanup.

        I also attached a Build.PL for a sample extension. That little bit of code is all that's needed to build an extension!

        Show
        Nick Wellnhofer added a comment - Patches 0014 and 0015 contain some additional cleanup. I also attached a Build.PL for a sample extension. That little bit of code is all that's needed to build an extension!
        Hide
        Nick Wellnhofer added a comment -

        Patches up to 0015 are now committed to trunk including changes that were discussed on lucy-dev.

        Here are some additional patches 0016 - 0019 that I'd like to add on top of my previous work.

        As I noted on lucy-dev I think we should make it easy for extension writers to work with the same "core", "lucy", "ruby" directory layout for their extensions. This includes copying "core" and possibly other files to the host language directory when building distributions. I added a C::C:::B->cf_base_path methods to help with that.

        I'd also like to create a "dist" action in C::C:::B based on ACTION_dist in Lucy::Build that helps with building extensions distros.

        Show
        Nick Wellnhofer added a comment - Patches up to 0015 are now committed to trunk including changes that were discussed on lucy-dev. Here are some additional patches 0016 - 0019 that I'd like to add on top of my previous work. As I noted on lucy-dev I think we should make it easy for extension writers to work with the same "core", "lucy", "ruby" directory layout for their extensions. This includes copying "core" and possibly other files to the host language directory when building distributions. I added a C::C: ::B->cf_base_path methods to help with that. I'd also like to create a "dist" action in C::C: ::B based on ACTION_dist in Lucy::Build that helps with building extensions distros.
        Hide
        Nick Wellnhofer added a comment -

        Now that we really support multiple parcels and dropped the dependency on Charmonizer, I consider this issue resolved.

        Show
        Nick Wellnhofer added a comment - Now that we really support multiple parcels and dropped the dependency on Charmonizer, I consider this issue resolved.

          People

          • Assignee:
            Nick Wellnhofer
            Reporter:
            Nick Wellnhofer
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development