Details
Minor Description
NOTE: I noticed you guys recently switched from 0.15 to 0.99, so at certain point, I will pull in lastest updates from MiNiFi C++ main branch on parent level and rebase or merge into my PR.
Description of Issue: Building MiNiFi C++ with just CMake is really slow. During the the build process, it clones all the external dependencies and then builds those external dependencies while its building MiNiFi CPP. Sometimes the MiNiFi CPP build fails too when external lib dependencies fail to download or fail to build. It would be faster to build MiNiFi C++ with the external dependencies already preinstalled.
Potential Solution: Integrate conan version 2 into MiNiFi C++ project to enable support for using conan install to install all or most of MiNiFi's external dependencies, using conan install --build=missing to build them as prebuilt binary conan packages, upload them to conancenter with conan upload, and then run cmake generate to generate the appropriate build files for the OS, then run make to build MiNiFi CPP. Also conan has really good support for cross platform compatibility for Linux and Windows. At this point because we already have most of MiNiFi CPP's external lib dependencies installed with conan, our CMake build will just focus on building the MiNiFi CPP code, enabling faster builds. Also I will need to account for backward compatibility support with the previous way of installing MiNiFi CPP external lib dependencies using CMake FetchContent_Declare(...) approach until conanfile.py approach covers installing all these dependencies. A valid verification of the MiNiFi C++ build would be to run GTESTs and if all the expected test cases pass, then it may be fair to conclude we would be ready to deprecate the slow CMake FetchContent_Declare(...) approach.
Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into script(s)):
- Install conan version 2 into MiNiFi C++ dev environment (my preference is using docker container)
- Create a conanfile.py file at root of MiNiFi C++ project where we run "conan install" or "conan install --build=missing" to install MiNiFi C++ external lib dependencies first. Then we run CMake to build MiNiFi C++.
- Find all MiNiFi C++ find_package(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages.
- Find all FetchContent_Declare(...) CMake function calls and check if there is already supported conan packages on the conancenter, so we can install those packages.
- At this point we will have used conan version 2 C++ package manager to install almost all MiNiFi C++'s external dependencies.
- With Conan, we will have installed these MiNiFi external lib dependencies as prebuilt binary conan packages.
- Or we will have instructed conan to first build each C++ external lib dependency, upload each lib to conancenter.
- Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we can then run CMake to build MiNiFi faster.
Here is my draft PR for this Jira Ticket for your reference: https://github.com/apache/nifi-minifi-cpp/pull/1775
Here are the commands that I run:
# make sure to install conan2 for your environment sudo pip install --force-reinstall -v "conan==2.3.1" # conanfile.py is in root dir of MiNiFi C++ project cd $HOME/nifi-minifi-cpp # create MINIFI_HOME env variable for binary executable minifi export MINIFI_HOME=$(pwd) # In case you need to delete all conan packages # conan remove "*" -c # install conan packages for MiNiFi C++ using conanfile.py invoking Conan conan install . --build=missing --output-folder=build_conan -pr=$HOME/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # build MiNiFi C++ using conanfile.py invoking Conan & CMake conan build . --output-folder=build_conan -pr=$HOME/nifi-minifi-cpp/etc/build/conan/profiles/release-linux # verify we can run minifi binary executable ./build_conan/bin/minifi
UPDATE (May 21, 2024): I created a small PR for conan integration on top of Gabor Gyimesi's PR-1749 "https://github.com/apache/nifi-minifi-cpp/pull/1749" that brings in conan packages for OpenSSL, LibCurl and Zlib as alternatives to their system library counterparts when we run standalone CMake to build MiNiFi CPP. I also included steps for Maton Szasz and others to get up to speed with how to use conan v2 to build MiNiFi CPP. On top of Gabor's PR-1749, my PR-261 (MINIFICPP-2346-P1) has +325 -225 git diff and only 5 files changed: https://github.com/lordgamez/nifi-minifi-cpp/pull/261
UPDATE(May 14, 2024): Updated due date to May 19, 2024 since the refactoring of MiNiFi CPP build infra over to conan version 2 is taking longer than expected. The good news is that I believe I have more than half of MiNiFi external libraries now managed by conan with backward compatibility for standalone CMake approach.
UPDATE (May 13, 2024) - 22205c9: Fetch and Pulled latest upstream main branch of parent MiNiFi to my fork main branch of MiNiFi repo; thus, merged in the latest updates from main branch of parent MiNiFi to my MINIFICPP-2346 branch. Resolved conflicts between a few CMake files coming from Date.cmake and BundledRocksdb.cmake files. For standalone CMake build appraoch, ran GTESTs with 100% (238/238) tests passing. I did the conan build approach and ran GTESTs with 99% (237/238) passing.
UPDATE (May 13, 2024) - 59ffefb: Enabled LIBRDKAFKA & GRPC_FOR_LOKI MiNiFi C++ extension using Conan & CMake, Improved Running Python Maven Executable for JNI & SFTP Test Server for conan approach and Improved standalone CMake approach for building PcapPlusPlus with its dependency on LibPcap. LIBRDKAFKA required creating a librdkafka conan package passing similar CMake definitions to it to build it like we did for standalone CMake approach. GRPC_FOR_LOKI required creating a grpc conan package that uses minifi's openssl and libsystemd conan packages. In both LIBRDKAFKA & GRPC_FOR_LOKI, I updated their .cmake Bundles to account for the condition of using their conan package, so minimal changes would be needed to their extension CMakeLists.txt file. In LIBRDKAFKA's extension CMakeLists.txt file, it didnt really change after integrating the conan package since I created a library alias for it similar to the standalone CMake approach. In GRPC_FOR_LOKI's extension CMakeLists.txt, minimal changes were made to that file mainly on the add_custom_command, target_include_directories and target_link_libraries commands. You'll see for Grpc.cmake for conan approach, I created similar variables to standalone CMake approach for GRPC_CPP_PLUGIN, INCLUDE_DIR, PROTOBUF_INCLUDE_DIR, PROTOBUF_COMPILER getting the conan related properties. For the conan side, I also made improvements to the Python Maven Executable Wrapper script since I noticed there were times at the beginning when running maven version test, it didn't always run it successfully, so I updated conan maven executable path to be explicit for passing to Python script. I then moved the location of running maven version check into a CMake function and ran it after we included JavaMaven and JavaOpenJDK conan packages in the MiNiFi root CMakeLists.txt file. Thus, it succesfully runs maven version check at beginning each time. Its important to make sure we have conan maven to successfully build NiFi JNI and Java SFTP Test Server. Finally, on the standalone CMake approach for building PcapPlusPlus, I noticed some issues with PCAP headers not being found when building PcapPlusPlus, so I switched from execute_process to run the configure scripts to using ExternalProject_Add and I also added a bundle for LibPcap since the system LibPcap version wasn't always installed on the system. This problem with building PcapPlusPlus dependency on LibPcap didn't happen with my conan approach. It took a lot of time to fix the standalone CMake backward compatibility approach, but I did it for now since we're keeping support that approach. 99% of GTESTs ( 237/238) passed for conan.
UPDATE (May 11, 2024) - c96d430: Refactoring MiNiFi C++ build infrastructure adding Conan Packager support
Enabled JNI, SENSORS, USB_CAMERA, OPENCV, and SFTP, which required creating Maven 3.9.6 conan package updating the conan recipe to expose the Maven Executable environment variable in CMake and installing prebuilt Java OpenJDK 21.0.2 conan package and creating a python wrapper run_maven.py script for calling the conan maven Java build tool to successfully build Java SFTP Test Server for SFTP extension testing and maven to build Apache NiFi 1.9.0 JNI Assembly and Framework; We had to go the python maven wrapper script approach since I ran into some issues calling conan maven executable from CMake directly; Also added prebuilt install for libssh2 conan package toward SFTP extensions; Thus, now the Java support for MiNiFi C++ is less reliant on the user’s system since we can handle building this part of MiNiFi C++ using conan; Also created an OpenCV 4.8.1 conan package updating the conan recipe to use MiNiFi C++ conan package dependencies from libtiff, ffmpeg and xz_utils (liblzma) and successfully build MiNiFi OpenCV extension; Also installed prebuilt libuvc conan package for enabling USB_CAMERA. For the conan build approach, 99% (235/236) of GTESTs passed. For the standalone CMake build approach, 99% (234/236) of GTESTs passed.
- Also we could look at updating Apache NiFi 1.9.0 to latest 2.0 version. I noticed we use a Java SFTP Test Server now that NiFi has native python custom extensibility, maybe we could look at Python SFTP Test Server?
UPDATE (May 11, 2024) - 64dd8ad: Enabled Build LUA SCRIPTING, ENCRYPT CONFIG, SPLUNK, ELASTIC SEARCH, TEST PROCESSORS, GRANFANA LOKI and CONTROLLER by Conan & CMake. For building these components with conan, the LUA SCRIPTING extension required me updating its Lua.cmake and Sol2.cmake files, so we can use the conan packages in CMake when USE_CONAN_PACKAGE condition is true. For lua and sol2 conan packages, I was able to just install the prebuilt binary conan packages from conancenter directly since conancenter provided the versions that match with MiNiFi C++'s standalone CMake approach's lua 5.4.6 and sol2 3.3.0 source built versions. MiNiFi controller being enabled required updating the ArgParse.cmake file, so we could also account for the condition when we build the controller using conan, thus I added back using the argparse conan package. After enabling these components, I was able to successfully build them for MiNiFi using conan. I also verified backward compatibility for building these components in standalone CMake worked too. We achieved similar results for GTESTs where out of 230 tests, 99% of them passed for conan build and 100% passed for standalone CMake build.
UPDATE (May 11, 2024) - f5cfb79: Enabled Build MQTT & PCAP MiNiFi PCAP Extension by Conan & CMake. Refactored the pcap extension CMake code moving the part that downloads and builds pcapplusplus over to its own PcapPlusPlus.cmake file. Also added conan build condition to that .cmake file, so we can build PcapPlusPlus using conan while still keeping backward compatibility using standalone CMake approach. While I was able to build MiNiFi C++ with PCAP Extension enabled using standalone CMake approach, when I built it using conan build approach, I found pcap extension complaining about libminifi core headers and realized include_directories(include) didnt share all the header directory paths from libminifi to pcap extension. Thus, I created a python script under libminifi/cmake/python to parse for the libminifi header directories generating a include_dirs.cmake file with the LIBMINIFI_INCLUDE_DIRS CMake variable list that gets stored in build folder and followed by libminifi/include_dirs.cmake. That way we keep our source libminifi directory and its subdirectories clean. Overall, I was able to build MiNiFi C++ with MQTT and PCAP extensions enabled using conan & cmake. conan build achieved 99% (222/223) of GTESTs passing. standalone cmake build achieved 100% of GTESTs passing.
UPDATE (May 9, 2024) - 63290db: Refactored IODBC and SOCI libraries, so they are now CMake Bundles for easier integration with Conan. Thus, minifi-sql library now calls include(BundledIODBC), use_bundled_iodbc(...) and include(BundledSOCI) and use_bundled_soci(...). Verified I can still build MiNiFi C++ using standalone CMake. 100% (220/220) of GTESTS passed using standalone CMake. Integrated conan for building SOCI & ODBC for MiNiFi SQL extension. 4232399 Added a SOCI conan recipe to create SOCI conan package with unixODBC conan package, so SOCI comes with targets SOCI::libsoci_core and SOCI::lib;soci_odbc. Verified I can build MiNiFi C++'s minifi-sql extension using conan build. However, with the conan build approach for minifi-sql extension, I was only able to build minifi-sql as a STATIC library succesfully. When I tried building minifi-sql as a SHARED library, I ran into multiple undefined references to SOCI functions. I think to fix the minifi-sql SHARED lib with conan will require making further updates to the SOCI conanfile.py's package_info() method. After building MiNiFi with conan including minifi-sql, I ran GTESTs and 99% of tests passed. I will look further into minifi-sql SHARED library to bring that approach later. For now, at least we can build minifi-sql as a STATIC library.
UPDATE (May 9, 2024) - ca3d6df: Enabled MiNiFi C++ extensions GPS and COAP. I added conan recipe for libcoap keeping the windows cmake patches for minifi. Since the original libcoap conan recipe's conanfile.py on conan-center-index was tailored for CMake building libcoap versions > 4.3.0, I modified the recipe, so we can build libcoap 4.2.1, which didnt have CMake support yet and thus, I updated that conanfile.py to run 'autogen.sh' to create the configure script, I passed the configure args to the configure script and then I used conan's Autotools() to run the configure script and then ran the generated make file to build libcoap 4.2.1. I updated BundledLibCoAP to account for the conan way, but kept backward compatibility for standalone CMake way. For a later update when I test on Windows, I need to add support into the libcoap 4.2.1 conanfile.py for adding and applying the windows cmake patch to build libcoap. For now, I am mainly testing this new conan build infra for MiNiFi on Ubuntu 22.04 docker container and backward compatibility for standalone CMake on Ubuntu too. 99% of GTESTs passed (215/216).
(UPDATE (May 8, 2024) - 93fe5d1: Enabled NANOFI lib and binary executable, Enabled MiNiFi C++ extensions SYSTEMD, PROCFS, LIBARCHIVE and XY_UTILS (LZMA). I added conan recipe for liblzma (xy_utils) and libarchive keeping the patches for minifi. One area I found an issue with was integrating openssl conan package with libarchive conan package and libarchive complained about not being able to find openssl::openssl target. I did try to solve the issue by updating the libarchive conan recipe, but wasn't able to solve that openssl integration issue. I think the problem comes from libarchive CMakeLists.txt and its integration with conan in the repo. I opened an issue on conan-center-index to help with resolving this issue in case I get around to solving it in time. Good news is that I was still able to create libarchive conan package with the configuration settings similar to how we do it the standalone CMake BundledLibArchive.cmake way just in libarchive conan recipe. I also verified I can build these minifi extensions (LibArchive, LZMA, Procfs, Systemd) and NANOFI using conan build and standalone CMake. The standalone CMake way still allows us to build LibArchive with OpenSSL enabled. 213/214 (99%) GTESTs passed after building MiNiFi C++ with conan.
UPDATE (May 7, 2024) - 2460e84: Enabled OPS and OPC MiNiFi Options, so those MiNiFi C++ extensions are built using Conan CMake and their bundled CMake dependencies use their particular open62541/1.3.3@minifi/dev and mbedtls/2.16.3@minifi/dev conan packages. I had to fix the mbedtls conan package from conan center index since it wasnt compatible when open62541 tried to integrate with mbedtls. In the thirdparty folder, you will see on line 139 where I updated libembedtls to be libmbedtls and fixing that typo allowed open62541 conan package to be able to be created with mbedtls as the encryption. I plan to make a follow up PR on conan center index for mbedtls conan recipe. Currently, we are able to build MiNiFi's additional extensions for OPS and OPC using conan. Verified backward compatibility with standalone CMake works, so I can also just build MiNiFi using CMake only.
UPDATE (May 5, 2024) - 446ddb6: When building MiNiFi C++ GTESTs for MiNiFi Core, MainExe and Standard-Processors and the other relevant that needed to be activated for these 3 essential components, I also set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON to build GTESTs using conan build. I updated Bundle Rocksdb and Find Rocksdb to account for conan approach while keeping backward compatibility with standalone CMake approach. I got the following error coming from extension robksdb-repos/database/RocksDbInstance.cpp on no match for operator, and I will add the fix for this issue:
/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/extensions/rocksdb-repos/database/RocksDbInstance.cpp:71:26: error: no match for ‘operator!=’ (operand types are ‘rocksdb::DBOptions’ and ‘rocksdb::DBOptions’) 71 | if (db_opts_copy != db_options_) { | ~~~~~~~~~~~~ ^~ ~~~~~~~~~~~ | | | | | rocksdb::DBOptions | rocksdb::DBOptions
We ran into this issue since when we created rocksdb/8.10.2 conan package, we didn't apply the 2 patches like was done in the standalone CMake approach, so I updated our conan approach where I created a rocksdb conan recipe in the nifi-minifi-cpp/thirdparty/rocksdb/ folder and added the 2 patches and applied them to create the rocksdb conan package successfully. Also I do want to note that I referenced the rocksdb conan recipe from conan center index repo. After creating the rocksdb conan package, then I built MiNiFi C++ GTESTs for MiNiFi Core, MainExe, Standard-Processors and the other relevant extensions that needed to be activated for these 3 essential components using conan build. Here we have the results after running ctest in the build folder:
192/194 Test #192: ConfigFileEncryptorTests .................... Passed 0.02 sec Start 193: ConfigFileTests 193/194 Test #193: ConfigFileTests ............................. Passed 0.01 sec Start 194: ControllerTests 194/194 Test #194: ControllerTests .............................***Failed 0.52 sec 99% tests passed, 2 tests failed out of 194 Total Test time (real) = 382.89 sec The following tests FAILED: 45 - LoggerTests (Failed) 194 - ControllerTests (Failed)Errors while running CTestOutput from these tests are in: /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_conan/Testing/Temporary/LastTest.logUse "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
We can see that 99% of the tests passed. Thus, 192/194 tests passed. This was using conan to build MiNiFi C++.
I also verified I can build MiNiFi C++ GTESTs for MiNiFi Core, MainExe and Standard-Processors and the other relevant that needed to be activated for these 3 essential components, I also set SKIP_TESTS to OFF, I set ENABLED_EXPRESSION_LANGUAGE to ON, ENABLE_BZIP2 to ON, BUILD_ROCKSDB to ON, ENABLE_ROCKSDB to ON to build GTESTs using standalone CMake. Here we can see similar results with running the MiNiFi GESTs:
192/194 Test #192: ConfigFileEncryptorTests .................... Passed 0.07 sec Start 193: ConfigFileTests 193/194 Test #193: ConfigFileTests ............................. Passed 0.08 sec Start 194: ControllerTests 194/194 Test #194: ControllerTests .............................***Failed 0.76 sec 99% tests passed, 1 tests failed out of 194 Total Test time (real) = 236.38 sec The following tests FAILED: 194 - ControllerTests (Failed)Errors while running CTestOutput from these tests are in: /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_cmake2/Testing/Temporary/LastTest.logUse "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
We can see that 99% of the tests passed. Thus, 193/194 tests passed. This was using standalone CMake to build MiNiFi C++.
UPDATE (May 4, 2024) - 7eb8beb: Verified I can build MiNiFi C++ with standalone CMake like how we currently do it even after integrating conan into the build infrastructure. CMake built MiNiFi Core, Main and Standard Processors successfully. Also double checked that conan build MiNiFi still works. (Feel free to reference the logs for the CMake standalone build and the conan build of MiNiFi CPP)
UPDATE (May 4, 2024) - 2d1b884: Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi core-minifi and MiNiFi Main's minifiexe and minifi-standard-processors extension with minimal set of external libraries needed. If you look at our conanfile.py, you'll see minifi_core_external_libraries, which is a tuple of the minimal external libraries needed to build these two essential building blocks of MiNFi CPP. Next when you look at conanfile.py MiNiFiCppMain class's generate(self) method, you'll see I have enabled USE_CONAN_PACKAGER to ON, SKIP_TESTS to ON, and then the essential external library dependencies that aren needed for the core-minifi library and minifiexe binary executable, which are ENABLE_OPENWSMAN, ENABLE_OPENSSL, ENABLE_CIVET, ENABLE_CURL. I notice most of these external libraries are in the extensions folder and add some dependencies on these custom extensions that are created. I think we should consider creating an extensions subfolder for core-minifi related extensions.
Here is the output I get after building the MiNiFi CPP code, you can see MiNiFi CPP libraries and binary executables built successfully using CONAN version 2 and CMake (feel free to check the attached log for the full conan build output): szaszm
Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT list Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT -- Creating API Documentation using /usr/bin/doxygen -- Configuring done -- Generating done -- Build files have been written to: /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py (minifi-cpp-main/0.15.0): Running CMake.build() conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build "/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20 [ 3%] Built target ossp-uuid-external-build [ 6%] Built target openwsman-external [ 7%] Built target ossp-uuid-external Consolidate compiler generated dependencies of target core-minifi [ 8%] Building CXX object libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o [ 8%] Linking CXX shared library ../bin/libcore-minifi.so [ 78%] Built target core-minifi Consolidate compiler generated dependencies of target minifi-civet-extensions Consolidate compiler generated dependencies of target minifi-openwsman Consolidate compiler generated dependencies of target minifi-http-curl Consolidate compiler generated dependencies of target minifiexe [ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so [ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so Consolidate compiler generated dependencies of target minifi-standard-processors [ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o [ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o [ 79%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o [ 80%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o [ 81%] Building CXX object minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o [ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so [ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so [ 84%] Built target minifi-civet-extensions [ 86%] Built target minifi-http-curl [ 87%] Built target minifi-openwsman [100%] Built target minifi-standard-processors [100%] Linking CXX executable ../bin/minifi [100%] Built target minifiexe
NEXT: I will verify I can build these MiNiFi CPP libraries and binary executables using USE_CMAKE_FETCH_CONTENT option set to ON and USE_CONAN_PACKAGER option set to OFF for the for the follow up commit.
Tracking Prebuilt Binary Conan Packages Available on Conancenter:
- 'abseil/20230125.3',
- 'argparse/3.0',
- 'asio/1.30.2'
Tracking Conan Packages Not Available on Conancenter, so we need to use an alternative hosting service to manage building package sources into prebuilt binary conan packages. Here are some of the free conan hosting services for open-source C/C++ projects:
- Conan Center Index (CCI): official, free, centralized repo for conan packages maintained by conan team
- Bincrafters Bintray Repo: community-driven repo that hosts large number of pre-built conan packages for open-source libraries for free
- Artifactory by JFrog: typically used as a commercial hosting service, but also provides a free tier for open-source projects.
- Self-Hosting: While there are multiple options, I will focus on GitHub Packages as its a free package hosting service, which can be used to host our own conan package repos, allowing us to store and distribute our pre-built conan packages without having to use 3rd party services like Bincrafters or pay for commercial solution.
- Advantage with GitHub Packages: its free, integrated with our existing GitHub workflow, allows us full control hosting our conan packages without relying on 3rd party services, useful if we need to have specific requirements or need flexibility in managing our package hosting.
Tracking Non Prebuilt Binary Conan Packages, which later I will create a GitHub Packages repo for MiNiFi CPP prebuilt conan packages (also account for Linux, Windows, Mac, etc):
- bustache: https://github.com/jamboree/bustache
FUTURE IMPROVEMENTS: Some more build infra we can improve in MiNiFi CPP after I finish my work for MINFICPP-2346: While I am working on migrating MiNiFi CPP build infrastructure toward leveraging conan packager, for a later PR we could do the following:
- Put the external libraries that MiNiFi CPP Core depends on into its own BundledMiNiFiCoreExtLibs.cmake (also feel free to suggest some other names), we can run this CMake code before adding add_subdirectory(libminifi)
- External libs that libminifi needs that are in MiNiFi CPP's root CMakeLists.txt (refactor for better organization):
yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads, gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio, magic_enum, RangeV3, Asio, MagicEnum
- We can also clean up the thirdparty folder as we migrate to conan packager for managing these depedencies, so in MiNiFi C+, we mainly keep it MiNiFi C+ code and its extension processors, etc.
- External libs that libminifi needs that are in MiNiFi CPP's root CMakeLists.txt (refactor for better organization):
- I tried building MiNiFi CPP Core just with just minifi-standard-processors and the other extensions disabled, but it seems even some of MiNiFi CPP Core depends on some of the extensions to be built with it, so it doesn't fail building, such as Openwsman. If we don't have ENABLE_OPENWSMAN, then we get this error when building MiNiFi CPP Core, so we should look having a build config to build minimal MiNiFi CPP Core:
libminifi/src/controllers/SSLContextService.cpp:520:80: error: ‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls
- I noticed that when building MiNiFi Core, MainExe, Standard-Processors and their external lib dependencies and then building the appropriate GTEST, I get a header error coming from rocksdb, but I haven't enabled rocksdb, so I think at a later point we should shift that test out, so when building these 3 core building blocks of MiNiFi, they dont rely on rocksdb unless MiNiFi Core and MainExe have a dependency for storing into that DB. Here is an example of the header error (I will ENABLE_ROCKSDB to ON):
[ 99%] Built target ControllerTests In file included from /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/DatabaseContentRepository.h:30, from /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/test/flow-tests/SessionTests.cpp:24: /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/database/RocksDatabase.h:26:10: fatal error: rocksdb/db.h: No such file or directory 26 | #include "rocksdb/db.h"
