Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.12.0
-
None
-
4688, 4687
Description
Hello everyone,
This thread follows some other threads dealing with concerns about online libraries (THRIFT-4688, THRIFT-4687), as well as emails being sent to the user mailing list about the Docker container on DockerHub.
If I understand the situation correctly, at the moment, with each new version of the software, you must prepare and update each library on each website independently.
In addition, each of those libraries is managed by different people, with whom contact is not always easy.
Also, I would like to propose here a system that would offer the following advantages:
- No more worrying about who is maintaining each platform/CDN/Repositories...
- No more manual uploading on each platform at each release
- Allow anyone to develop an extension to put a Thrift package online on a particular site (For example, a version for Yocto as I requested in a recent email), and be sure this package will be updated at each new release.
I will try to explain my idea in the following.
Vocabulary
- CDN : Content delivery network
- Manager : An algorithm to upload content to a CDN/online repository
- Orchestrator : The algorithm that orchestrates the execution of managers, provides credentials, manages manager feedback, and generates the final report
- Package : The library/component we want to upload on the CDN
In the following, all online resources providers/repositories (like Pypi, NPMJs, DockerHub ...) will be called "CDN"
General principle
The idea would be to develop a single interface for module deployment, with an orchestrator to automatically configure and perform these deployments.
The organization would be like this :
- A Keepass database will store all passwords/identifiers required for each platform (NPMJS, Pypi, DockerHub, ...) :
- Each module will specify what credentials it needs from this database.
- This database could be committed directly on the repo, to be sure each committers have the most recent version.
- The master password of this database will only be known by committers (or project chair).
I think KeePass is quite perfect for this, Keepass databases are very lightweight, highly secure, and almost impossible to brute-force.
- For each CDN, a manager is in charge of bundling (create the package to upload from Thrift current release sources), uploading and checking the package on the CDN :
- First, the orchestrator load required IDs from the KeePass database
- The orchestrator creates a temporary directory where the manager will be able to bundle everything it needs.
- The manager receives credentials, a path to a temporary directory and a path to Thrift sources
- The manager checks online if the current version of the package already exists online. If not, it will upload it
- The manager prepares the bundled package, upload it and check if everything went well.
All these steps could be executed in a dedicated Docker/Vagrant environment containing all needed libraries to bundle the package (So that the person/commiter executing the script does not have to download 500 dependencies).
The algorithm diagram is provided in attached (it's a draft)
The directory structure of this solution could be something like this (I chose Python arbitrarily here):
root
└── deployment/
├── access.kdbx
├── deploy.py
├── libraries # Some managers
│ ├── javascript
│ │ ├── npm.py
│ │ ├── bower.py
│ │ └── a_random_cdn.py
│ ├── nodejs
│ │ └── npm.py
│ └── python
│ └── pypi.js
│ └── ...
└── miscellaneous # Some other managers
├── docker
│ └── docker.py
└── yocto
└── yocto.py
└── ...
I am not a regular contributor on open source projects. So I don't know if I'm making my proposal in the best way. But I really like this project and I would like to help improve it
I had this idea based on what had been put in place in a project I was recently leading. It is not perfect but does the job properly, and it's a solution I find quite elegant because we don't have to worry about the credentials of each site.
If you find this idea interesting, I could propose a prototype in a personal repository with some examples of managers for Npm, Pypi, and DockerHub (for example) in several weeks.
