[AIRAVATA-3346] Implement remote FS abstraction of user storage

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Django Portal
• Assignee: Marcus Christie
• Reporter: Marcus Christie
• Created: 23/Jun/20 21:41
• Updated: 20/Jan/21 22:01
• Resolved: 20/Jan/21 22:01

Description

Running the Airavata Django portal locally, a developer experiences the portal just like a user of a deployed portal, except when it comes to gateway user storage, which is only available on the deployed portal. So for example, on seagrid.org, I can log in, view my experiments, and download my output files. If I run the Airavata Django portal locally and have configured it to connect to the same Keycloak and API server, I can also login and view my experiments, but I can't download my output files (or upload files, or clone experiments, etc.).

However, there exists a REST API to get and put files into gateway user storage so in theory it should be possible to create a filesystem abstraction, locally, that uses the REST API underneath to make it appear that the user's files are all available.

The benefit of this is that it opens up local development to be able to test a broader range of use cases. I know when I'm developing the Django portal code, there are some things that are just very hard to test because I don't actually have access to the files locally. Also, for extension developers, this will be helpful too. For example, currently creating a custom output view provider requires using a fake output file to test locally because the data files aren't available.

This seems like a good option: PyFilesystem2 https://pypi.org/project/fs/

One thing, we need token based authentication in the REST API, but I've already started work on that in a separate branch.

Another thing, it could be good to put in this work now since it might ease transitioning to Airavata MFT for the data storage backend. MFT would just be another PyFilesystem2 filesystem implementation.

TODO

• [x] Need support for Bearer token authentication so that the REST API can be used directly
• [x] Update API view function so that they are APIViews and use the Django REST Framework authentication/permission support
• [x] Implement remote API support in ADP SDK user_storage functions
  • to get the user storage view working remotely:
    • [x] user_storage.dir_exists
    • [x] user_storage.listdir
    • [x] user_storage.get_file
    • [x] user_storage.delete_dir
    • [x] user_storage.create_user_dir
    • [x] user_storage.save
  • to get input file upload working remotely:
    • [x] user_storage.save_input_file
  • to get experiment data directory creation and launch working remotely:
    • [x] not on user_storage but ExperimentViewSet.launch
  • to get experiment cloning working remotely:
    • [x] not on user_storage but ExperimentViewSet.clone
• additional user_storage functions
  • [x] update_file_content
• [ ] support tus uploads to the remote data store?
• [x] Bug: infinite loop error when token is invalid
• [x] factor out common code for calling the remote API in SDK
• [ ] verify that this works for output view provider
• [ ] update tutorial
  • we should be able to open the output file directly, but if output view provider needs to access other files in the experiment data directory then there should be advice on how to use the user_storage module to access those files