Uploaded image for project: 'Libcloud'
  1. Libcloud
  2. LIBCLOUD-369

Support for just listing containers with prefix

    Details

    • Type: Wish
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0-beta3
    • Component/s: Storage
    • Labels:
      None

      Description

      Both S3 and Swift support only listing objects in a container that match a given prefix. With libcloud I can only list the whole container, which is rather inefficient for big (~50K objects) containers. Is someone already working on this? And if not, would it be worthwhile including in libcloud?

      1. ex_prefix.patch
        6 kB
        Stefan Friesel

        Activity

        Hide
        kami Tomaz Muraus added a comment -

        Afaik, no one is working on it at the moment.

        I do think it would be worthwhile to include it in Libcloud as an extension method.

        Show
        kami Tomaz Muraus added a comment - Afaik, no one is working on it at the moment. I do think it would be worthwhile to include it in Libcloud as an extension method.
        Hide
        sfriesel Stefan Friesel added a comment -

        Adding extension methods I end up with this: https://github.com/sfriesel/libcloud/commit/49d77c549c027c29cc89addae5734bd249a5a1c2
        Extension parameters look more natural imho: https://github.com/sfriesel/libcloud/commit/a26256e2983057d32fa41d1c14da4f963366801a
        I'm quite new to libcloud so I'm not sure what the preferred style is.

        Either way I still have to add unittests.

        Show
        sfriesel Stefan Friesel added a comment - Adding extension methods I end up with this: https://github.com/sfriesel/libcloud/commit/49d77c549c027c29cc89addae5734bd249a5a1c2 Extension parameters look more natural imho: https://github.com/sfriesel/libcloud/commit/a26256e2983057d32fa41d1c14da4f963366801a I'm quite new to libcloud so I'm not sure what the preferred style is. Either way I still have to add unittests.
        Hide
        kami Tomaz Muraus added a comment -

        Stefan Friesel In this case using an extension argument seems to work better.

        Please also make sure to add a Sphinx style docstring which documents what this argument does (e.g. https://github.com/apache/libcloud/blob/trunk/libcloud/compute/base.py#L612).

        Show
        kami Tomaz Muraus added a comment - Stefan Friesel In this case using an extension argument seems to work better. Please also make sure to add a Sphinx style docstring which documents what this argument does (e.g. https://github.com/apache/libcloud/blob/trunk/libcloud/compute/base.py#L612 ).
        Hide
        sfriesel Stefan Friesel added a comment -

        I added the docstrings, matching list_container methods, and unittests (I re-used the fixtures of the non-prefix test though - is that okay?).
        New branch with everything squashed in one commit: https://github.com/sfriesel/libcloud/compare/ex_prefix

        I successfully tested the parameter against S3 EU West and a local OpenStack setup.
        I couldn't get the docs to generate properly, but I have the same problem on trunk.

        Show
        sfriesel Stefan Friesel added a comment - I added the docstrings, matching list_container methods, and unittests (I re-used the fixtures of the non-prefix test though - is that okay?). New branch with everything squashed in one commit: https://github.com/sfriesel/libcloud/compare/ex_prefix I successfully tested the parameter against S3 EU West and a local OpenStack setup. I couldn't get the docs to generate properly, but I have the same problem on trunk.
        Hide
        kami Tomaz Muraus added a comment -

        The pull request looks good to me.

        Please attach a final patch to this ticket.

        Show
        kami Tomaz Muraus added a comment - The pull request looks good to me. Please attach a final patch to this ticket.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 8929aab4512bbe05e79a6d632bdd69e4fc28f736 in branch refs/heads/trunk from Stefan Friesel
        [ https://git-wip-us.apache.org/repos/asf?p=libcloud.git;h=8929aab ]

        LIBCLOUD-369 Add extension method for listing container objects with prefix

        Swift and S3 support a prefix parameter when listing a container
        to narrow down the result.

        Signed-off-by: Tomaz Muraus <tomaz@apache.org>

        Show
        jira-bot ASF subversion and git services added a comment - Commit 8929aab4512bbe05e79a6d632bdd69e4fc28f736 in branch refs/heads/trunk from Stefan Friesel [ https://git-wip-us.apache.org/repos/asf?p=libcloud.git;h=8929aab ] LIBCLOUD-369 Add extension method for listing container objects with prefix Swift and S3 support a prefix parameter when listing a container to narrow down the result. Signed-off-by: Tomaz Muraus <tomaz@apache.org>
        Hide
        kami Tomaz Muraus added a comment -

        I've merged the PR into trunk - thanks!

        Show
        kami Tomaz Muraus added a comment - I've merged the PR into trunk - thanks!

          People

          • Assignee:
            kami Tomaz Muraus
            Reporter:
            sfriesel Stefan Friesel
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development