diff --git a/CHANGELOG.md b/CHANGELOG.md index 2620575a..e246ea52 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,16 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](http://semver.org/). +## [2.0.10] + +### Added + +- Added params support to FS URLs + +### Fixed + +- Many fixes to FTPFS contributed by Martin Larralde. + ## [2.0.9] ### Changed diff --git a/docs/source/concepts.rst b/docs/source/concepts.rst index 2dfefbff..654913a8 100644 --- a/docs/source/concepts.rst +++ b/docs/source/concepts.rst @@ -54,10 +54,10 @@ For example:: '/home/will/test.txt' Not all filesystems map to a system path (for example, files in a -:meth:`~fs.memoryfs.MemoryFS` will only ever exists in memory). +:class:`~fs.memoryfs.MemoryFS` will only ever exists in memory). If you call ``getsyspath`` on a filesystem which doesn't map to a system -path, it will raise a :meth:`~fs.errors.NoSysPath` exception. If you +path, it will raise a :class:`~fs.errors.NoSysPath` exception. If you prefer a *look before you leap* approach, you can check if a resource has a system path by calling :meth:`~fs.base.FS.hassyspath` diff --git a/docs/source/external.rst b/docs/source/external.rst index 61b9d19f..167764d6 100644 --- a/docs/source/external.rst +++ b/docs/source/external.rst @@ -1,26 +1,8 @@ External Filesystems ==================== -The following filesystems work with the FS interface, but are not built-in the the fs module. Please see the documentation for how to install them. +See the following wiki page for a list of filesystems not in the core library, and community contributed filesystems. -SSH |ci ssh| ------------- -`fs.sshfs `_ implements a Pyfilesystem2 filesystem running over the `SSH `_ protocol, using `paramiko `_. +https://www.pyfilesystem.org/page/index-of-filesystems/ -.. |ci ssh| image:: https://img.shields.io/travis/althonos/fs.sshfs/master.svg - :target: https://travis-ci.org/althonos/fs.sshfs/branches - -SMB |ci smb| ------------- -`fs.smbfs `_ implements a Pyfilesystem2 filesystem running over the `SMB `_ protocol, using `pysmb `_. - -.. |ci smb| image:: https://img.shields.io/travis/althonos/fs.smbfs/master.svg - :target: https://travis-ci.org/althonos/fs.smbfs/branches - - -WebDAV |ci webdav| ------------------- -`fs.webdavfs `_ implements Pyfilesystem2 over the `WebDAV `_ protocol. - -.. |ci webdav| image:: https://img.shields.io/travis/PyFilesystem/webdavfs/master.svg - :target: https://travis-ci.org/PyFilesystem/webdavfs/branches +If you have developed a filesystem that you would like added to the above page, please let us know by opening a `Github issue `_. \ No newline at end of file diff --git a/docs/source/openers.rst b/docs/source/openers.rst index 72d74ff3..1fa643e5 100644 --- a/docs/source/openers.rst +++ b/docs/source/openers.rst @@ -3,18 +3,18 @@ FS URLs ======= -PyFilesystem can open filesystems via a FS URL, which are similar to the URLs you might enter in to a browser. +PyFilesystem can open a filesystem via an *FS URL*, which is similar to a URL you might enter in to a browser. FS URLs are useful if you want to specify a filesystem dynamically, such as in a conf file or from the command line. -Using FS URLs can be useful if you want to be able to specify a filesystem dynamically, in a conf file (for instance). +Format +------ -FS URLs are parsed in to the following format:: - - ://:@ +FS URLs are formatted in the following way:: + ://:@ The components are as follows: -* ```` Identifies the type of filesystem to create. e.g. ``osfs``, ``ftp``. +* ```` Identifies the type of filesystem to create. e.g. ``osfs``, ``ftp``. * ```` Optional username. * ```` Optional password. * ```` A *resource*, which may be a domain, path, or both. @@ -25,13 +25,34 @@ Here are a few examples:: osfs://c://system32 ftp://ftp.example.org/pub mem:// + ftp://will:daffodil@ftp.example.org/private + -If ```` is not specified then it is assumed to be an :class:`~fs.osfs.OSFS`. The following FS URLs are equivalent:: +If ```` is not specified then it is assumed to be an :class:`~fs.osfs.OSFS`, i.e. the following FS URLs are equivalent:: osfs://~/projects ~/projects -To open a filesysem with a FS URL, you can use :meth:`~fs.opener.Registry.open_fs`, which may be imported and used as follows:: +.. note:: + The `username` and `passwords` fields may not contain a colon (``:``) or an ``@`` symbol. If you need these symbols they may be `percent encoded `_. + + +URL Parameters +-------------- + +FS URLs may also be appended with a ``?`` symbol followed by a url-encoded query string. For example:: + + myprotocol://example.org?key1=value1&key2 + +The query string would be decoded as ``{"key1": "value1", "key2": ""}``. + +Query strings are used to provide additional filesystem-specific information used when opening. See the filesystem documentation for information on what query string parameters are supported. + + +Opening FS URLS +--------------- + +To open a filesysem with a FS URL, you can use :meth:`~fs.opener.registry.Registry.open_fs`, which may be imported and used as follows:: from fs import open_fs projects_fs = open_fs('osfs://~/projects') diff --git a/docs/source/reference/opener.rst b/docs/source/reference/opener.rst index 9d1883c5..6c7c7de4 100644 --- a/docs/source/reference/opener.rst +++ b/docs/source/reference/opener.rst @@ -6,6 +6,9 @@ Open filesystems from a URL. .. automodule:: fs.opener.base :members: +.. automodule:: fs.opener.parse + :members: + .. automodule:: fs.opener.registry :members: diff --git a/fs/_version.py b/fs/_version.py index e5d18b20..44deabc3 100644 --- a/fs/_version.py +++ b/fs/_version.py @@ -1 +1,2 @@ -__version__ = "2.0.9" +"""Version, used in module and setup.py.""" +__version__ = "2.0.10a2" diff --git a/fs/base.py b/fs/base.py index afe8f148..55736718 100644 --- a/fs/base.py +++ b/fs/base.py @@ -515,7 +515,7 @@ def getmeta(self, namespace="standard"): specified with the `namespace` parameter. The default namespace, ``"standard"``, contains common information regarding the filesystem's capabilities. Some filesystems may provide other - namespaces, which expose less common, or implementation specific + namespaces which expose less common or implementation specific information. If a requested namespace is not supported by a filesystem, then an empty dictionary will be returned. @@ -524,7 +524,7 @@ def getmeta(self, namespace="standard"): =================== ============================================ key Description ------------------- -------------------------------------------- - case_insensitive True if this filesystem is case sensitive. + case_insensitive True if this filesystem is case insensitive. invalid_path_chars A string containing the characters that may may not be used on this filesystem. max_path_length Maximum number of characters permitted in a @@ -548,7 +548,7 @@ def getmeta(self, namespace="standard"): """ if namespace == 'standard': - meta = self._meta + meta = self._meta.copy() else: meta = {} return meta diff --git a/fs/errors.py b/fs/errors.py index 389f488f..a1b0670d 100644 --- a/fs/errors.py +++ b/fs/errors.py @@ -31,6 +31,7 @@ 'InvalidCharsInPath', 'InvalidPath', 'MissingInfoNamespace', + 'NoSysPath', 'NoURL', 'OperationFailed', 'OperationTimeout', diff --git a/fs/info.py b/fs/info.py index 533aabe9..42149d7f 100644 --- a/fs/info.py +++ b/fs/info.py @@ -263,7 +263,7 @@ def permissions(self): Requires the ``"access"`` namespace. :rtype: :class:`fs.permissions.Permissions` - :raises ~fs.errors.MissingInfoNamespace: if the 'ACCESS' + :raises ~fs.errors.MissingInfoNamespace: if the 'access' namespace is not in the Info. """ diff --git a/fs/opener/__init__.py b/fs/opener/__init__.py index 086500a3..749e42cf 100644 --- a/fs/opener/__init__.py +++ b/fs/opener/__init__.py @@ -12,13 +12,13 @@ # Import objects into fs.opener namespace from .base import Opener +from .parse import parse_fs_url as parse from .registry import registry # Alias functions defined as Registry methods open_fs = registry.open_fs open = registry.open manage_fs = registry.manage_fs -parse = registry.parse # __all__ with aliases and classes __all__ = [ diff --git a/fs/opener/base.py b/fs/opener/base.py index d61f4244..7ff5afcc 100644 --- a/fs/opener/base.py +++ b/fs/opener/base.py @@ -32,7 +32,7 @@ def open_fs(self, fs_url, parse_result, writeable, create, cwd): :param str fs_url: A filesystem URL :param parse_result: A parsed filesystem URL. - :type parse_result: :class:`ParseResult` + :type parse_result: :class:`~fs.opener.parse.ParseResult` :param bool writeable: True if the filesystem must be writeable. :param bool create: True if the filesystem should be created if it does not exist. diff --git a/fs/opener/parse.py b/fs/opener/parse.py new file mode 100644 index 00000000..e39c48fe --- /dev/null +++ b/fs/opener/parse.py @@ -0,0 +1,105 @@ +""" +fs.opener.parse +=============== + +Parses FS URLs in to their constituent parts. + +""" + +from __future__ import absolute_import +from __future__ import print_function +from __future__ import unicode_literals + +import collections +import re + +from six.moves.urllib.parse import parse_qs, unquote + +from .errors import ParseError + + +_ParseResult = collections.namedtuple( + 'ParseResult', + [ + 'protocol', + 'username', + 'password', + 'resource', + 'params', + 'path' + ] +) + +class ParseResult(_ParseResult): + """A named tuple containing fields of a parsed FS URL. + + * ``protocol`` The protocol part of the url, e.g. ``osfs`` or + ``ftp``. + * ``username`` A username, or ``None`` . + * ``password`` An password, or ``None``. + * ``resource`` A *resource*, typically a domain and path, e.g. + ``ftp.example.org/dir`` + * ``params`` An dictionary of parameters extracted from the query + string. + * ``path`` An optional path within the filesystem. + + """ + + +_RE_FS_URL = re.compile(r''' +^ +(.*?) +:\/\/ + +(?: +(?:(.*?)@(.*?)) +|(.*?) +) + +(?: +!(.*?)$ +)*$ +''', re.VERBOSE) + + +def parse_fs_url(fs_url): + """ + Parse a Filesystem URL and return a + :class:`~fs.opener.parse.ParseResult`, or raise + :class:`~fs.errors.ParseError` (subclass of ValueError) if the FS URL is + not value. + + :param str fs_url: A filesystem URL + :rtype: :class:`~fs.opener.parse.ParseResult` + + """ + + match = _RE_FS_URL.match(fs_url) + if match is None: + raise ParseError('{!r} is not a fs2 url'.format(fs_url)) + + fs_name, credentials, url1, url2, path = match.groups() + if credentials: + username, _, password = credentials.partition(':') + username = unquote(username) + password = unquote(password) + url = url1 + else: + username = None + password = None + url = url2 + url, has_qs, _params = url.partition('?') + resource = unquote(url) + if has_qs: + params = parse_qs(_params, keep_blank_values=True) + params = {k:v[0] for k, v in params.items()} + else: + params = {} + return ParseResult( + fs_name, + username, + password, + resource, + params, + path + ) diff --git a/fs/opener/registry.py b/fs/opener/registry.py index 2641649c..4bd91613 100644 --- a/fs/opener/registry.py +++ b/fs/opener/registry.py @@ -12,14 +12,13 @@ from __future__ import print_function from __future__ import unicode_literals -import re -import six import contextlib -import collections +import six import pkg_resources from .base import Opener -from .errors import ParseError, UnsupportedProtocol, EntryPointError +from .errors import UnsupportedProtocol, EntryPointError +from .parse import parse_fs_url class Registry(object): @@ -28,62 +27,6 @@ class Registry(object): """ - ParseResult = collections.namedtuple( - 'ParseResult', - [ - 'protocol', - 'username', - 'password', - 'resource', - 'path' - ] - ) - - _RE_FS_URL = re.compile(r''' - ^ - (.*?) - :\/\/ - - (?: - (?:(.*?)@(.*?)) - |(.*?) - ) - - (?: - !(.*?)$ - )*$ - ''', re.VERBOSE) - - @classmethod - def parse(cls, fs_url): - """ - Parse a Filesystem URL and return a :class:`ParseResult`, or - raise :class:`ParseError` (subclass of ValueError) if the FS URL - is not value. - - :param str fs_url: A filesystem URL - :rtype: :class:`ParseResult` - - """ - match = cls._RE_FS_URL.match(fs_url) - if match is None: - raise ParseError('{!r} is not a fs2 url'.format(fs_url)) - - fs_name, credentials, url1, url2, path = match.groups() - if credentials: - username, _, password = credentials.partition(':') - url = url1 - else: - username = None - password = None - url = url2 - return cls.ParseResult( - fs_name, - username, - password, - url, - path - ) def __init__(self, default_opener='osfs'): """ @@ -103,6 +46,7 @@ def __repr__(self): @property def protocols(self): + """A list of supported protocols.""" if self._protocols is None: self._protocols = [ entry_point.name @@ -117,7 +61,8 @@ def get_opener(self, protocol): :param str protocol: A filesystem protocol. :rtype: ``Opener``. - :raises `~fs.opener.errors.UnsupportedProtocol`: If no opener could be found. + :raises `~fs.opener.errors.UnsupportedProtocol`: If no opener + could be found. :raises `EntryPointLoadingError`: If the returned entry point is not an ``Opener`` subclass or could not be loaded successfully. @@ -186,7 +131,7 @@ def open(self, # URL may just be a path fs_url = "{}://{}".format(default_protocol, fs_url) - parse_result = self.parse(fs_url) + parse_result = parse_fs_url(fs_url) protocol = parse_result.protocol open_path = parse_result.path @@ -211,9 +156,7 @@ def open_fs(self, Open a filesystem object from a FS URL (ignoring the path component). - :param str fs_url: A filesystem URL - :param parse_result: A parsed filesystem URL. - :type parse_result: :class:`ParseResult` + :param str fs_url: A filesystem URL. :param bool writeable: True if the filesystem must be writeable. :param bool create: True if the filesystem should be created if it does not exist. @@ -244,10 +187,10 @@ def manage_fs(self, fs_url, create=False, writeable=True, cwd='.'): :param fs_url: A FS instance or a FS URL. :type fs_url: str or FS - :param bool create: If ``True``, then create the filesytem if it - doesn't already exist. - :param bool writeable: If ``True``, then the filesystem should be - writeable. + :param bool create: If ``True``, then create the filesystem if + it doesn't already exist. + :param bool writeable: If ``True``, then the filesystem should + be writeable. :param str cwd: The current working directory, if opening a :class:`~fs.osfs.OSFS`. diff --git a/fs/wrap.py b/fs/wrap.py index 126e1145..ab01ca21 100644 --- a/fs/wrap.py +++ b/fs/wrap.py @@ -1,3 +1,24 @@ +""" +fs.wrap +======= + +A collection of :class:`~fs.wrapfs.WrapFS` objects that modify the +behavior of another filesystem. + +Here's an example that opens a filesystem then makes it *read only*:: + + from fs import open_fs + from fs.wrap import read_only + + projects_fs = open_fs('~/projects') + read_only_projects_fs = read_only(projects_fs) + + # Will raise ResourceReadOnly exception + read_only_projects_fs.remove('__init__.py') + + +""" + from __future__ import print_function from __future__ import unicode_literals diff --git a/setup.py b/setup.py index a4f7090a..485ad179 100644 --- a/setup.py +++ b/setup.py @@ -26,7 +26,7 @@ "appdirs~=1.4.3", "pytz", "setuptools", - "six~=1.10.0", + "six~=1.10", ] setup( diff --git a/tests/test_opener.py b/tests/test_opener.py index 5c77038e..87eb36b9 100644 --- a/tests/test_opener.py +++ b/tests/test_opener.py @@ -1,17 +1,19 @@ from __future__ import unicode_literals import os -import six import mock import tempfile import unittest import pkg_resources +import six + from fs import open_fs, opener from fs.osfs import OSFS from fs.opener import registry, errors from fs.memoryfs import MemoryFS from fs.appfs import UserDataFS +from fs.opener.parse import ParseResult class TestParse(unittest.TestCase): @@ -26,47 +28,118 @@ def test_parse_not_url(self): def test_parse_simple(self): parsed = opener.parse('osfs://foo/bar') - expected = opener.registry.ParseResult( + expected = ParseResult( 'osfs', None, None, 'foo/bar', + {}, None ) self.assertEqual(expected, parsed) def test_parse_credentials(self): parsed = opener.parse('ftp://user:pass@ftp.example.org') - expected = opener.registry.ParseResult( + expected = ParseResult( 'ftp', 'user', 'pass', 'ftp.example.org', + {}, None ) self.assertEqual(expected, parsed) parsed = opener.parse('ftp://user@ftp.example.org') - expected = opener.registry.ParseResult( + expected = ParseResult( 'ftp', 'user', '', 'ftp.example.org', + {}, None ) self.assertEqual(expected, parsed) def test_parse_path(self): parsed = opener.parse('osfs://foo/bar!example.txt') - expected = opener.registry.ParseResult( + expected = ParseResult( 'osfs', None, None, 'foo/bar', + {}, 'example.txt' ) self.assertEqual(expected, parsed) + def test_parse_params(self): + parsed = opener.parse('ftp://ftp.example.org?proxy=ftp.proxy.org') + expected = ParseResult( + 'ftp', + None, + None, + 'ftp.example.org', + { + 'proxy':'ftp.proxy.org' + }, + None + ) + self.assertEqual(expected, parsed) + + def test_parse_params_multiple(self): + parsed = opener.parse('ftp://ftp.example.org?foo&bar=1') + expected = ParseResult( + 'ftp', + None, + None, + 'ftp.example.org', + { + 'foo':'', + 'bar':'1' + }, + None + ) + self.assertEqual(expected, parsed) + + def test_parse_user_password_proxy(self): + parsed = opener.parse('ftp://user:password@ftp.example.org?proxy=ftp.proxy.org') + expected = ParseResult( + 'ftp', + 'user', + 'password', + 'ftp.example.org', + { + 'proxy': 'ftp.proxy.org' + }, + None + ) + self.assertEqual(expected, parsed) + + def test_parse_user_password_decode(self): + parsed = opener.parse('ftp://user%40large:password@ftp.example.org') + expected = ParseResult( + 'ftp', + 'user@large', + 'password', + 'ftp.example.org', + {}, + None + ) + self.assertEqual(expected, parsed) + + def test_parse_resource_decode(self): + parsed = opener.parse('ftp://user%40large:password@ftp.example.org/%7Econnolly') + expected = ParseResult( + 'ftp', + 'user@large', + 'password', + 'ftp.example.org/~connolly', + {}, + None + ) + self.assertEqual(expected, parsed) + class TestRegistry(unittest.TestCase):