{
    "file_diffs": [
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil; coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Helper utility to list things.  ATM list content of S3 bucket\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport sys\nimport time\nfrom os.path import exists, lexists, join as opj, abspath, isabs, getmtime\nfrom os.path import curdir, isfile, islink, isdir, dirname, basename, split, realpath\nfrom os.path import relpath\nfrom os import listdir, lstat, remove, makedirs\nimport json as js\nimport hashlib\n\nfrom six.moves.urllib.request import urlopen, Request\nfrom six.moves.urllib.error import HTTPError\n\nfrom ..cmdline.helpers import get_repo_instance\nfrom ..utils import auto_repr\nfrom .base import Interface\nfrom datalad.interface.base import build_doc\nfrom ..ui import ui\nfrom ..utils import safe_print\nfrom ..consts import METADATA_DIR\nfrom ..consts import METADATA_FILENAME\nfrom ..dochelpers import exc_str\nfrom ..support.param import Parameter\nfrom ..support import ansi_colors\nfrom ..support.constraints import EnsureStr, EnsureNone\nfrom ..distribution.dataset import Dataset\n\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.support.annexrepo import GitRepo\n\nimport string\nimport humanize\nfrom datalad.utils import is_interactive\n\nfrom logging import getLogger\nlgr = getLogger('datalad.api.ls')\n\n\n@build_doc\nclass Ls(Interface):\n    \"\"\"List summary information about URLs and dataset(s)\n\n    ATM only s3:// URLs and datasets are supported\n\n    Examples:\n\n      $ datalad ls s3://openfmri/tarballs/ds202  # to list S3 bucket\n      $ datalad ls                               # to list current dataset\n    \"\"\"\n    # XXX prevent common args from being added to the docstring\n    _no_eval_results = True\n\n    # TODO: during big RF refactor this one away since it must not be instance's\n    # attribute.  For now introduced to make `datalad ls` be relatively usable\n    # in terms of speed\n    _cached_subdatasets = {}\n\n    _params_ = dict(\n        loc=Parameter(\n            doc=\"URL or path to list, e.g. s3://...\",\n            metavar='PATH/URL',\n            nargs=\"*\",\n            constraints=EnsureStr() | EnsureNone(),\n        ),\n        recursive=Parameter(\n            args=(\"-r\", \"--recursive\"),\n            action=\"store_true\",\n            doc=\"recurse into subdirectories\",\n        ),\n        fast=Parameter(\n            args=(\"-F\", \"--fast\"),\n            action=\"store_true\",\n            doc=\"only perform fast operations.  Would be overridden by --all\",\n        ),\n        all_=Parameter(\n            args=(\"-a\", \"--all\"),\n            dest='all_',\n            action=\"store_true\",\n            doc=\"list all (versions of) entries, not e.g. only latest entries \"\n                \"in case of S3\",\n        ),\n        long_=Parameter(\n            args=(\"-L\", \"--long\"),\n            dest='long_',\n            action=\"store_true\",\n            doc=\"list more information on entries (e.g. acl, urls in s3, annex \"\n                \"sizes etc)\",\n        ),\n        config_file=Parameter(\n            doc=\"\"\"path to config file which could help the 'ls'.  E.g. for s3://\n            URLs could be some ~/.s3cfg file which would provide credentials\"\"\",\n            constraints=EnsureStr() | EnsureNone()\n        ),\n        list_content=Parameter(\n            choices=(None, 'first10', 'md5', 'full'),\n            doc=\"\"\"list also the content or only first 10 bytes (first10), or md5\n            checksum of an entry.  Might require expensive transfer and dump\n            binary output to your screen.  Do not enable unless you know what you\n            are after\"\"\",\n            default=None\n        ),\n        json=Parameter(\n            choices=('file', 'display', 'delete'),\n            doc=\"\"\"metadata json of dataset for creating web user interface.\n            display: prints jsons to stdout or\n            file: writes each subdir metadata to json file in subdir of dataset or\n            delete: deletes all metadata json files in dataset\"\"\",\n        ),\n    )\n\n    @staticmethod\n    def __call__(loc, recursive=False, fast=False, all_=False, long_=False,\n                 config_file=None, list_content=False, json=None):\n        if isinstance(loc, list) and not len(loc):\n            # nothing given, CWD assumed -- just like regular ls\n            loc = '.'\n\n        kw = dict(fast=fast, recursive=recursive, all_=all_, long_=long_)\n        if isinstance(loc, list):\n            return [Ls.__call__(loc_, config_file=config_file,\n                                list_content=list_content, json=json, **kw)\n                    for loc_ in loc]\n\n        # TODO: do some clever handling of kwargs as to remember what were defaults\n        # and what any particular implementation actually needs, and then issuing\n        # warning if some custom value/option was specified which doesn't apply to the\n        # given url\n\n        # rename to not angry Python gods who took all_ good words\n        kw['long_'] = kw.pop('long_')\n\n        loc_type = \"unknown\"\n        if loc.startswith('s3://'):\n            return _ls_s3(loc, config_file=config_file, list_content=list_content,\n                          **kw)\n        elif lexists(loc):\n            if isdir(loc):\n                ds = Dataset(loc)\n                if ds.is_installed():\n                    return _ls_json(loc, json=json, **kw) if json else _ls_dataset(loc, **kw)\n                    loc_type = False\n                else:\n                    loc_type = \"dir\"  # we know that so far for sure\n                    # it might have been an uninstalled dataset within super-dataset\n                    superds = ds.get_superdataset()\n                    if superds:\n                        try:\n                            subdatasets = Ls._cached_subdatasets[superds.path]\n                        except KeyError:\n                            subdatasets = Ls._cached_subdatasets[superds.path] \\\n                                = superds.subdatasets(result_xfm='relpaths')\n                        if relpath(ds.path, superds.path) in subdatasets:\n                            loc_type = \"not installed\"\n            else:\n                loc_type = \"file\"\n                # could list properties -- under annex or git, either clean/dirty\n                # etc\n                # repo = get_repo_instance(dirname(loc))\n\n        if loc_type:\n            #raise ValueError(\"ATM supporting only s3:// URLs and paths to local datasets\")\n            # TODO: unify all_ the output here -- _ls functions should just return something\n            # to be displayed\n            ui.message(\n                \"{}  {}\".format(\n                    ansi_colors.color_word(loc, ansi_colors.DATASET),\n                    ansi_colors.color_word(\n                        loc_type,\n                        ansi_colors.RED\n                        if loc_type in {'unknown', 'not installed'}\n                        else ansi_colors.BLUE)\n                )\n            )\n\n\n#\n# Dataset listing\n#\n\n@auto_repr\nclass AbsentRepoModel(object):\n    \"\"\"Just a base for those where repo wasn't installed yet\"\"\"\n\n    def __init__(self, path):\n        self.path = path\n        self.repo = None\n\n    @property\n    def type(self):\n        return \"N/A\"\n\n\n@auto_repr\nclass GitModel(object):\n    \"\"\"A base class for models which have some .repo available\"\"\"\n\n    __slots__ = ['_branch', 'repo', '_path']\n\n    def __init__(self, repo):\n        self.repo = repo\n        # lazy evaluation variables\n        self._branch = None\n        self._path = None\n\n    @property\n    def path(self):\n        return self.repo.path if self._path is None else self._path\n\n    @path.setter\n    def path(self, v):\n        self._path = v\n\n    @property\n    def branch(self):\n        if self._branch is None:\n            try:\n                self._branch = self.repo.get_active_branch()\n            except:  # MIH: InvalidGitRepositoryError?\n                return None\n        return self._branch\n\n    @property\n    def clean(self):\n        return not self.repo.dirty\n\n    @property\n    def describe(self):\n        return self.repo.describe(tags=True)\n\n    @property\n    def date(self):\n        \"\"\"Date of the last commit\n        \"\"\"\n        return self.repo.get_committed_date()\n\n    @property\n    def count_objects(self):\n        return self.repo.count_objects\n\n    @property\n    def git_local_size(self):\n        count_objects = self.count_objects\n        return count_objects['size'] if count_objects else None\n\n    @property\n    def type(self):\n        return {False: 'git', True: 'annex'}[isinstance(self.repo, AnnexRepo)]\n\n\n@auto_repr\nclass AnnexModel(GitModel):\n\n    __slots__ = ['_info'] + GitModel.__slots__\n\n    def __init__(self, *args, **kwargs):\n        super(AnnexModel, self).__init__(*args, **kwargs)\n        self._info = None\n\n    @property\n    def info(self):\n        if self._info is None and self.type == 'annex':\n            self._info = self.repo.repo_info()\n        return self._info\n\n    @property\n    def annex_worktree_size(self):\n        info = self.info\n        return info['size of annexed files in working tree'] if info else 0.0\n\n    @property\n    def annex_local_size(self):\n        info = self.info\n        return info['local annex size'] if info else 0.0\n\n\n@auto_repr\nclass FsModel(AnnexModel):\n\n    __slots__ = AnnexModel.__slots__\n\n    def __init__(self, path, *args, **kwargs):\n        super(FsModel, self).__init__(*args, **kwargs)\n        self._path = path\n\n    @property\n    def path(self):\n        return self._path\n\n    @property\n    def symlink(self):\n        \"\"\"if symlink returns path the symlink points to else returns None\"\"\"\n        if islink(self._path):                    # if symlink\n            target_path = realpath(self._path)    # find link target\n            # convert to absolute path if not\n            return target_path if exists(target_path) else None\n        return None\n\n    @property\n    def date(self):\n        \"\"\"Date of last modification\"\"\"\n        if self.type_ is not ['git', 'annex']:\n            return lstat(self._path).st_mtime\n        else:\n            super(self.__class__, self).date\n\n    @property\n    def size(self):\n        \"\"\"Size of the node computed based on its type\"\"\"\n        type_ = self.type_\n        sizes = {'total': 0.0, 'ondisk': 0.0, 'git': 0.0, 'annex': 0.0, 'annex_worktree': 0.0}\n\n        if type_ in ['file', 'link', 'link-broken']:\n            # if node is under annex, ask annex for node size, ondisk_size\n            if isinstance(self.repo, AnnexRepo) and self.repo.is_under_annex(self._path):\n                size = self.repo.info(self._path, batch=True)['size']\n                ondisk_size = size \\\n                    if self.repo.file_has_content(self._path) \\\n                    else 0\n            # else ask fs for node size (= ondisk_size)\n            else:\n                size = ondisk_size = 0 \\\n                    if type_ == 'link-broken' \\\n                    else lstat(self.symlink or self._path).st_size\n\n            sizes.update({'total': size, 'ondisk': ondisk_size})\n\n        if self.repo.path == self._path:\n            sizes.update({'git': self.git_local_size,\n                          'annex': self.annex_local_size,\n                          'annex_worktree': self.annex_worktree_size})\n        return sizes\n\n    @property\n    def type_(self):\n        \"\"\"outputs the node type\n\n        Types: link, link-broken, file, dir, annex-repo, git-repo\"\"\"\n        if islink(self.path):\n            return 'link' if self.symlink else 'link-broken'\n        elif isfile(self.path):\n            return 'file'\n        elif exists(opj(self.path, \".git\", \"annex\")):\n            return 'annex'\n        elif exists(opj(self.path, \".git\")):\n            return 'git'\n        elif isdir(self.path):\n            return 'dir'\n        else:\n            return None\n\n\nclass LsFormatter(string.Formatter):\n    # condition by interactive\n    if is_interactive():\n        BLUE = ansi_colors.COLOR_SEQ % ansi_colors.BLUE\n        RED = ansi_colors.COLOR_SEQ % ansi_colors.RED\n        GREEN = ansi_colors.COLOR_SEQ % ansi_colors.GREEN\n        RESET = ansi_colors.RESET_SEQ\n        DATASET = ansi_colors.COLOR_SEQ % ansi_colors.UNDERLINE\n    else:\n        BLUE = RED = GREEN = RESET = DATASET = u\"\"\n\n    # TODO: we might want to just ignore and force utf8 while explicitly .encode()'ing output!\n    # unicode versions which look better but which blow during tests etc\n    # Those might be reset by the constructor\n    OK = u\"✓\"\n    NOK = u\"✗\"\n    NONE = u\"✗\"\n\n    def __init__(self, *args, **kwargs):\n        super(LsFormatter, self).__init__(*args, **kwargs)\n        for setting_encoding in (sys.getdefaultencoding(),\n                                 sys.stdout.encoding):\n            try:\n                u\"✓\".encode(setting_encoding)\n            except UnicodeEncodeError:\n                lgr.debug(\"encoding %s found to not support unicode, resetting to safe alternatives\", setting_encoding)\n                self.OK = 'OK'   # u\"✓\"\n                self.NOK = 'X'  # u\"✗\"\n                self.NONE = '-'  # u\"✗\"\n                break\n\n\n    def convert_field(self, value, conversion):\n        #print(\"%r->%r\" % (value, conversion))\n        if conversion == 'D':  # Date\n            if value is not None:\n                return time.strftime(u\"%Y-%m-%d/%H:%M:%S\", time.localtime(value))\n            else:\n                return u'-'\n        elif conversion == 'S':  # Human size\n            #return value\n            if value is not None:\n                return humanize.naturalsize(value)\n            else:\n                return u'-'\n        elif conversion == 'X':  # colored bool\n            chr, col = (self.OK, self.GREEN) if value else (self.NOK, self.RED)\n            return u\"%s%s%s\" % (col, chr, self.RESET)\n        elif conversion == 'N':  # colored Red - if None\n            if value is None:\n                # return \"%s✖%s\" % (self.RED, self.RESET)\n                return u\"%s%s%s\" % (self.RED, self.NONE, self.RESET)\n            return value\n        elif conversion in {'B', 'R', 'U'}:\n            return u\"%s%s%s\" % ({'B': self.BLUE, 'R': self.RED, 'U': self.DATASET}[conversion], value, self.RESET)\n\n        return super(LsFormatter, self).convert_field(value, conversion)\n\n    def format_field(self, value, format_spec):\n        # TODO: move all the \"coloring\" into formatting, so we could correctly indent\n        # given the format and only then color it up\n        # print \"> %r, %r\" % (value, format_spec)\n        return super(LsFormatter, self).format_field(value, format_spec)\n\n\ndef format_ds_model(formatter, ds_model, format_str, format_exc):\n    try:\n        #print(\"WORKING ON %s\" % ds_model.path)\n        if not exists(ds_model.path) or not ds_model.repo:\n            return formatter.format(format_exc, ds=ds_model, msg=u\"not installed\")\n        ds_formatted = formatter.format(format_str, ds=ds_model)\n        #print(\"FINISHED ON %s\" % ds_model.path)\n        return ds_formatted\n    except Exception as exc:\n        return formatter.format(format_exc, ds=ds_model, msg=exc_str(exc))\n\n# from joblib import Parallel, delayed\n\n\ndef _ls_dataset(loc, fast=False, recursive=False, all_=False, long_=False):\n    isabs_loc = isabs(loc)\n    topdir = '' if isabs_loc else abspath(curdir)\n\n    topds = Dataset(loc)\n    dss = [topds] + (\n        [Dataset(opj(loc, sm))\n         for sm in topds.subdatasets(recursive=recursive, result_xfm='relpaths')]\n        if recursive else [])\n\n    dsms = []\n    for ds in dss:\n        if not ds.is_installed():\n            dsm = AbsentRepoModel(ds.path)\n        elif isinstance(ds.repo, AnnexRepo):\n            dsm = AnnexModel(ds.repo)\n        elif isinstance(ds.repo, GitRepo):\n            dsm = GitModel(ds.repo)\n        else:\n            raise RuntimeError(\"Got some dataset which don't know how to handle %s\"\n                               % ds)\n        dsms.append(dsm)\n\n    # adjust path strings\n    for ds_model in dsms:\n        #path = ds_model.path[len(topdir) + 1 if topdir else 0:]\n        path = relpath(ds_model.path, topdir) if topdir else ds_model.path\n        if not path:\n            path = '.'\n        ds_model.path = path\n    dsms = sorted(dsms, key=lambda m: m.path)\n\n    maxpath = max(len(ds_model.path) for ds_model in dsms)\n    path_fmt = u\"{ds.path!U:<%d}\" % (maxpath + (11 if is_interactive() else 0))  # + to accommodate ansi codes\n    pathtype_fmt = path_fmt + u\"  [{ds.type}]\"\n    full_fmt = pathtype_fmt + u\"  {ds.branch!N}  {ds.describe!N} {ds.date!D}\"\n    if (not fast) or long_:\n        full_fmt += u\"  {ds.clean!X}\"\n\n    fmts = {\n        AbsentRepoModel: pathtype_fmt,\n        GitModel: full_fmt,\n        AnnexModel: full_fmt\n    }\n    if long_:\n        fmts[AnnexModel] += u\"  {ds.annex_local_size!S}/{ds.annex_worktree_size!S}\"\n\n    formatter = LsFormatter()\n    # weird problems happen in the parallel run -- TODO - figure it out\n    # for out in Parallel(n_jobs=1)(\n    #         delayed(format_ds_model)(formatter, dsm, full_fmt, format_exc=path_fmt + \"  {msg!R}\")\n    #         for dsm in dss):\n    #     print(out)\n    for dsm in dsms:\n        fmt = fmts[dsm.__class__]\n        ds_str = format_ds_model(formatter, dsm, fmt, format_exc=path_fmt + u\"  {msg!R}\")\n        safe_print(ds_str)\n        # workaround for explosion of git cat-file --batch processes\n        # https://github.com/datalad/datalad/issues/1888\n        if dsm.repo is not None:\n            dsm.repo.repo.close()\n            del dsm.repo\n            dsm.repo = None\n\n\ndef machinesize(humansize):\n    \"\"\"convert human-size string to machine-size\"\"\"\n    try:\n        size_str, size_unit = humansize.split(\" \")\n    except AttributeError:\n        return float(humansize)\n    unit_converter = {'Byte': 0, 'Bytes': 0, 'kB': 1, 'MB': 2, 'GB': 3, 'TB': 4, 'PB': 5}\n    machinesize = float(size_str) * (1000 ** unit_converter[size_unit])\n    return machinesize\n\n\ndef leaf_name(path):\n    \"\"\"takes a relative or absolute path and returns name of node at that location\"\"\"\n    head, tail = split(abspath(path))\n    return tail or basename(head)\n\n\ndef ignored(path, only_hidden=False):\n    \"\"\"if path is in the ignorelist return True\n\n    ignore list includes hidden files and git or annex maintained folders\n    when only_hidden set, only ignores hidden files and folders not git or annex maintained folders\n    \"\"\"\n    if isdir(opj(path, \".git\")) and not only_hidden:\n        return True\n    return '.' == leaf_name(path)[0] or leaf_name(path) == 'index.html'\n\n\ndef metadata_locator(fs_metadata=None, path=None, ds_path=None, metadata_path=None):\n    \"\"\"path to metadata file of node associated with the fs_metadata dictionary\n\n    Parameters\n    ----------\n    fs_metadata: dict\n      Metadata json of a node\n    path: str\n      Path to directory of metadata to be rendered\n    ds_path: str\n      Path to dataset root\n    metadata_path: str\n      Path to metadata root. Calculated relative to ds_path\n\n    Returns\n    -------\n    str\n      path to metadata of current node\n    \"\"\"\n\n    # use implicit paths unless paths explicitly specified\n    # Note: usage of ds_path as if it was the Repo's path. Therefore use\n    # realpath, since we switched to have symlinks resolved in repos but not in\n    # datasets\n    ds_path = realpath(ds_path) if ds_path else fs_metadata['repo']\n    path = path or fs_metadata['path']\n    metadata_path = metadata_path or '.git/datalad/metadata'\n    # directory metadata directory tree location\n    metadata_dir = opj(ds_path, metadata_path)\n    # relative path of current directory wrt dataset root\n    dir_path = relpath(path, ds_path) if isabs(path) else path\n    # normalize to / -- TODO, switch to '.' which is now actually the name since path is relative in web meta?\n    if dir_path in ('.', None, ''):\n        dir_path = '/'\n    # create md5 hash of current directory's relative path\n    metadata_hash = hashlib.md5(dir_path.encode('utf-8')).hexdigest()\n    # construct final path to metadata file\n    metadata_file = opj(metadata_dir, metadata_hash)\n\n    return metadata_file\n\n\ndef fs_extract(nodepath, repo, basepath='/'):\n    \"\"\"extract required info of nodepath with its associated parent repository and returns it as a dictionary\n\n    Parameters\n    ----------\n    nodepath : str\n        Full path to the location we are exploring (must be a directory within\n        `repo`\n    repo : GitRepo\n        Is the repository nodepath belongs to\n    \"\"\"\n    # Create FsModel from filesystem nodepath and its associated parent repository\n    node = FsModel(nodepath, repo)\n    pretty_size = {stype: humanize.naturalsize(svalue) for stype, svalue in node.size.items()}\n    pretty_date = time.strftime(u\"%Y-%m-%d %H:%M:%S\", time.localtime(node.date))\n    name = leaf_name(node._path) if leaf_name(node._path) != \"\" else leaf_name(node.repo.path)\n    rec = {\n        \"name\": name, \"path\": relpath(node._path, basepath),\n        \"type\": node.type_, \"size\": pretty_size, \"date\": pretty_date,\n    }\n    # if there is meta-data for the dataset (done by aggregate-metadata)\n    # we include it\n    metadata_path = opj(nodepath, METADATA_DIR, METADATA_FILENAME)\n    if exists(metadata_path):\n        # might need flattening!  TODO: flatten when aggregating?  why wasn't done?\n        metadata = js.load(open(metadata_path))\n        # might be too heavy to carry around, so will do basic flattening manually\n        # and in a basic fashion\n        # import jsonld\n        metadata_reduced = metadata[0]\n        for m in metadata[1:]:\n            metadata_reduced.update(m)\n        # but verify that they all had the same id\n        if metadata:\n            metaid = metadata[0]['@id']\n            assert all(m['@id'] == metaid for m in metadata)\n        rec[\"metadata\"] = metadata_reduced\n    return rec\n\n\ndef fs_render(fs_metadata, json=None, **kwargs):\n    \"\"\"takes node to render and based on json option passed renders to file, stdout or deletes json at root\n\n    Parameters\n    ----------\n    fs_metadata: dict\n      Metadata json to be rendered\n    json: str ('file', 'display', 'delete')\n      Render to file, stdout or delete json\n    \"\"\"\n\n    metadata_file = metadata_locator(fs_metadata, **kwargs)\n\n    if json == 'file':\n        # create metadata_root directory if it doesn't exist\n        metadata_dir = dirname(metadata_file)\n        if not exists(metadata_dir):\n            makedirs(metadata_dir)\n        # write directory metadata to json\n        with open(metadata_file, 'w') as f:\n            js.dump(fs_metadata, f)\n\n    # else if json flag set to delete, remove .dir.json of current directory\n    elif json == 'delete' and exists(metadata_file):\n        remove(metadata_file)\n\n    # else dump json to stdout\n    elif json == 'display':\n        safe_print(js.dumps(fs_metadata) + '\\n')\n\n\ndef fs_traverse(path, repo, parent=None, render=True, recursive=False, json=None, basepath=None):\n    \"\"\"Traverse path through its nodes and returns a dictionary of relevant attributes attached to each node\n\n    Parameters\n    ----------\n    path: str\n      Path to the directory to be traversed\n    repo: AnnexRepo or GitRepo\n      Repo object the directory belongs too\n    parent: dict\n      Extracted info about parent directory\n    recursive: bool\n      Recurse into subdirectories (note that subdatasets are not traversed)\n    render: bool\n       To render from within function or not. Set to false if results to be manipulated before final render\n\n    Returns\n    -------\n    list of dict\n      extracts and returns a (recursive) list of directory info at path\n      does not traverse into annex, git or hidden directories\n    \"\"\"\n    fs = fs_extract(path, repo, basepath=basepath or path)\n    if isdir(path):                     # if node is a directory\n        children = [fs.copy()]          # store its info in its children dict too  (Yarik is not sure why, but I guess for .?)\n        # ATM seems some pieces still rely on having this duplication, so left as is\n        # TODO: strip away\n        for node in listdir(path):\n            nodepath = opj(path, node)\n\n            # TODO:  it might be a subdir which is non-initialized submodule!\n            # if not ignored, append child node info to current nodes dictionary\n            if not ignored(nodepath):\n                # if recursive, create info dictionary (within) each child node too\n                if recursive:\n                    subdir = fs_traverse(nodepath,\n                                         repo,\n                                         parent=None,  # children[0],\n                                         recursive=recursive,\n                                         json=json,\n                                         basepath=basepath or path)\n                    subdir.pop('nodes', None)\n                else:\n                    # read child metadata from its metadata file if it exists\n                    subdir_json = metadata_locator(path=node, ds_path=basepath or path)\n                    if exists(subdir_json):\n                        with open(subdir_json) as data_file:\n                            subdir = js.load(data_file)\n                            subdir.pop('nodes', None)\n                    # else extract whatever information you can about the child\n                    else:\n                        # Yarik: this one is way too lean...\n                        subdir = fs_extract(nodepath,\n                                            repo,\n                                            basepath=basepath or path)\n                # append child metadata to list\n                children.extend([subdir])\n\n        # sum sizes of all 1st level children\n        children_size = {}\n        for node in children[1:]:\n            for size_type, child_size in node['size'].items():\n                children_size[size_type] = children_size.get(size_type, 0) + machinesize(child_size)\n\n        # update current node sizes to the humanized aggregate children size\n        fs['size'] = children[0]['size'] = \\\n            {size_type: humanize.naturalsize(child_size)\n             for size_type, child_size in children_size.items()}\n\n        children[0]['name'] = '.'       # replace current node name with '.' to emulate unix syntax\n        if parent:\n            parent['name'] = '..'       # replace parent node name with '..' to emulate unix syntax\n            children.insert(1, parent)  # insert parent info after current node info in children dict\n\n        fs['nodes'] = children          # add children info to main fs dictionary\n        if render:                      # render directory node at location(path)\n            fs_render(fs, json=json, ds_path=basepath or path)\n            lgr.info('Directory: %s' % path)\n\n    return fs\n\n\ndef ds_traverse(rootds, parent=None, json=None, recursive=False, all_=False,\n                long_=False):\n    \"\"\"Hierarchical dataset traverser\n\n    Parameters\n    ----------\n    rootds: Dataset\n      Root dataset to be traversed\n    parent: Dataset\n      Parent dataset of the current rootds\n    recursive: bool\n       Recurse into subdirectories of the current dataset\n    all_: bool\n       Recurse into subdatasets of the root dataset\n\n    Returns\n    -------\n    list of dict\n      extracts and returns a (recursive) list of dataset(s) info at path\n    \"\"\"\n    # extract parent info to pass to traverser\n    fsparent = fs_extract(parent.path, parent.repo, basepath=rootds.path) if parent else None\n\n    # (recursively) traverse file tree of current dataset\n    fs = fs_traverse(rootds.path, rootds.repo,\n                     render=False, parent=fsparent, recursive=all_,\n                     json=json)\n    size_list = [fs['size']]\n\n    # (recursively) traverse each subdataset\n    children = []\n    # yoh: was in return results branch returning full datasets:\n    # for subds in rootds.subdatasets(result_xfm='datasets'):\n    # but since rpath is needed/used, decided to return relpaths\n    for subds_rpath in rootds.subdatasets(result_xfm='relpaths'):\n\n        subds_path = opj(rootds.path, subds_rpath)\n        subds = Dataset(subds_path)\n        subds_json = metadata_locator(path='.', ds_path=subds_path)\n\n        def handle_not_installed():\n            # for now just traverse as fs\n            lgr.warning(\"%s is either not installed or lacks meta-data\", subds)\n            subfs = fs_extract(subds_path, rootds, basepath=rootds.path)\n            # but add a custom type that it is a not installed subds\n            subfs['type'] = 'uninitialized'\n            # we need to kick it out from 'children'\n            # TODO:  this is inefficient and cruel -- \"ignored\" should be made\n            # smarted to ignore submodules for the repo\n            if fs['nodes']:\n                fs['nodes'] = [c for c in fs['nodes'] if c['path'] != subds_rpath]\n            return subfs\n\n        if not subds.is_installed():\n            subfs = handle_not_installed()\n        elif recursive:\n            subfs = ds_traverse(subds,\n                                json=json,\n                                recursive=recursive,\n                                all_=all_,\n                                parent=rootds)\n            subfs.pop('nodes', None)\n            size_list.append(subfs['size'])\n        # else just pick the data from metadata_file of each subdataset\n        else:\n            lgr.info(subds.path)\n            if exists(subds_json):\n                with open(subds_json) as data_file:\n                    subfs = js.load(data_file)\n                    subfs.pop('nodes', None)    # remove children\n                    subfs['path'] = subds_rpath # reassign the path\n                    size_list.append(subfs['size'])\n            else:\n                # the same drill as if not installed\n                lgr.warning(\"%s is installed but no meta-data yet\", subds)\n                subfs = handle_not_installed()\n\n        children.extend([subfs])\n\n    # sum sizes of all 1st level children dataset\n    children_size = {}\n    for subdataset_size in size_list:\n        for size_type, subds_size in subdataset_size.items():\n            children_size[size_type] = children_size.get(size_type, 0) + machinesize(subds_size)\n\n    # update current dataset sizes to the humanized aggregate subdataset sizes\n    fs['size'] = {size_type: humanize.naturalsize(size)\n                  for size_type, size in children_size.items()}\n    fs['nodes'][0]['size'] = fs['size']  # update self's updated size in nodes sublist too!\n\n    # add dataset specific entries to its dict\n    rootds_model = GitModel(rootds.repo)\n    fs['tags'] = rootds_model.describe\n    fs['branch'] = rootds_model.branch\n    index_file = opj(rootds.path, '.git', 'index')\n    fs['index-mtime'] = time.strftime(\n        u\"%Y-%m-%d %H:%M:%S\",\n        time.localtime(getmtime(index_file))) if exists(index_file) else ''\n\n    # append children datasets info to current dataset\n    fs['nodes'].extend(children)\n\n    # render current dataset\n    lgr.info('Dataset: %s' % rootds.path)\n    fs_render(fs, json=json, ds_path=rootds.path)\n    return fs\n\n\ndef _ls_json(loc, fast=False, **kwargs):\n    # hierarchically traverse file tree of (sub-)dataset(s) under path passed(loc)\n    return ds_traverse(Dataset(loc), parent=None, **kwargs)\n\n\n#\n# S3 listing\n#\ndef _ls_s3(loc, fast=False, recursive=False, all_=False, long_=False,\n           config_file=None, list_content=False):\n    \"\"\"List S3 bucket content\"\"\"\n    if loc.startswith('s3://'):\n        bucket_prefix = loc[5:]\n    else:\n        raise ValueError(\"passed location should be an s3:// url\")\n\n    import boto\n    from hashlib import md5\n    from boto.s3.key import Key\n    from boto.s3.prefix import Prefix\n    from boto.exception import S3ResponseError\n    from ..support.configparserinc import SafeConfigParser  # provides PY2,3 imports\n\n    if '/' in bucket_prefix:\n        bucket_name, prefix = bucket_prefix.split('/', 1)\n    else:\n        bucket_name, prefix = bucket_prefix, None\n\n    if prefix and '?' in prefix:\n        ui.message(\"We do not care about URL options ATM, they get stripped\")\n        prefix = prefix[:prefix.index('?')]\n\n    ui.message(\"Connecting to bucket: %s\" % bucket_name)\n    if config_file:\n        config = SafeConfigParser()\n        config.read(config_file)\n        access_key = config.get('default', 'access_key')\n        secret_key = config.get('default', 'secret_key')\n\n        # TODO: remove duplication -- reuse logic within downloaders/s3.py to get connected\n        conn = boto.connect_s3(access_key, secret_key)\n        try:\n            bucket = conn.get_bucket(bucket_name)\n        except S3ResponseError as e:\n            ui.message(\"E: Cannot access bucket %s by name\" % bucket_name)\n            all_buckets = conn.get_all_buckets()\n            all_bucket_names = [b.name for b in all_buckets]\n            ui.message(\"I: Found following buckets %s\" % ', '.join(all_bucket_names))\n            if bucket_name in all_bucket_names:\n                bucket = all_buckets[all_bucket_names.index(bucket_name)]\n            else:\n                raise RuntimeError(\"E: no bucket named %s thus exiting\" % bucket_name)\n    else:\n        # TODO: expose credentials\n        # We don't need any provider here really but only credentials\n        from datalad.downloaders.providers import Providers\n        providers = Providers.from_config_files()\n        provider = providers.get_provider(loc)\n\n        if not provider:\n            raise ValueError(\n                \"Don't know how to deal with this url %s -- no provider defined for %s. \"\n                \"Define a new provider (DOCS: TODO) or specify just s3cmd config file instead for now.\"\n                % loc\n            )\n        downloader = provider.get_downloader(loc)\n\n        # should authenticate etc, and when ready we will ask for a bucket ;)\n        bucket = downloader.access(lambda url: downloader.bucket, loc)\n\n    info = []\n    for iname, imeth in [\n        (\"Versioning\", bucket.get_versioning_status),\n        (\"   Website\", bucket.get_website_endpoint),\n        (\"       ACL\", bucket.get_acl),\n    ]:\n        try:\n            ival = imeth()\n        except Exception as e:\n            ival = str(e).split('\\n')[0]\n        info.append(\" {iname}: {ival}\".format(**locals()))\n    ui.message(\"Bucket info:\\n %s\" % '\\n '.join(info))\n\n    kwargs = {} if recursive else {'delimiter': '/'}\n\n    ACCESS_METHODS = [\n        bucket.list_versions,\n        bucket.list\n    ]\n\n    prefix_all_versions = None\n    for acc in ACCESS_METHODS:\n        try:\n            prefix_all_versions = list(acc(prefix, **kwargs))\n            break\n        except Exception as exc:\n            lgr.debug(\"Failed to access via %s: %s\", acc, exc_str(exc))\n\n    if not prefix_all_versions:\n        ui.error(\"No output was provided for prefix %r\" % prefix)\n    else:\n        max_length = max((len(e.name) for e in prefix_all_versions))\n        max_size_length = max((len(str(getattr(e, 'size', 0))) for e in prefix_all_versions))\n\n    results = []\n    for e in prefix_all_versions:\n        results.append(e)\n        if isinstance(e, Prefix):\n            ui.message(\"%s\" % (e.name, ),)\n            continue\n\n        base_msg = (\"%%-%ds %%s\" % max_length) % (e.name, e.last_modified)\n        if isinstance(e, Key):\n            if not (e.is_latest or all_):\n                # Skip this one\n                continue\n            ui.message(base_msg + \" %%%dd\" % max_size_length % e.size, cr=' ')\n            # OPT: delayed import\n            from ..support.s3 import get_key_url\n            url = get_key_url(e, schema='http')\n            try:\n                _ = urlopen(Request(url))\n                urlok = \"OK\"\n            except HTTPError as err:\n                urlok = \"E: %s\" % err.code\n\n            try:\n                acl = e.get_acl()\n            except S3ResponseError as exc:\n                acl = exc.code if exc.code in ('AccessDenied',) else str(exc)\n\n            content = \"\"\n            if list_content:\n                # IO intensive, make an option finally!\n                try:\n                    # _ = e.next()[:5]  if we are able to fetch the content\n                    kwargs = dict(version_id=e.version_id)\n                    if list_content in {'full', 'first10'}:\n                        if list_content in 'first10':\n                            kwargs['headers'] = {'Range': 'bytes=0-9'}\n                        content = repr(e.get_contents_as_string(**kwargs))\n                    elif list_content == 'md5':\n                        digest = md5()\n                        digest.update(e.get_contents_as_string(**kwargs))\n                        content = digest.hexdigest()\n                    else:\n                        raise ValueError(list_content)\n                    # content = \"[S3: OK]\"\n                except S3ResponseError as err:\n                    content = str(err)\n                finally:\n                    content = \" \" + content\n            ui.message(\n                \"ver:%-32s  acl:%s  %s [%s]%s\"\n                % (getattr(e, 'version_id', None),\n                   acl, url, urlok, content)\n                if long_ else ''\n            )\n        else:\n            ui.message(base_msg + \" \" + str(type(e)).split('.')[-1].rstrip(\"\\\"'>\"))\n    return results\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil; coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Helper utility to list things.  ATM list content of S3 bucket\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport sys\nimport time\nfrom os.path import exists, lexists, join as opj, abspath, isabs, getmtime\nfrom os.path import curdir, isfile, islink, isdir, dirname, basename, split, realpath\nfrom os.path import relpath\nfrom os import listdir, lstat, remove, makedirs\nimport json as js\nimport hashlib\n\nfrom six.moves.urllib.request import urlopen, Request\nfrom six.moves.urllib.error import HTTPError\n\nfrom ..cmdline.helpers import get_repo_instance\nfrom ..utils import auto_repr\nfrom .base import Interface\nfrom datalad.interface.base import build_doc\nfrom ..ui import ui\nfrom ..utils import safe_print\nfrom ..consts import METADATA_DIR\nfrom ..consts import METADATA_FILENAME\nfrom ..dochelpers import exc_str\nfrom ..support.param import Parameter\nfrom ..support import ansi_colors\nfrom ..support.constraints import EnsureStr, EnsureNone\nfrom ..distribution.dataset import Dataset\n\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.support.annexrepo import GitRepo\n\nimport string\nimport humanize\nfrom datalad.utils import is_interactive\n\nfrom logging import getLogger\nlgr = getLogger('datalad.api.ls')\n\n\n@build_doc\nclass Ls(Interface):\n    \"\"\"List summary information about URLs and dataset(s)\n\n    ATM only s3:// URLs and datasets are supported\n\n    Examples:\n\n      $ datalad ls s3://openfmri/tarballs/ds202  # to list S3 bucket\n      $ datalad ls                               # to list current dataset\n    \"\"\"\n    # XXX prevent common args from being added to the docstring\n    _no_eval_results = True\n\n    # TODO: during big RF refactor this one away since it must not be instance's\n    # attribute.  For now introduced to make `datalad ls` be relatively usable\n    # in terms of speed\n    _cached_subdatasets = {}\n\n    _params_ = dict(\n        loc=Parameter(\n            doc=\"URL or path to list, e.g. s3://...\",\n            metavar='PATH/URL',\n            nargs=\"*\",\n            constraints=EnsureStr() | EnsureNone(),\n        ),\n        recursive=Parameter(\n            args=(\"-r\", \"--recursive\"),\n            action=\"store_true\",\n            doc=\"recurse into subdirectories\",\n        ),\n        fast=Parameter(\n            args=(\"-F\", \"--fast\"),\n            action=\"store_true\",\n            doc=\"only perform fast operations.  Would be overridden by --all\",\n        ),\n        all_=Parameter(\n            args=(\"-a\", \"--all\"),\n            dest='all_',\n            action=\"store_true\",\n            doc=\"list all (versions of) entries, not e.g. only latest entries \"\n                \"in case of S3\",\n        ),\n        long_=Parameter(\n            args=(\"-L\", \"--long\"),\n            dest='long_',\n            action=\"store_true\",\n            doc=\"list more information on entries (e.g. acl, urls in s3, annex \"\n                \"sizes etc)\",\n        ),\n        config_file=Parameter(\n            doc=\"\"\"path to config file which could help the 'ls'.  E.g. for s3://\n            URLs could be some ~/.s3cfg file which would provide credentials\"\"\",\n            constraints=EnsureStr() | EnsureNone()\n        ),\n        list_content=Parameter(\n            choices=(None, 'first10', 'md5', 'full'),\n            doc=\"\"\"list also the content or only first 10 bytes (first10), or md5\n            checksum of an entry.  Might require expensive transfer and dump\n            binary output to your screen.  Do not enable unless you know what you\n            are after\"\"\",\n            default=None\n        ),\n        json=Parameter(\n            choices=('file', 'display', 'delete'),\n            doc=\"\"\"metadata json of dataset for creating web user interface.\n            display: prints jsons to stdout or\n            file: writes each subdir metadata to json file in subdir of dataset or\n            delete: deletes all metadata json files in dataset\"\"\",\n        ),\n    )\n\n    @staticmethod\n    def __call__(loc, recursive=False, fast=False, all_=False, long_=False,\n                 config_file=None, list_content=False, json=None):\n        if isinstance(loc, list) and not len(loc):\n            # nothing given, CWD assumed -- just like regular ls\n            loc = '.'\n\n        kw = dict(fast=fast, recursive=recursive, all_=all_, long_=long_)\n        if isinstance(loc, list):\n            return [Ls.__call__(loc_, config_file=config_file,\n                                list_content=list_content, json=json, **kw)\n                    for loc_ in loc]\n\n        # TODO: do some clever handling of kwargs as to remember what were defaults\n        # and what any particular implementation actually needs, and then issuing\n        # warning if some custom value/option was specified which doesn't apply to the\n        # given url\n\n        # rename to not angry Python gods who took all_ good words\n        kw['long_'] = kw.pop('long_')\n\n        loc_type = \"unknown\"\n        if loc.startswith('s3://'):\n            return _ls_s3(loc, config_file=config_file, list_content=list_content,\n                          **kw)\n        elif lexists(loc):\n            if isdir(loc):\n                ds = Dataset(loc)\n                if ds.is_installed():\n                    return _ls_json(loc, json=json, **kw) if json else _ls_dataset(loc, **kw)\n                    loc_type = False\n                else:\n                    loc_type = \"dir\"  # we know that so far for sure\n                    # it might have been an uninstalled dataset within super-dataset\n                    superds = ds.get_superdataset()\n                    if superds:\n                        try:\n                            subdatasets = Ls._cached_subdatasets[superds.path]\n                        except KeyError:\n                            subdatasets = Ls._cached_subdatasets[superds.path] \\\n                                = superds.subdatasets(result_xfm='relpaths')\n                        if relpath(ds.path, superds.path) in subdatasets:\n                            loc_type = \"not installed\"\n            else:\n                loc_type = \"file\"\n                # could list properties -- under annex or git, either clean/dirty\n                # etc\n                # repo = get_repo_instance(dirname(loc))\n\n        if loc_type:\n            #raise ValueError(\"ATM supporting only s3:// URLs and paths to local datasets\")\n            # TODO: unify all_ the output here -- _ls functions should just return something\n            # to be displayed\n            ui.message(\n                \"{}  {}\".format(\n                    ansi_colors.color_word(loc, ansi_colors.DATASET),\n                    ansi_colors.color_word(\n                        loc_type,\n                        ansi_colors.RED\n                        if loc_type in {'unknown', 'not installed'}\n                        else ansi_colors.BLUE)\n                )\n            )\n\n\n#\n# Dataset listing\n#\n\n@auto_repr\nclass AbsentRepoModel(object):\n    \"\"\"Just a base for those where repo wasn't installed yet\"\"\"\n\n    def __init__(self, path):\n        self.path = path\n        self.repo = None\n\n    @property\n    def type(self):\n        return \"N/A\"\n\n\n@auto_repr\nclass GitModel(object):\n    \"\"\"A base class for models which have some .repo available\"\"\"\n\n    __slots__ = ['_branch', 'repo', '_path']\n\n    def __init__(self, repo):\n        self.repo = repo\n        # lazy evaluation variables\n        self._branch = None\n        self._path = None\n\n    @property\n    def path(self):\n        return self.repo.path if self._path is None else self._path\n\n    @path.setter\n    def path(self, v):\n        self._path = v\n\n    @property\n    def branch(self):\n        if self._branch is None:\n            try:\n                self._branch = self.repo.get_active_branch()\n            except:  # MIH: InvalidGitRepositoryError?\n                return None\n        return self._branch\n\n    @property\n    def clean(self):\n        return not self.repo.dirty\n\n    @property\n    def describe(self):\n        return self.repo.describe(tags=True)\n\n    @property\n    def date(self):\n        \"\"\"Date of the last commit\n        \"\"\"\n        return self.repo.get_committed_date()\n\n    @property\n    def count_objects(self):\n        return self.repo.count_objects\n\n    @property\n    def git_local_size(self):\n        count_objects = self.count_objects\n        return count_objects['size'] if count_objects else None\n\n    @property\n    def type(self):\n        return {False: 'git', True: 'annex'}[isinstance(self.repo, AnnexRepo)]\n\n\n@auto_repr\nclass AnnexModel(GitModel):\n\n    __slots__ = ['_info'] + GitModel.__slots__\n\n    def __init__(self, *args, **kwargs):\n        super(AnnexModel, self).__init__(*args, **kwargs)\n        self._info = None\n\n    @property\n    def info(self):\n        if self._info is None and self.type == 'annex':\n            self._info = self.repo.repo_info()\n        return self._info\n\n    @property\n    def annex_worktree_size(self):\n        info = self.info\n        return info['size of annexed files in working tree'] if info else 0.0\n\n    @property\n    def annex_local_size(self):\n        info = self.info\n        return info['local annex size'] if info else 0.0\n\n\n@auto_repr\nclass FsModel(AnnexModel):\n\n    __slots__ = AnnexModel.__slots__\n\n    def __init__(self, path, *args, **kwargs):\n        super(FsModel, self).__init__(*args, **kwargs)\n        self._path = path\n\n    @property\n    def path(self):\n        return self._path\n\n    @property\n    def symlink(self):\n        \"\"\"if symlink returns path the symlink points to else returns None\"\"\"\n        if islink(self._path):                    # if symlink\n            target_path = realpath(self._path)    # find link target\n            # convert to absolute path if not\n            return target_path if exists(target_path) else None\n        return None\n\n    @property\n    def date(self):\n        \"\"\"Date of last modification\"\"\"\n        if self.type_ is not ['git', 'annex']:\n            return lstat(self._path).st_mtime\n        else:\n            super(self.__class__, self).date\n\n    @property\n    def size(self):\n        \"\"\"Size of the node computed based on its type\"\"\"\n        type_ = self.type_\n        sizes = {'total': 0.0, 'ondisk': 0.0, 'git': 0.0, 'annex': 0.0, 'annex_worktree': 0.0}\n\n        if type_ in ['file', 'link', 'link-broken']:\n            # if node is under annex, ask annex for node size, ondisk_size\n            if isinstance(self.repo, AnnexRepo) and self.repo.is_under_annex(self._path):\n                size = self.repo.info(self._path, batch=True)['size']\n                ondisk_size = size \\\n                    if self.repo.file_has_content(self._path) \\\n                    else 0\n            # else ask fs for node size (= ondisk_size)\n            else:\n                size = ondisk_size = 0 \\\n                    if type_ == 'link-broken' \\\n                    else lstat(self.symlink or self._path).st_size\n\n            sizes.update({'total': size, 'ondisk': ondisk_size})\n\n        if self.repo.path == self._path:\n            sizes.update({'git': self.git_local_size,\n                          'annex': self.annex_local_size,\n                          'annex_worktree': self.annex_worktree_size})\n        return sizes\n\n    @property\n    def type_(self):\n        \"\"\"outputs the node type\n\n        Types: link, link-broken, file, dir, annex-repo, git-repo\"\"\"\n        if islink(self.path):\n            return 'link' if self.symlink else 'link-broken'\n        elif isfile(self.path):\n            return 'file'\n        elif exists(opj(self.path, \".git\", \"annex\")):\n            return 'annex'\n        elif exists(opj(self.path, \".git\")):\n            return 'git'\n        elif isdir(self.path):\n            return 'dir'\n        else:\n            return None\n\n\nclass LsFormatter(string.Formatter):\n    # condition by interactive\n    if is_interactive():\n        BLUE = ansi_colors.COLOR_SEQ % ansi_colors.BLUE\n        RED = ansi_colors.COLOR_SEQ % ansi_colors.RED\n        GREEN = ansi_colors.COLOR_SEQ % ansi_colors.GREEN\n        RESET = ansi_colors.RESET_SEQ\n        DATASET = ansi_colors.COLOR_SEQ % ansi_colors.UNDERLINE\n    else:\n        BLUE = RED = GREEN = RESET = DATASET = u\"\"\n\n    # TODO: we might want to just ignore and force utf8 while explicitly .encode()'ing output!\n    # unicode versions which look better but which blow during tests etc\n    # Those might be reset by the constructor\n    OK = 'OK'   # u\"✓\"\n    NOK = 'X'  # u\"✗\"\n    NONE = '-'  # u\"✗\"\n\n    def __init__(self, *args, **kwargs):\n        super(LsFormatter, self).__init__(*args, **kwargs)\n        if sys.stdout.encoding is None:\n            lgr.debug(\"encoding not set, using safe alternatives\")\n        else:\n            try:\n                u\"✓\".encode(sys.stdout.encoding)\n            except UnicodeEncodeError:\n                lgr.debug(\"encoding %s does not support unicode, \"\n                          \"using safe alternatives\",\n                          sys.stdout.encoding)\n            else:\n                self.OK = u\"✓\"\n                self.NOK = u\"✗\"\n                self.NONE = u\"✗\"\n\n    def convert_field(self, value, conversion):\n        #print(\"%r->%r\" % (value, conversion))\n        if conversion == 'D':  # Date\n            if value is not None:\n                return time.strftime(u\"%Y-%m-%d/%H:%M:%S\", time.localtime(value))\n            else:\n                return u'-'\n        elif conversion == 'S':  # Human size\n            #return value\n            if value is not None:\n                return humanize.naturalsize(value)\n            else:\n                return u'-'\n        elif conversion == 'X':  # colored bool\n            chr, col = (self.OK, self.GREEN) if value else (self.NOK, self.RED)\n            return u\"%s%s%s\" % (col, chr, self.RESET)\n        elif conversion == 'N':  # colored Red - if None\n            if value is None:\n                # return \"%s✖%s\" % (self.RED, self.RESET)\n                return u\"%s%s%s\" % (self.RED, self.NONE, self.RESET)\n            return value\n        elif conversion in {'B', 'R', 'U'}:\n            return u\"%s%s%s\" % ({'B': self.BLUE, 'R': self.RED, 'U': self.DATASET}[conversion], value, self.RESET)\n\n        return super(LsFormatter, self).convert_field(value, conversion)\n\n    def format_field(self, value, format_spec):\n        # TODO: move all the \"coloring\" into formatting, so we could correctly indent\n        # given the format and only then color it up\n        # print \"> %r, %r\" % (value, format_spec)\n        return super(LsFormatter, self).format_field(value, format_spec)\n\n\ndef format_ds_model(formatter, ds_model, format_str, format_exc):\n    try:\n        #print(\"WORKING ON %s\" % ds_model.path)\n        if not exists(ds_model.path) or not ds_model.repo:\n            return formatter.format(format_exc, ds=ds_model, msg=u\"not installed\")\n        ds_formatted = formatter.format(format_str, ds=ds_model)\n        #print(\"FINISHED ON %s\" % ds_model.path)\n        return ds_formatted\n    except Exception as exc:\n        return formatter.format(format_exc, ds=ds_model, msg=exc_str(exc))\n\n# from joblib import Parallel, delayed\n\n\ndef _ls_dataset(loc, fast=False, recursive=False, all_=False, long_=False):\n    isabs_loc = isabs(loc)\n    topdir = '' if isabs_loc else abspath(curdir)\n\n    topds = Dataset(loc)\n    dss = [topds] + (\n        [Dataset(opj(loc, sm))\n         for sm in topds.subdatasets(recursive=recursive, result_xfm='relpaths')]\n        if recursive else [])\n\n    dsms = []\n    for ds in dss:\n        if not ds.is_installed():\n            dsm = AbsentRepoModel(ds.path)\n        elif isinstance(ds.repo, AnnexRepo):\n            dsm = AnnexModel(ds.repo)\n        elif isinstance(ds.repo, GitRepo):\n            dsm = GitModel(ds.repo)\n        else:\n            raise RuntimeError(\"Got some dataset which don't know how to handle %s\"\n                               % ds)\n        dsms.append(dsm)\n\n    # adjust path strings\n    for ds_model in dsms:\n        #path = ds_model.path[len(topdir) + 1 if topdir else 0:]\n        path = relpath(ds_model.path, topdir) if topdir else ds_model.path\n        if not path:\n            path = '.'\n        ds_model.path = path\n    dsms = sorted(dsms, key=lambda m: m.path)\n\n    maxpath = max(len(ds_model.path) for ds_model in dsms)\n    path_fmt = u\"{ds.path!U:<%d}\" % (maxpath + (11 if is_interactive() else 0))  # + to accommodate ansi codes\n    pathtype_fmt = path_fmt + u\"  [{ds.type}]\"\n    full_fmt = pathtype_fmt + u\"  {ds.branch!N}  {ds.describe!N} {ds.date!D}\"\n    if (not fast) or long_:\n        full_fmt += u\"  {ds.clean!X}\"\n\n    fmts = {\n        AbsentRepoModel: pathtype_fmt,\n        GitModel: full_fmt,\n        AnnexModel: full_fmt\n    }\n    if long_:\n        fmts[AnnexModel] += u\"  {ds.annex_local_size!S}/{ds.annex_worktree_size!S}\"\n\n    formatter = LsFormatter()\n    # weird problems happen in the parallel run -- TODO - figure it out\n    # for out in Parallel(n_jobs=1)(\n    #         delayed(format_ds_model)(formatter, dsm, full_fmt, format_exc=path_fmt + \"  {msg!R}\")\n    #         for dsm in dss):\n    #     print(out)\n    for dsm in dsms:\n        fmt = fmts[dsm.__class__]\n        ds_str = format_ds_model(formatter, dsm, fmt, format_exc=path_fmt + u\"  {msg!R}\")\n        safe_print(ds_str)\n        # workaround for explosion of git cat-file --batch processes\n        # https://github.com/datalad/datalad/issues/1888\n        if dsm.repo is not None:\n            dsm.repo.repo.close()\n            del dsm.repo\n            dsm.repo = None\n\n\ndef machinesize(humansize):\n    \"\"\"convert human-size string to machine-size\"\"\"\n    try:\n        size_str, size_unit = humansize.split(\" \")\n    except AttributeError:\n        return float(humansize)\n    unit_converter = {'Byte': 0, 'Bytes': 0, 'kB': 1, 'MB': 2, 'GB': 3, 'TB': 4, 'PB': 5}\n    machinesize = float(size_str) * (1000 ** unit_converter[size_unit])\n    return machinesize\n\n\ndef leaf_name(path):\n    \"\"\"takes a relative or absolute path and returns name of node at that location\"\"\"\n    head, tail = split(abspath(path))\n    return tail or basename(head)\n\n\ndef ignored(path, only_hidden=False):\n    \"\"\"if path is in the ignorelist return True\n\n    ignore list includes hidden files and git or annex maintained folders\n    when only_hidden set, only ignores hidden files and folders not git or annex maintained folders\n    \"\"\"\n    if isdir(opj(path, \".git\")) and not only_hidden:\n        return True\n    return '.' == leaf_name(path)[0] or leaf_name(path) == 'index.html'\n\n\ndef metadata_locator(fs_metadata=None, path=None, ds_path=None, metadata_path=None):\n    \"\"\"path to metadata file of node associated with the fs_metadata dictionary\n\n    Parameters\n    ----------\n    fs_metadata: dict\n      Metadata json of a node\n    path: str\n      Path to directory of metadata to be rendered\n    ds_path: str\n      Path to dataset root\n    metadata_path: str\n      Path to metadata root. Calculated relative to ds_path\n\n    Returns\n    -------\n    str\n      path to metadata of current node\n    \"\"\"\n\n    # use implicit paths unless paths explicitly specified\n    # Note: usage of ds_path as if it was the Repo's path. Therefore use\n    # realpath, since we switched to have symlinks resolved in repos but not in\n    # datasets\n    ds_path = realpath(ds_path) if ds_path else fs_metadata['repo']\n    path = path or fs_metadata['path']\n    metadata_path = metadata_path or '.git/datalad/metadata'\n    # directory metadata directory tree location\n    metadata_dir = opj(ds_path, metadata_path)\n    # relative path of current directory wrt dataset root\n    dir_path = relpath(path, ds_path) if isabs(path) else path\n    # normalize to / -- TODO, switch to '.' which is now actually the name since path is relative in web meta?\n    if dir_path in ('.', None, ''):\n        dir_path = '/'\n    # create md5 hash of current directory's relative path\n    metadata_hash = hashlib.md5(dir_path.encode('utf-8')).hexdigest()\n    # construct final path to metadata file\n    metadata_file = opj(metadata_dir, metadata_hash)\n\n    return metadata_file\n\n\ndef fs_extract(nodepath, repo, basepath='/'):\n    \"\"\"extract required info of nodepath with its associated parent repository and returns it as a dictionary\n\n    Parameters\n    ----------\n    nodepath : str\n        Full path to the location we are exploring (must be a directory within\n        `repo`\n    repo : GitRepo\n        Is the repository nodepath belongs to\n    \"\"\"\n    # Create FsModel from filesystem nodepath and its associated parent repository\n    node = FsModel(nodepath, repo)\n    pretty_size = {stype: humanize.naturalsize(svalue) for stype, svalue in node.size.items()}\n    pretty_date = time.strftime(u\"%Y-%m-%d %H:%M:%S\", time.localtime(node.date))\n    name = leaf_name(node._path) if leaf_name(node._path) != \"\" else leaf_name(node.repo.path)\n    rec = {\n        \"name\": name, \"path\": relpath(node._path, basepath),\n        \"type\": node.type_, \"size\": pretty_size, \"date\": pretty_date,\n    }\n    # if there is meta-data for the dataset (done by aggregate-metadata)\n    # we include it\n    metadata_path = opj(nodepath, METADATA_DIR, METADATA_FILENAME)\n    if exists(metadata_path):\n        # might need flattening!  TODO: flatten when aggregating?  why wasn't done?\n        metadata = js.load(open(metadata_path))\n        # might be too heavy to carry around, so will do basic flattening manually\n        # and in a basic fashion\n        # import jsonld\n        metadata_reduced = metadata[0]\n        for m in metadata[1:]:\n            metadata_reduced.update(m)\n        # but verify that they all had the same id\n        if metadata:\n            metaid = metadata[0]['@id']\n            assert all(m['@id'] == metaid for m in metadata)\n        rec[\"metadata\"] = metadata_reduced\n    return rec\n\n\ndef fs_render(fs_metadata, json=None, **kwargs):\n    \"\"\"takes node to render and based on json option passed renders to file, stdout or deletes json at root\n\n    Parameters\n    ----------\n    fs_metadata: dict\n      Metadata json to be rendered\n    json: str ('file', 'display', 'delete')\n      Render to file, stdout or delete json\n    \"\"\"\n\n    metadata_file = metadata_locator(fs_metadata, **kwargs)\n\n    if json == 'file':\n        # create metadata_root directory if it doesn't exist\n        metadata_dir = dirname(metadata_file)\n        if not exists(metadata_dir):\n            makedirs(metadata_dir)\n        # write directory metadata to json\n        with open(metadata_file, 'w') as f:\n            js.dump(fs_metadata, f)\n\n    # else if json flag set to delete, remove .dir.json of current directory\n    elif json == 'delete' and exists(metadata_file):\n        remove(metadata_file)\n\n    # else dump json to stdout\n    elif json == 'display':\n        safe_print(js.dumps(fs_metadata) + '\\n')\n\n\ndef fs_traverse(path, repo, parent=None, render=True, recursive=False, json=None, basepath=None):\n    \"\"\"Traverse path through its nodes and returns a dictionary of relevant attributes attached to each node\n\n    Parameters\n    ----------\n    path: str\n      Path to the directory to be traversed\n    repo: AnnexRepo or GitRepo\n      Repo object the directory belongs too\n    parent: dict\n      Extracted info about parent directory\n    recursive: bool\n      Recurse into subdirectories (note that subdatasets are not traversed)\n    render: bool\n       To render from within function or not. Set to false if results to be manipulated before final render\n\n    Returns\n    -------\n    list of dict\n      extracts and returns a (recursive) list of directory info at path\n      does not traverse into annex, git or hidden directories\n    \"\"\"\n    fs = fs_extract(path, repo, basepath=basepath or path)\n    if isdir(path):                     # if node is a directory\n        children = [fs.copy()]          # store its info in its children dict too  (Yarik is not sure why, but I guess for .?)\n        # ATM seems some pieces still rely on having this duplication, so left as is\n        # TODO: strip away\n        for node in listdir(path):\n            nodepath = opj(path, node)\n\n            # TODO:  it might be a subdir which is non-initialized submodule!\n            # if not ignored, append child node info to current nodes dictionary\n            if not ignored(nodepath):\n                # if recursive, create info dictionary (within) each child node too\n                if recursive:\n                    subdir = fs_traverse(nodepath,\n                                         repo,\n                                         parent=None,  # children[0],\n                                         recursive=recursive,\n                                         json=json,\n                                         basepath=basepath or path)\n                    subdir.pop('nodes', None)\n                else:\n                    # read child metadata from its metadata file if it exists\n                    subdir_json = metadata_locator(path=node, ds_path=basepath or path)\n                    if exists(subdir_json):\n                        with open(subdir_json) as data_file:\n                            subdir = js.load(data_file)\n                            subdir.pop('nodes', None)\n                    # else extract whatever information you can about the child\n                    else:\n                        # Yarik: this one is way too lean...\n                        subdir = fs_extract(nodepath,\n                                            repo,\n                                            basepath=basepath or path)\n                # append child metadata to list\n                children.extend([subdir])\n\n        # sum sizes of all 1st level children\n        children_size = {}\n        for node in children[1:]:\n            for size_type, child_size in node['size'].items():\n                children_size[size_type] = children_size.get(size_type, 0) + machinesize(child_size)\n\n        # update current node sizes to the humanized aggregate children size\n        fs['size'] = children[0]['size'] = \\\n            {size_type: humanize.naturalsize(child_size)\n             for size_type, child_size in children_size.items()}\n\n        children[0]['name'] = '.'       # replace current node name with '.' to emulate unix syntax\n        if parent:\n            parent['name'] = '..'       # replace parent node name with '..' to emulate unix syntax\n            children.insert(1, parent)  # insert parent info after current node info in children dict\n\n        fs['nodes'] = children          # add children info to main fs dictionary\n        if render:                      # render directory node at location(path)\n            fs_render(fs, json=json, ds_path=basepath or path)\n            lgr.info('Directory: %s' % path)\n\n    return fs\n\n\ndef ds_traverse(rootds, parent=None, json=None, recursive=False, all_=False,\n                long_=False):\n    \"\"\"Hierarchical dataset traverser\n\n    Parameters\n    ----------\n    rootds: Dataset\n      Root dataset to be traversed\n    parent: Dataset\n      Parent dataset of the current rootds\n    recursive: bool\n       Recurse into subdirectories of the current dataset\n    all_: bool\n       Recurse into subdatasets of the root dataset\n\n    Returns\n    -------\n    list of dict\n      extracts and returns a (recursive) list of dataset(s) info at path\n    \"\"\"\n    # extract parent info to pass to traverser\n    fsparent = fs_extract(parent.path, parent.repo, basepath=rootds.path) if parent else None\n\n    # (recursively) traverse file tree of current dataset\n    fs = fs_traverse(rootds.path, rootds.repo,\n                     render=False, parent=fsparent, recursive=all_,\n                     json=json)\n    size_list = [fs['size']]\n\n    # (recursively) traverse each subdataset\n    children = []\n    # yoh: was in return results branch returning full datasets:\n    # for subds in rootds.subdatasets(result_xfm='datasets'):\n    # but since rpath is needed/used, decided to return relpaths\n    for subds_rpath in rootds.subdatasets(result_xfm='relpaths'):\n\n        subds_path = opj(rootds.path, subds_rpath)\n        subds = Dataset(subds_path)\n        subds_json = metadata_locator(path='.', ds_path=subds_path)\n\n        def handle_not_installed():\n            # for now just traverse as fs\n            lgr.warning(\"%s is either not installed or lacks meta-data\", subds)\n            subfs = fs_extract(subds_path, rootds, basepath=rootds.path)\n            # but add a custom type that it is a not installed subds\n            subfs['type'] = 'uninitialized'\n            # we need to kick it out from 'children'\n            # TODO:  this is inefficient and cruel -- \"ignored\" should be made\n            # smarted to ignore submodules for the repo\n            if fs['nodes']:\n                fs['nodes'] = [c for c in fs['nodes'] if c['path'] != subds_rpath]\n            return subfs\n\n        if not subds.is_installed():\n            subfs = handle_not_installed()\n        elif recursive:\n            subfs = ds_traverse(subds,\n                                json=json,\n                                recursive=recursive,\n                                all_=all_,\n                                parent=rootds)\n            subfs.pop('nodes', None)\n            size_list.append(subfs['size'])\n        # else just pick the data from metadata_file of each subdataset\n        else:\n            lgr.info(subds.path)\n            if exists(subds_json):\n                with open(subds_json) as data_file:\n                    subfs = js.load(data_file)\n                    subfs.pop('nodes', None)    # remove children\n                    subfs['path'] = subds_rpath # reassign the path\n                    size_list.append(subfs['size'])\n            else:\n                # the same drill as if not installed\n                lgr.warning(\"%s is installed but no meta-data yet\", subds)\n                subfs = handle_not_installed()\n\n        children.extend([subfs])\n\n    # sum sizes of all 1st level children dataset\n    children_size = {}\n    for subdataset_size in size_list:\n        for size_type, subds_size in subdataset_size.items():\n            children_size[size_type] = children_size.get(size_type, 0) + machinesize(subds_size)\n\n    # update current dataset sizes to the humanized aggregate subdataset sizes\n    fs['size'] = {size_type: humanize.naturalsize(size)\n                  for size_type, size in children_size.items()}\n    fs['nodes'][0]['size'] = fs['size']  # update self's updated size in nodes sublist too!\n\n    # add dataset specific entries to its dict\n    rootds_model = GitModel(rootds.repo)\n    fs['tags'] = rootds_model.describe\n    fs['branch'] = rootds_model.branch\n    index_file = opj(rootds.path, '.git', 'index')\n    fs['index-mtime'] = time.strftime(\n        u\"%Y-%m-%d %H:%M:%S\",\n        time.localtime(getmtime(index_file))) if exists(index_file) else ''\n\n    # append children datasets info to current dataset\n    fs['nodes'].extend(children)\n\n    # render current dataset\n    lgr.info('Dataset: %s' % rootds.path)\n    fs_render(fs, json=json, ds_path=rootds.path)\n    return fs\n\n\ndef _ls_json(loc, fast=False, **kwargs):\n    # hierarchically traverse file tree of (sub-)dataset(s) under path passed(loc)\n    return ds_traverse(Dataset(loc), parent=None, **kwargs)\n\n\n#\n# S3 listing\n#\ndef _ls_s3(loc, fast=False, recursive=False, all_=False, long_=False,\n           config_file=None, list_content=False):\n    \"\"\"List S3 bucket content\"\"\"\n    if loc.startswith('s3://'):\n        bucket_prefix = loc[5:]\n    else:\n        raise ValueError(\"passed location should be an s3:// url\")\n\n    import boto\n    from hashlib import md5\n    from boto.s3.key import Key\n    from boto.s3.prefix import Prefix\n    from boto.exception import S3ResponseError\n    from ..support.configparserinc import SafeConfigParser  # provides PY2,3 imports\n\n    if '/' in bucket_prefix:\n        bucket_name, prefix = bucket_prefix.split('/', 1)\n    else:\n        bucket_name, prefix = bucket_prefix, None\n\n    if prefix and '?' in prefix:\n        ui.message(\"We do not care about URL options ATM, they get stripped\")\n        prefix = prefix[:prefix.index('?')]\n\n    ui.message(\"Connecting to bucket: %s\" % bucket_name)\n    if config_file:\n        config = SafeConfigParser()\n        config.read(config_file)\n        access_key = config.get('default', 'access_key')\n        secret_key = config.get('default', 'secret_key')\n\n        # TODO: remove duplication -- reuse logic within downloaders/s3.py to get connected\n        conn = boto.connect_s3(access_key, secret_key)\n        try:\n            bucket = conn.get_bucket(bucket_name)\n        except S3ResponseError as e:\n            ui.message(\"E: Cannot access bucket %s by name\" % bucket_name)\n            all_buckets = conn.get_all_buckets()\n            all_bucket_names = [b.name for b in all_buckets]\n            ui.message(\"I: Found following buckets %s\" % ', '.join(all_bucket_names))\n            if bucket_name in all_bucket_names:\n                bucket = all_buckets[all_bucket_names.index(bucket_name)]\n            else:\n                raise RuntimeError(\"E: no bucket named %s thus exiting\" % bucket_name)\n    else:\n        # TODO: expose credentials\n        # We don't need any provider here really but only credentials\n        from datalad.downloaders.providers import Providers\n        providers = Providers.from_config_files()\n        provider = providers.get_provider(loc)\n\n        if not provider:\n            raise ValueError(\n                \"Don't know how to deal with this url %s -- no provider defined for %s. \"\n                \"Define a new provider (DOCS: TODO) or specify just s3cmd config file instead for now.\"\n                % loc\n            )\n        downloader = provider.get_downloader(loc)\n\n        # should authenticate etc, and when ready we will ask for a bucket ;)\n        bucket = downloader.access(lambda url: downloader.bucket, loc)\n\n    info = []\n    for iname, imeth in [\n        (\"Versioning\", bucket.get_versioning_status),\n        (\"   Website\", bucket.get_website_endpoint),\n        (\"       ACL\", bucket.get_acl),\n    ]:\n        try:\n            ival = imeth()\n        except Exception as e:\n            ival = str(e).split('\\n')[0]\n        info.append(\" {iname}: {ival}\".format(**locals()))\n    ui.message(\"Bucket info:\\n %s\" % '\\n '.join(info))\n\n    kwargs = {} if recursive else {'delimiter': '/'}\n\n    ACCESS_METHODS = [\n        bucket.list_versions,\n        bucket.list\n    ]\n\n    prefix_all_versions = None\n    for acc in ACCESS_METHODS:\n        try:\n            prefix_all_versions = list(acc(prefix, **kwargs))\n            break\n        except Exception as exc:\n            lgr.debug(\"Failed to access via %s: %s\", acc, exc_str(exc))\n\n    if not prefix_all_versions:\n        ui.error(\"No output was provided for prefix %r\" % prefix)\n    else:\n        max_length = max((len(e.name) for e in prefix_all_versions))\n        max_size_length = max((len(str(getattr(e, 'size', 0))) for e in prefix_all_versions))\n\n    results = []\n    for e in prefix_all_versions:\n        results.append(e)\n        if isinstance(e, Prefix):\n            ui.message(\"%s\" % (e.name, ),)\n            continue\n\n        base_msg = (\"%%-%ds %%s\" % max_length) % (e.name, e.last_modified)\n        if isinstance(e, Key):\n            if not (e.is_latest or all_):\n                # Skip this one\n                continue\n            ui.message(base_msg + \" %%%dd\" % max_size_length % e.size, cr=' ')\n            # OPT: delayed import\n            from ..support.s3 import get_key_url\n            url = get_key_url(e, schema='http')\n            try:\n                _ = urlopen(Request(url))\n                urlok = \"OK\"\n            except HTTPError as err:\n                urlok = \"E: %s\" % err.code\n\n            try:\n                acl = e.get_acl()\n            except S3ResponseError as exc:\n                acl = exc.code if exc.code in ('AccessDenied',) else str(exc)\n\n            content = \"\"\n            if list_content:\n                # IO intensive, make an option finally!\n                try:\n                    # _ = e.next()[:5]  if we are able to fetch the content\n                    kwargs = dict(version_id=e.version_id)\n                    if list_content in {'full', 'first10'}:\n                        if list_content in 'first10':\n                            kwargs['headers'] = {'Range': 'bytes=0-9'}\n                        content = repr(e.get_contents_as_string(**kwargs))\n                    elif list_content == 'md5':\n                        digest = md5()\n                        digest.update(e.get_contents_as_string(**kwargs))\n                        content = digest.hexdigest()\n                    else:\n                        raise ValueError(list_content)\n                    # content = \"[S3: OK]\"\n                except S3ResponseError as err:\n                    content = str(err)\n                finally:\n                    content = \" \" + content\n            ui.message(\n                \"ver:%-32s  acl:%s  %s [%s]%s\"\n                % (getattr(e, 'version_id', None),\n                   acl, url, urlok, content)\n                if long_ else ''\n            )\n        else:\n            ui.message(base_msg + \" \" + str(type(e)).split('.')[-1].rstrip(\"\\\"'>\"))\n    return results\n",
            "header": {
                "file": {
                    "path": "datalad/interface/ls.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "e979d255c",
                "new_commit_hash": "523fa0e71",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/interface/ls.py"
            },
            "plus_file": {
                "path": "b/datalad/interface/ls.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 375,
                            "length": 23
                        },
                        "new_range": {
                            "start": 375,
                            "length": 25
                        },
                        "section": "class LsFormatter(string.Formatter):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    # TODO: we might want to just ignore and force utf8 while explicitly .encode()'ing output!",
                                "type": "context"
                            },
                            {
                                "content": "    # unicode versions which look better but which blow during tests etc",
                                "type": "context"
                            },
                            {
                                "content": "    # Those might be reset by the constructor",
                                "type": "context"
                            },
                            {
                                "content": "    OK = u\"✓\"",
                                "type": "deleted"
                            },
                            {
                                "content": "    NOK = u\"✗\"",
                                "type": "deleted"
                            },
                            {
                                "content": "    NONE = u\"✗\"",
                                "type": "deleted"
                            },
                            {
                                "content": "    OK = 'OK'   # u\"✓\"",
                                "type": "added"
                            },
                            {
                                "content": "    NOK = 'X'  # u\"✗\"",
                                "type": "added"
                            },
                            {
                                "content": "    NONE = '-'  # u\"✗\"",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    def __init__(self, *args, **kwargs):",
                                "type": "context"
                            },
                            {
                                "content": "        super(LsFormatter, self).__init__(*args, **kwargs)",
                                "type": "context"
                            },
                            {
                                "content": "        for setting_encoding in (sys.getdefaultencoding(),",
                                "type": "deleted"
                            },
                            {
                                "content": "                                 sys.stdout.encoding):",
                                "type": "deleted"
                            },
                            {
                                "content": "        if sys.stdout.encoding is None:",
                                "type": "added"
                            },
                            {
                                "content": "            lgr.debug(\"encoding not set, using safe alternatives\")",
                                "type": "added"
                            },
                            {
                                "content": "        else:",
                                "type": "added"
                            },
                            {
                                "content": "            try:",
                                "type": "context"
                            },
                            {
                                "content": "                u\"✓\".encode(setting_encoding)",
                                "type": "deleted"
                            },
                            {
                                "content": "                u\"✓\".encode(sys.stdout.encoding)",
                                "type": "added"
                            },
                            {
                                "content": "            except UnicodeEncodeError:",
                                "type": "context"
                            },
                            {
                                "content": "                lgr.debug(\"encoding %s found to not support unicode, resetting to safe alternatives\", setting_encoding)",
                                "type": "deleted"
                            },
                            {
                                "content": "                self.OK = 'OK'   # u\"✓\"",
                                "type": "deleted"
                            },
                            {
                                "content": "                self.NOK = 'X'  # u\"✗\"",
                                "type": "deleted"
                            },
                            {
                                "content": "                self.NONE = '-'  # u\"✗\"",
                                "type": "deleted"
                            },
                            {
                                "content": "                break",
                                "type": "deleted"
                            },
                            {
                                "content": "",
                                "type": "deleted"
                            },
                            {
                                "content": "                lgr.debug(\"encoding %s does not support unicode, \"",
                                "type": "added"
                            },
                            {
                                "content": "                          \"using safe alternatives\",",
                                "type": "added"
                            },
                            {
                                "content": "                          sys.stdout.encoding)",
                                "type": "added"
                            },
                            {
                                "content": "            else:",
                                "type": "added"
                            },
                            {
                                "content": "                self.OK = u\"✓\"",
                                "type": "added"
                            },
                            {
                                "content": "                self.NOK = u\"✗\"",
                                "type": "added"
                            },
                            {
                                "content": "                self.NONE = u\"✗\"",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    def convert_field(self, value, conversion):",
                                "type": "context"
                            },
                            {
                                "content": "        #print(\"%r->%r\" % (value, conversion))",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/interface/ls.py",
                            "type": "class",
                            "name": "LsFormatter",
                            "content": "class LsFormatter(string.Formatter):\n    # condition by interactive\n    if is_interactive():\n        BLUE = ansi_colors.COLOR_SEQ % ansi_colors.BLUE\n        RED = ansi_colors.COLOR_SEQ % ansi_colors.RED\n        GREEN = ansi_colors.COLOR_SEQ % ansi_colors.GREEN\n        RESET = ansi_colors.RESET_SEQ\n        DATASET = ansi_colors.COLOR_SEQ % ansi_colors.UNDERLINE\n    else:\n        BLUE = RED = GREEN = RESET = DATASET = u\"\"\n\n    # TODO: we might want to just ignore and force utf8 while explicitly .encode()'ing output!\n    # unicode versions which look better but which blow during tests etc\n    # Those might be reset by the constructor\n    OK = u\"✓\"\n    NOK = u\"✗\"\n    NONE = u\"✗\"\n\n    def __init__(self, *args, **kwargs):\n        super(LsFormatter, self).__init__(*args, **kwargs)\n        for setting_encoding in (sys.getdefaultencoding(),\n                                 sys.stdout.encoding):\n            try:\n                u\"✓\".encode(setting_encoding)\n            except UnicodeEncodeError:\n                lgr.debug(\"encoding %s found to not support unicode, resetting to safe alternatives\", setting_encoding)\n                self.OK = 'OK'   # u\"✓\"\n                self.NOK = 'X'  # u\"✗\"\n                self.NONE = '-'  # u\"✗\"\n                break\n\n\n    def convert_field(self, value, conversion):\n        #print(\"%r->%r\" % (value, conversion))\n        if conversion == 'D':  # Date\n            if value is not None:\n                return time.strftime(u\"%Y-%m-%d/%H:%M:%S\", time.localtime(value))\n            else:\n                return u'-'\n        elif conversion == 'S':  # Human size\n            #return value\n            if value is not None:\n                return humanize.naturalsize(value)\n            else:\n                return u'-'\n        elif conversion == 'X':  # colored bool\n            chr, col = (self.OK, self.GREEN) if value else (self.NOK, self.RED)\n            return u\"%s%s%s\" % (col, chr, self.RESET)\n        elif conversion == 'N':  # colored Red - if None\n            if value is None:\n                # return \"%s✖%s\" % (self.RED, self.RESET)\n                return u\"%s%s%s\" % (self.RED, self.NONE, self.RESET)\n            return value\n        elif conversion in {'B', 'R', 'U'}:\n            return u\"%s%s%s\" % ({'B': self.BLUE, 'R': self.RED, 'U': self.DATASET}[conversion], value, self.RESET)\n\n        return super(LsFormatter, self).convert_field(value, conversion)\n\n    def format_field(self, value, format_spec):\n        # TODO: move all the \"coloring\" into formatting, so we could correctly indent\n        # given the format and only then color it up\n        # print \"> %r, %r\" % (value, format_spec)\n        return super(LsFormatter, self).format_field(value, format_spec)",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 364,
                            "end_lineno": 426,
                            "parent": null
                        },
                        {
                            "file_name": "datalad/interface/ls.py",
                            "type": "method",
                            "name": "LsFormatter.__init__",
                            "content": "class LsFormatter(string.Formatter):\n    # condition by interactive\n    if is_interactive():\n        BLUE = ansi_colors.COLOR_SEQ % ansi_colors.BLUE\n        RED = ansi_colors.COLOR_SEQ % ansi_colors.RED\n        GREEN = ansi_colors.COLOR_SEQ % ansi_colors.GREEN\n        RESET = ansi_colors.RESET_SEQ\n        DATASET = ansi_colors.COLOR_SEQ % ansi_colors.UNDERLINE\n    else:\n        BLUE = RED = GREEN = RESET = DATASET = u\"\"\n\n    # TODO: we might want to just ignore and force utf8 while explicitly .encode()'ing output!\n    # unicode versions which look better but which blow during tests etc\n    # Those might be reset by the constructor\n    OK = u\"✓\"\n    NOK = u\"✗\"\n    NONE = u\"✗\"\n\n    def __init__(self, *args, **kwargs):\n        super(LsFormatter, self).__init__(*args, **kwargs)\n        for setting_encoding in (sys.getdefaultencoding(),\n                                 sys.stdout.encoding):\n            try:\n                u\"✓\".encode(setting_encoding)\n            except UnicodeEncodeError:\n                lgr.debug(\"encoding %s found to not support unicode, resetting to safe alternatives\", setting_encoding)\n                self.OK = 'OK'   # u\"✓\"\n                self.NOK = 'X'  # u\"✗\"\n                self.NONE = '-'  # u\"✗\"\n                break\n\n\n    def convert_field(self, value, conversion):\n        #print(\"%r->%r\" % (value, conversion))\n        if conversion == 'D':  # Date\n            if value is not None:\n                return time.strftime(u\"%Y-%m-%d/%H:%M:%S\", time.localtime(value))\n            else:\n                return u'-'\n        elif conversion == 'S':  # Human size\n            #return value\n            if value is not None:\n                return humanize.naturalsize(value)\n            else:\n                return u'-'\n        elif conversion == 'X':  # colored bool\n            chr, col = (self.OK, self.GREEN) if value else (self.NOK, self.RED)\n            return u\"%s%s%s\" % (col, chr, self.RESET)\n        elif conversion == 'N':  # colored Red - if None\n            if value is None:\n                # return \"%s✖%s\" % (self.RED, self.RESET)\n                return u\"%s%s%s\" % (self.RED, self.NONE, self.RESET)\n            return value\n        elif conversion in {'B', 'R', 'U'}:\n            return u\"%s%s%s\" % ({'B': self.BLUE, 'R': self.RED, 'U': self.DATASET}[conversion], value, self.RESET)\n\n        return super(LsFormatter, self).convert_field(value, conversion)\n\n    def format_field(self, value, format_spec):\n        # TODO: move all the \"coloring\" into formatting, so we could correctly indent\n        # given the format and only then color it up\n        # print \"> %r, %r\" % (value, format_spec)\n        return super(LsFormatter, self).format_field(value, format_spec)",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 382,
                            "end_lineno": 393,
                            "parent": {
                                "file_name": "datalad/interface/ls.py",
                                "type": "class",
                                "name": "LsFormatter",
                                "content": "class LsFormatter(string.Formatter):\n    # condition by interactive\n    if is_interactive():\n        BLUE = ansi_colors.COLOR_SEQ % ansi_colors.BLUE\n        RED = ansi_colors.COLOR_SEQ % ansi_colors.RED\n        GREEN = ansi_colors.COLOR_SEQ % ansi_colors.GREEN\n        RESET = ansi_colors.RESET_SEQ\n        DATASET = ansi_colors.COLOR_SEQ % ansi_colors.UNDERLINE\n    else:\n        BLUE = RED = GREEN = RESET = DATASET = u\"\"\n\n    # TODO: we might want to just ignore and force utf8 while explicitly .encode()'ing output!\n    # unicode versions which look better but which blow during tests etc\n    # Those might be reset by the constructor\n    OK = u\"✓\"\n    NOK = u\"✗\"\n    NONE = u\"✗\"\n\n    def __init__(self, *args, **kwargs):\n        super(LsFormatter, self).__init__(*args, **kwargs)\n        for setting_encoding in (sys.getdefaultencoding(),\n                                 sys.stdout.encoding):\n            try:\n                u\"✓\".encode(setting_encoding)\n            except UnicodeEncodeError:\n                lgr.debug(\"encoding %s found to not support unicode, resetting to safe alternatives\", setting_encoding)\n                self.OK = 'OK'   # u\"✓\"\n                self.NOK = 'X'  # u\"✗\"\n                self.NONE = '-'  # u\"✗\"\n                break\n\n\n    def convert_field(self, value, conversion):\n        #print(\"%r->%r\" % (value, conversion))\n        if conversion == 'D':  # Date\n            if value is not None:\n                return time.strftime(u\"%Y-%m-%d/%H:%M:%S\", time.localtime(value))\n            else:\n                return u'-'\n        elif conversion == 'S':  # Human size\n            #return value\n            if value is not None:\n                return humanize.naturalsize(value)\n            else:\n                return u'-'\n        elif conversion == 'X':  # colored bool\n            chr, col = (self.OK, self.GREEN) if value else (self.NOK, self.RED)\n            return u\"%s%s%s\" % (col, chr, self.RESET)\n        elif conversion == 'N':  # colored Red - if None\n            if value is None:\n                # return \"%s✖%s\" % (self.RED, self.RESET)\n                return u\"%s%s%s\" % (self.RED, self.NONE, self.RESET)\n            return value\n        elif conversion in {'B', 'R', 'U'}:\n            return u\"%s%s%s\" % ({'B': self.BLUE, 'R': self.RED, 'U': self.DATASET}[conversion], value, self.RESET)\n\n        return super(LsFormatter, self).convert_field(value, conversion)\n\n    def format_field(self, value, format_spec):\n        # TODO: move all the \"coloring\" into formatting, so we could correctly indent\n        # given the format and only then color it up\n        # print \"> %r, %r\" % (value, format_spec)\n        return super(LsFormatter, self).format_field(value, format_spec)",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 364,
                                "end_lineno": 426,
                                "parent": null
                            }
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil; coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Primarily a smoke test for ls\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport logging\nimport hashlib\nimport sys\n\nfrom glob import glob\nfrom collections import Counter\nfrom mock import patch\n\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.distribution.dataset import Dataset\nfrom ...api import ls\nfrom ...utils import swallow_outputs, swallow_logs, chpwd\nfrom ...tests.utils import assert_equal, assert_in, assert_raises, assert_not_equal\nfrom ...tests.utils import use_cassette\nfrom ...tests.utils import with_tempfile\nfrom ...tests.utils import with_tree\nfrom ...tests.utils import skip_if_no_network\nfrom ..ls import LsFormatter\nfrom datalad.interface.ls import ignored, fs_traverse, _ls_json, machinesize\nfrom os.path import exists, join as opj\nfrom os.path import relpath\nfrom os import mkdir\n\nfrom datalad.downloaders.tests.utils import get_test_providers\n\n\n@skip_if_no_network\n@use_cassette('test_ls_s3')\ndef test_ls_s3():\n    url = 's3://datalad-test0-versioned/'\n    with swallow_outputs():\n        # just to skip if no credentials\n        get_test_providers(url)\n\n    with swallow_outputs() as cmo:\n        res = ls(url)\n        assert_equal(len(res), 17)  # all the entries\n        counts = Counter(map(lambda x: x.__class__.__name__, res))\n        assert_equal(counts, {'Key': 14, 'DeleteMarker': 3})\n        assert_in('Bucket info:', cmo.out)\ntest_ls_s3.tags = ['network']\n\n\n@with_tempfile\ndef test_ls_repos(toppath):\n    # smoke test pretty much\n    GitRepo(toppath + '1', create=True)\n    AnnexRepo(toppath + '2', create=True)\n    repos = glob(toppath + '*')\n    # now make that sibling directory from which we will ls later\n    mkdir(toppath)\n    def _test(*args_):\n        #print args_\n        for args in args_:\n            for recursive in [False, True]:\n                # in both cases shouldn't fail\n                with swallow_outputs() as cmo:\n                    ls(args, recursive=recursive)\n                    assert_equal(len(cmo.out.rstrip().split('\\n')), len(args))\n                    assert_in('[annex]', cmo.out)\n                    assert_in('[git]', cmo.out)\n                    assert_in('master', cmo.out)\n                    if \"bogus\" in args:\n                        assert_in('unknown', cmo.out)\n\n    _test(repos, repos + [\"/some/bogus/file\"])\n    # check from within a sibling directory with relative paths\n    with chpwd(toppath):\n        _test([relpath(x, toppath) for x in repos])\n\n\n@with_tempfile\ndef test_ls_uninstalled(path):\n    ds = Dataset(path)\n    ds.create()\n    ds.create('sub')\n    ds.uninstall('sub', check=False)\n    with swallow_outputs() as cmo:\n        ls([path], recursive=True)\n        assert_in('not installed', cmo.out)\n\n\ndef test_machinesize():\n    assert_equal(1.0, machinesize(1))\n    for key, value in {'Byte': 0, 'Bytes': 0, 'kB': 1, 'MB': 2, 'GB': 3, 'TB': 4, 'PB': 5}.items():\n        assert_equal(1.0*(1000**value), machinesize('1 ' + key))\n    assert_raises(ValueError, machinesize, 't byte')\n\n\n@with_tree(\n    tree={'dir': {'file1.txt': '123', 'file2.txt': '456'},\n          '.hidden': {'.hidden_file': '121'}})\ndef test_ignored(topdir):\n    # create annex, git repos\n    AnnexRepo(opj(topdir, 'annexdir'), create=True)\n    GitRepo(opj(topdir, 'gitdir'), create=True)\n\n    # non-git or annex should not be ignored\n    assert_equal(ignored(topdir), False)\n    # git, annex and hidden nodes should be ignored\n    for subdir in [\"annexdir\", \"gitdir\", \".hidden\"]:\n        assert_equal(ignored(opj(topdir, subdir)), True)\n    # ignore only hidden nodes(not git or annex repos) flag should work\n    assert_equal(ignored(opj(topdir, \"annexdir\"), only_hidden=True), False)\n\n\n@with_tree(\n    tree={'dir': {'.fgit': {'ab.txt': '123'},\n                  'subdir': {'file1.txt': '124', 'file2.txt': '123'},\n                  'subgit': {'fgit.txt': '123'}},\n          'topfile.txt': '123',\n          '.hidden': {'.hidden_file': '123'}})\ndef test_fs_traverse(topdir):\n    # setup temp directory tree for testing\n    annex = AnnexRepo(topdir)\n    AnnexRepo(opj(topdir, 'annexdir'), create=True)\n    GitRepo(opj(topdir, 'gitdir'), create=True)\n    GitRepo(opj(topdir, 'dir', 'subgit'), create=True)\n    annex.add(opj(topdir, 'dir'), commit=True)\n    annex.drop(opj(topdir, 'dir', 'subdir', 'file2.txt'), options=['--force'])\n\n    # traverse file system in recursive and non-recursive modes\n    for recursive in [True, False]:\n        # test fs_traverse in display mode\n        with swallow_logs(new_level=logging.INFO) as log, swallow_outputs() as cmo:\n            fs = fs_traverse(topdir, AnnexRepo(topdir), recursive=recursive, json='display')\n            if recursive:\n                # fs_traverse logs should contain all not ignored subdirectories\n                for subdir in [opj(topdir, 'dir'), opj(topdir, 'dir', 'subdir')]:\n                    assert_in('Directory: ' + subdir, log.out)\n                # fs_traverse stdout contains subdirectory\n                assert_in(('file2.txt' and 'dir'), cmo.out)\n\n            # extract info of the top-level child directory\n            child = [item for item in fs['nodes'] if item['name'] == 'dir'][0]\n            # size of dir type child in non-recursive modes should be 0 Bytes(default) as\n            # dir type child's size currently has no metadata file for traverser to pick its size from\n            # and would require a recursive traversal w/ write to child metadata file mode\n            assert_equal(child['size']['total'], {True: '6 Bytes', False: '0 Bytes'}[recursive])\n\n    for recursive in [True, False]:\n        # run fs_traverse in write to json 'file' mode\n        fs = fs_traverse(topdir, AnnexRepo(topdir), recursive=recursive, json='file')\n        # fs_traverse should return a dictionary\n        assert_equal(isinstance(fs, dict), True)\n        # not including git and annex folders\n        assert_equal([item for item in fs['nodes'] if ('gitdir' or 'annexdir') == item['name']], [])\n        # extract info of the top-level child directory\n        child = [item for item in fs['nodes'] if item['name'] == 'dir'][0]\n        # verify node type\n        assert_equal(child['type'], 'dir')\n        # same node size on running fs_traversal in recursive followed by non-recursive mode\n        # verifies child's metadata file being used to find its size\n        # running in reverse order (non-recursive followed by recursive mode) will give (0, actual size)\n        assert_equal(child['size']['total'], '6 Bytes')\n\n        # verify subdirectory traversal if run in recursive mode\n        # In current RF 'nodes' are stripped away during recursive traversal\n        # for now... later we might reincarnate them \"differently\"\n        # TODO!\n        if False:  # recursive:\n            # sub-dictionary should not include git and hidden directory info\n            assert_equal([item for item in child['nodes'] if ('subgit' or '.fgit') == item['name']], [])\n            # extract subdirectory dictionary, else fail\n            subchild = [subitem for subitem in child[\"nodes\"] if subitem['name'] == 'subdir'][0]\n            # extract info of file1.txts, else fail\n            link = [subnode for subnode in subchild[\"nodes\"] if subnode['name'] == 'file1.txt'][0]\n            # verify node's sizes and type\n            assert_equal(link['size']['total'], '3 Bytes')\n            assert_equal(link['size']['ondisk'], link['size']['total'])\n            assert_equal(link['type'], 'link')\n            # extract info of file2.txt, else fail\n            brokenlink = [subnode for subnode in subchild[\"nodes\"] if subnode['name'] == 'file2.txt'][0]\n            # verify node's sizes and type\n            assert_equal(brokenlink['type'], 'link-broken')\n            assert_equal(brokenlink['size']['ondisk'], '0 Bytes')\n            assert_equal(brokenlink['size']['total'], '3 Bytes')\n\n\n@with_tree(\n    tree={'dir': {'.fgit': {'ab.txt': '123'},\n                  'subdir': {'file1.txt': '123', 'file2.txt': '123'},\n                  'subgit': {'fgit.txt': '123'}},\n          '.hidden': {'.hidden_file': '123'}})\ndef test_ls_json(topdir):\n    annex = AnnexRepo(topdir, create=True)\n    ds = Dataset(topdir)\n    # create some file and commit it\n    with open(opj(ds.path, 'subdsfile.txt'), 'w') as f:\n        f.write('123')\n    ds.add(path='subdsfile.txt')\n    ds.save(\"Hello!\", version_tag=1)\n    # add a subdataset\n    ds.install('subds', source=topdir)\n\n    git = GitRepo(opj(topdir, 'dir', 'subgit'), create=True)                    # create git repo\n    git.add(opj(topdir, 'dir', 'subgit', 'fgit.txt'), commit=True)              # commit to git to init git repo\n    annex.add(opj(topdir, 'dir', 'subgit'), commit=True)                        # add the non-dataset git repo to annex\n    annex.add(opj(topdir, 'dir'), commit=True)                                  # add to annex (links)\n    annex.drop(opj(topdir, 'dir', 'subdir', 'file2.txt'), options=['--force'])  # broken-link\n\n    meta_dir = opj('.git', 'datalad', 'metadata')\n    meta_path = opj(topdir, meta_dir)\n\n    def get_metahash(*path):\n        return hashlib.md5(opj(*path).encode('utf-8')).hexdigest()\n\n    for all_ in [True, False]:\n        for recursive in [True, False]:\n            for state in ['file', 'delete']:\n                with swallow_logs(), swallow_outputs():\n                    ds = _ls_json(topdir, json=state, all_=all_, recursive=recursive)\n\n                # subdataset should have its json created and deleted when all=True else not\n                subds_metahash = get_metahash('/')\n                subds_metapath = opj(topdir, 'subds', meta_dir, subds_metahash)\n                assert_equal(exists(subds_metapath), (state == 'file' and recursive))\n\n                # root should have its json file created and deleted in all cases\n                ds_metahash = get_metahash('/')\n                ds_metapath = opj(meta_path, ds_metahash)\n                assert_equal(exists(ds_metapath), state == 'file')\n\n                # children should have their metadata json's created and deleted only when recursive=True\n                child_metahash = get_metahash('dir', 'subdir')\n                child_metapath = opj(meta_path, child_metahash)\n                assert_equal(exists(child_metapath), (state == 'file' and all_))\n\n                # ignored directories should not have json files created in any case\n                for subdir in [('.hidden'), ('dir', 'subgit')]:\n                    child_metahash = get_metahash(*subdir)\n                    assert_equal(exists(opj(meta_path, child_metahash)), False)\n\n                # check if its updated in its nodes sublist too. used by web-ui json. regression test\n                assert_equal(ds['nodes'][0]['size']['total'], ds['size']['total'])\n\n                # check size of subdataset\n                subds = [item for item in ds['nodes'] if item['name'] == ('subdsfile.txt' or 'subds')][0]\n                assert_equal(subds['size']['total'], '3 Bytes')\n\n                # run non-recursive dataset traversal after subdataset metadata already created\n                # to verify sub-dataset metadata being picked up from its metadata file in such cases\n                if state == 'file' and recursive and not all_:\n                    ds = _ls_json(topdir, json='file', all_=False)\n                    subds = [item for item in ds['nodes'] if item['name'] == ('subdsfile.txt' or 'subds')][0]\n                    assert_equal(subds['size']['total'], '3 Bytes')\n\n\n@with_tempfile\ndef test_ls_noarg(toppath):\n    # smoke test pretty much\n    AnnexRepo(toppath, create=True)\n\n    # this test is pointless for now and until ls() actually returns\n    # something\n    with swallow_outputs():\n        ls_out = ls(toppath)\n        with chpwd(toppath):\n            assert_equal(ls_out, ls([]))\n            assert_equal(ls_out, ls('.'))\n\n\ndef test_ls_formatter():\n    # we will use those unicode symbols only whenever both are supporting\n    # UTF-8\n    for sysenc, sysioenc, OK in (\n            ('ascii', 'ascii', 'OK'),\n            ('ascii', 'UTF-8', 'OK'),\n            ('UTF-8', 'ascii', 'OK'),\n            ('UTF-8', 'UTF-8', u\"✓\")):\n\n        # we cannot overload sys.stdout.encoding\n        class fake_stdout(object):\n            encoding = sysioenc\n            def write(self, *args):\n                pass\n\n        with patch.object(sys, 'getdefaultencoding', return_value=sysenc), \\\n            patch.object(sys, 'stdout', fake_stdout()):\n            formatter = LsFormatter()\n            assert_equal(formatter.OK, OK)\n            assert_in(OK, formatter.convert_field(True, 'X'))\n\n\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil; coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Primarily a smoke test for ls\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport logging\nimport hashlib\nimport sys\n\nfrom glob import glob\nfrom collections import Counter\nfrom mock import patch\n\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.distribution.dataset import Dataset\nfrom ...api import ls\nfrom ...utils import swallow_outputs, swallow_logs, chpwd\nfrom ...tests.utils import assert_equal, assert_in, assert_raises, assert_not_equal\nfrom ...tests.utils import use_cassette\nfrom ...tests.utils import with_tempfile\nfrom ...tests.utils import with_tree\nfrom ...tests.utils import skip_if_no_network\nfrom ..ls import LsFormatter\nfrom datalad.interface.ls import ignored, fs_traverse, _ls_json, machinesize\nfrom os.path import exists, join as opj\nfrom os.path import relpath\nfrom os import mkdir\n\nfrom datalad.downloaders.tests.utils import get_test_providers\n\n\n@skip_if_no_network\n@use_cassette('test_ls_s3')\ndef test_ls_s3():\n    url = 's3://datalad-test0-versioned/'\n    with swallow_outputs():\n        # just to skip if no credentials\n        get_test_providers(url)\n\n    with swallow_outputs() as cmo:\n        res = ls(url)\n        assert_equal(len(res), 17)  # all the entries\n        counts = Counter(map(lambda x: x.__class__.__name__, res))\n        assert_equal(counts, {'Key': 14, 'DeleteMarker': 3})\n        assert_in('Bucket info:', cmo.out)\ntest_ls_s3.tags = ['network']\n\n\n@with_tempfile\ndef test_ls_repos(toppath):\n    # smoke test pretty much\n    GitRepo(toppath + '1', create=True)\n    AnnexRepo(toppath + '2', create=True)\n    repos = glob(toppath + '*')\n    # now make that sibling directory from which we will ls later\n    mkdir(toppath)\n    def _test(*args_):\n        #print args_\n        for args in args_:\n            for recursive in [False, True]:\n                # in both cases shouldn't fail\n                with swallow_outputs() as cmo:\n                    ls(args, recursive=recursive)\n                    assert_equal(len(cmo.out.rstrip().split('\\n')), len(args))\n                    assert_in('[annex]', cmo.out)\n                    assert_in('[git]', cmo.out)\n                    assert_in('master', cmo.out)\n                    if \"bogus\" in args:\n                        assert_in('unknown', cmo.out)\n\n    _test(repos, repos + [\"/some/bogus/file\"])\n    # check from within a sibling directory with relative paths\n    with chpwd(toppath):\n        _test([relpath(x, toppath) for x in repos])\n\n\n@with_tempfile\ndef test_ls_uninstalled(path):\n    ds = Dataset(path)\n    ds.create()\n    ds.create('sub')\n    ds.uninstall('sub', check=False)\n    with swallow_outputs() as cmo:\n        ls([path], recursive=True)\n        assert_in('not installed', cmo.out)\n\n\ndef test_machinesize():\n    assert_equal(1.0, machinesize(1))\n    for key, value in {'Byte': 0, 'Bytes': 0, 'kB': 1, 'MB': 2, 'GB': 3, 'TB': 4, 'PB': 5}.items():\n        assert_equal(1.0*(1000**value), machinesize('1 ' + key))\n    assert_raises(ValueError, machinesize, 't byte')\n\n\n@with_tree(\n    tree={'dir': {'file1.txt': '123', 'file2.txt': '456'},\n          '.hidden': {'.hidden_file': '121'}})\ndef test_ignored(topdir):\n    # create annex, git repos\n    AnnexRepo(opj(topdir, 'annexdir'), create=True)\n    GitRepo(opj(topdir, 'gitdir'), create=True)\n\n    # non-git or annex should not be ignored\n    assert_equal(ignored(topdir), False)\n    # git, annex and hidden nodes should be ignored\n    for subdir in [\"annexdir\", \"gitdir\", \".hidden\"]:\n        assert_equal(ignored(opj(topdir, subdir)), True)\n    # ignore only hidden nodes(not git or annex repos) flag should work\n    assert_equal(ignored(opj(topdir, \"annexdir\"), only_hidden=True), False)\n\n\n@with_tree(\n    tree={'dir': {'.fgit': {'ab.txt': '123'},\n                  'subdir': {'file1.txt': '124', 'file2.txt': '123'},\n                  'subgit': {'fgit.txt': '123'}},\n          'topfile.txt': '123',\n          '.hidden': {'.hidden_file': '123'}})\ndef test_fs_traverse(topdir):\n    # setup temp directory tree for testing\n    annex = AnnexRepo(topdir)\n    AnnexRepo(opj(topdir, 'annexdir'), create=True)\n    GitRepo(opj(topdir, 'gitdir'), create=True)\n    GitRepo(opj(topdir, 'dir', 'subgit'), create=True)\n    annex.add(opj(topdir, 'dir'), commit=True)\n    annex.drop(opj(topdir, 'dir', 'subdir', 'file2.txt'), options=['--force'])\n\n    # traverse file system in recursive and non-recursive modes\n    for recursive in [True, False]:\n        # test fs_traverse in display mode\n        with swallow_logs(new_level=logging.INFO) as log, swallow_outputs() as cmo:\n            fs = fs_traverse(topdir, AnnexRepo(topdir), recursive=recursive, json='display')\n            if recursive:\n                # fs_traverse logs should contain all not ignored subdirectories\n                for subdir in [opj(topdir, 'dir'), opj(topdir, 'dir', 'subdir')]:\n                    assert_in('Directory: ' + subdir, log.out)\n                # fs_traverse stdout contains subdirectory\n                assert_in(('file2.txt' and 'dir'), cmo.out)\n\n            # extract info of the top-level child directory\n            child = [item for item in fs['nodes'] if item['name'] == 'dir'][0]\n            # size of dir type child in non-recursive modes should be 0 Bytes(default) as\n            # dir type child's size currently has no metadata file for traverser to pick its size from\n            # and would require a recursive traversal w/ write to child metadata file mode\n            assert_equal(child['size']['total'], {True: '6 Bytes', False: '0 Bytes'}[recursive])\n\n    for recursive in [True, False]:\n        # run fs_traverse in write to json 'file' mode\n        fs = fs_traverse(topdir, AnnexRepo(topdir), recursive=recursive, json='file')\n        # fs_traverse should return a dictionary\n        assert_equal(isinstance(fs, dict), True)\n        # not including git and annex folders\n        assert_equal([item for item in fs['nodes'] if ('gitdir' or 'annexdir') == item['name']], [])\n        # extract info of the top-level child directory\n        child = [item for item in fs['nodes'] if item['name'] == 'dir'][0]\n        # verify node type\n        assert_equal(child['type'], 'dir')\n        # same node size on running fs_traversal in recursive followed by non-recursive mode\n        # verifies child's metadata file being used to find its size\n        # running in reverse order (non-recursive followed by recursive mode) will give (0, actual size)\n        assert_equal(child['size']['total'], '6 Bytes')\n\n        # verify subdirectory traversal if run in recursive mode\n        # In current RF 'nodes' are stripped away during recursive traversal\n        # for now... later we might reincarnate them \"differently\"\n        # TODO!\n        if False:  # recursive:\n            # sub-dictionary should not include git and hidden directory info\n            assert_equal([item for item in child['nodes'] if ('subgit' or '.fgit') == item['name']], [])\n            # extract subdirectory dictionary, else fail\n            subchild = [subitem for subitem in child[\"nodes\"] if subitem['name'] == 'subdir'][0]\n            # extract info of file1.txts, else fail\n            link = [subnode for subnode in subchild[\"nodes\"] if subnode['name'] == 'file1.txt'][0]\n            # verify node's sizes and type\n            assert_equal(link['size']['total'], '3 Bytes')\n            assert_equal(link['size']['ondisk'], link['size']['total'])\n            assert_equal(link['type'], 'link')\n            # extract info of file2.txt, else fail\n            brokenlink = [subnode for subnode in subchild[\"nodes\"] if subnode['name'] == 'file2.txt'][0]\n            # verify node's sizes and type\n            assert_equal(brokenlink['type'], 'link-broken')\n            assert_equal(brokenlink['size']['ondisk'], '0 Bytes')\n            assert_equal(brokenlink['size']['total'], '3 Bytes')\n\n\n@with_tree(\n    tree={'dir': {'.fgit': {'ab.txt': '123'},\n                  'subdir': {'file1.txt': '123', 'file2.txt': '123'},\n                  'subgit': {'fgit.txt': '123'}},\n          '.hidden': {'.hidden_file': '123'}})\ndef test_ls_json(topdir):\n    annex = AnnexRepo(topdir, create=True)\n    ds = Dataset(topdir)\n    # create some file and commit it\n    with open(opj(ds.path, 'subdsfile.txt'), 'w') as f:\n        f.write('123')\n    ds.add(path='subdsfile.txt')\n    ds.save(\"Hello!\", version_tag=1)\n    # add a subdataset\n    ds.install('subds', source=topdir)\n\n    git = GitRepo(opj(topdir, 'dir', 'subgit'), create=True)                    # create git repo\n    git.add(opj(topdir, 'dir', 'subgit', 'fgit.txt'), commit=True)              # commit to git to init git repo\n    annex.add(opj(topdir, 'dir', 'subgit'), commit=True)                        # add the non-dataset git repo to annex\n    annex.add(opj(topdir, 'dir'), commit=True)                                  # add to annex (links)\n    annex.drop(opj(topdir, 'dir', 'subdir', 'file2.txt'), options=['--force'])  # broken-link\n\n    meta_dir = opj('.git', 'datalad', 'metadata')\n    meta_path = opj(topdir, meta_dir)\n\n    def get_metahash(*path):\n        return hashlib.md5(opj(*path).encode('utf-8')).hexdigest()\n\n    for all_ in [True, False]:\n        for recursive in [True, False]:\n            for state in ['file', 'delete']:\n                with swallow_logs(), swallow_outputs():\n                    ds = _ls_json(topdir, json=state, all_=all_, recursive=recursive)\n\n                # subdataset should have its json created and deleted when all=True else not\n                subds_metahash = get_metahash('/')\n                subds_metapath = opj(topdir, 'subds', meta_dir, subds_metahash)\n                assert_equal(exists(subds_metapath), (state == 'file' and recursive))\n\n                # root should have its json file created and deleted in all cases\n                ds_metahash = get_metahash('/')\n                ds_metapath = opj(meta_path, ds_metahash)\n                assert_equal(exists(ds_metapath), state == 'file')\n\n                # children should have their metadata json's created and deleted only when recursive=True\n                child_metahash = get_metahash('dir', 'subdir')\n                child_metapath = opj(meta_path, child_metahash)\n                assert_equal(exists(child_metapath), (state == 'file' and all_))\n\n                # ignored directories should not have json files created in any case\n                for subdir in [('.hidden'), ('dir', 'subgit')]:\n                    child_metahash = get_metahash(*subdir)\n                    assert_equal(exists(opj(meta_path, child_metahash)), False)\n\n                # check if its updated in its nodes sublist too. used by web-ui json. regression test\n                assert_equal(ds['nodes'][0]['size']['total'], ds['size']['total'])\n\n                # check size of subdataset\n                subds = [item for item in ds['nodes'] if item['name'] == ('subdsfile.txt' or 'subds')][0]\n                assert_equal(subds['size']['total'], '3 Bytes')\n\n                # run non-recursive dataset traversal after subdataset metadata already created\n                # to verify sub-dataset metadata being picked up from its metadata file in such cases\n                if state == 'file' and recursive and not all_:\n                    ds = _ls_json(topdir, json='file', all_=False)\n                    subds = [item for item in ds['nodes'] if item['name'] == ('subdsfile.txt' or 'subds')][0]\n                    assert_equal(subds['size']['total'], '3 Bytes')\n\n\n@with_tempfile\ndef test_ls_noarg(toppath):\n    # smoke test pretty much\n    AnnexRepo(toppath, create=True)\n\n    # this test is pointless for now and until ls() actually returns\n    # something\n    with swallow_outputs():\n        ls_out = ls(toppath)\n        with chpwd(toppath):\n            assert_equal(ls_out, ls([]))\n            assert_equal(ls_out, ls('.'))\n\n\ndef test_ls_formatter():\n    # we will use unicode symbols only when sys.stdio supports UTF-8\n    for sysioenc, OK in [(None, \"OK\"),\n                         ('ascii', 'OK'),\n                         ('UTF-8', u\"✓\")]:\n\n        # we cannot overload sys.stdout.encoding\n        class fake_stdout(object):\n            encoding = sysioenc\n            def write(self, *args):\n                pass\n\n        with patch.object(sys, 'stdout', fake_stdout()):\n            formatter = LsFormatter()\n            assert_equal(formatter.OK, OK)\n            assert_in(OK, formatter.convert_field(True, 'X'))\n",
            "header": {
                "file": {
                    "path": "datalad/interface/tests/test_ls.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "479fbb111",
                "new_commit_hash": "f04e42604",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/interface/tests/test_ls.py"
            },
            "plus_file": {
                "path": "b/datalad/interface/tests/test_ls.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 276,
                            "length": 13
                        },
                        "new_range": {
                            "start": 276,
                            "length": 10
                        },
                        "section": "def test_ls_noarg(toppath):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def test_ls_formatter():",
                                "type": "context"
                            },
                            {
                                "content": "    # we will use those unicode symbols only whenever both are supporting",
                                "type": "deleted"
                            },
                            {
                                "content": "    # UTF-8",
                                "type": "deleted"
                            },
                            {
                                "content": "    for sysenc, sysioenc, OK in (",
                                "type": "deleted"
                            },
                            {
                                "content": "            ('ascii', 'ascii', 'OK'),",
                                "type": "deleted"
                            },
                            {
                                "content": "            ('ascii', 'UTF-8', 'OK'),",
                                "type": "deleted"
                            },
                            {
                                "content": "            ('UTF-8', 'ascii', 'OK'),",
                                "type": "deleted"
                            },
                            {
                                "content": "            ('UTF-8', 'UTF-8', u\"✓\")):",
                                "type": "deleted"
                            },
                            {
                                "content": "    # we will use unicode symbols only when sys.stdio supports UTF-8",
                                "type": "added"
                            },
                            {
                                "content": "    for sysioenc, OK in [(None, \"OK\"),",
                                "type": "added"
                            },
                            {
                                "content": "                         ('ascii', 'OK'),",
                                "type": "added"
                            },
                            {
                                "content": "                         ('UTF-8', u\"✓\")]:",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "        # we cannot overload sys.stdout.encoding",
                                "type": "context"
                            },
                            {
                                "content": "        class fake_stdout(object):",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/interface/tests/test_ls.py",
                            "type": "function",
                            "name": "test_ls_formatter",
                            "content": "def test_ls_formatter():\n    # we will use those unicode symbols only whenever both are supporting\n    # UTF-8\n    for sysenc, sysioenc, OK in (\n            ('ascii', 'ascii', 'OK'),\n            ('ascii', 'UTF-8', 'OK'),\n            ('UTF-8', 'ascii', 'OK'),\n            ('UTF-8', 'UTF-8', u\"✓\")):\n\n        # we cannot overload sys.stdout.encoding\n        class fake_stdout(object):\n            encoding = sysioenc\n            def write(self, *args):\n                pass\n\n        with patch.object(sys, 'getdefaultencoding', return_value=sysenc), \\\n            patch.object(sys, 'stdout', fake_stdout()):\n            formatter = LsFormatter()\n            assert_equal(formatter.OK, OK)\n            assert_in(OK, formatter.convert_field(True, 'X'))",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 278,
                            "end_lineno": 297,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 290,
                            "length": 10
                        },
                        "new_range": {
                            "start": 287,
                            "length": 7
                        },
                        "section": "def test_ls_formatter():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            def write(self, *args):",
                                "type": "context"
                            },
                            {
                                "content": "                pass",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "        with patch.object(sys, 'getdefaultencoding', return_value=sysenc), \\",
                                "type": "deleted"
                            },
                            {
                                "content": "            patch.object(sys, 'stdout', fake_stdout()):",
                                "type": "deleted"
                            },
                            {
                                "content": "        with patch.object(sys, 'stdout', fake_stdout()):",
                                "type": "added"
                            },
                            {
                                "content": "            formatter = LsFormatter()",
                                "type": "context"
                            },
                            {
                                "content": "            assert_equal(formatter.OK, OK)",
                                "type": "context"
                            },
                            {
                                "content": "            assert_in(OK, formatter.convert_field(True, 'X'))",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "deleted"
                            },
                            {
                                "content": "",
                                "type": "deleted"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/interface/tests/test_ls.py",
                            "type": "function",
                            "name": "test_ls_formatter",
                            "content": "def test_ls_formatter():\n    # we will use those unicode symbols only whenever both are supporting\n    # UTF-8\n    for sysenc, sysioenc, OK in (\n            ('ascii', 'ascii', 'OK'),\n            ('ascii', 'UTF-8', 'OK'),\n            ('UTF-8', 'ascii', 'OK'),\n            ('UTF-8', 'UTF-8', u\"✓\")):\n\n        # we cannot overload sys.stdout.encoding\n        class fake_stdout(object):\n            encoding = sysioenc\n            def write(self, *args):\n                pass\n\n        with patch.object(sys, 'getdefaultencoding', return_value=sysenc), \\\n            patch.object(sys, 'stdout', fake_stdout()):\n            formatter = LsFormatter()\n            assert_equal(formatter.OK, OK)\n            assert_in(OK, formatter.convert_field(True, 'X'))",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 278,
                            "end_lineno": 297,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "77ce648eb80246d9aadb5be1596a98859f4b05cb^",
    "new_commit_hash": "77ce648eb80246d9aadb5be1596a98859f4b05cb",
    "commit_message": "BF: ls -- Don't use sys.getdefaultencoding for unicode check\n\nFor Python 2, we should expect sys.getdefaultencoding to return\n'ascii' regardless of what encoding the output device supports (e.g.,\nsee this stackoverflow answer [*]), so we shouldn't base our encoding\ndecision off of sys.getdefaultencoding.\n\nAlso, avoid a type error by explicitly handling a sys.stdout.encoding\nvalue of None, which happens with Python 2 when stdout is piped.\nWe're disabling unicode in this case, which is consistent with the\nnext patch that will disable unicode if stdout isn't a tty.\n\n[*] https://stackoverflow.com/questions/15530635/why-is-sys-getdefaultencoding-different-from-sys-stdout-encoding-and-how-does#33726891",
    "commit_date": "2017-12-13T23:00:20-05:00",
    "metadata": {}
}
{
    "repo_name": "datalad",
    "new_commit_hash": "77ce648eb80246d9aadb5be1596a98859f4b05cb",
    "test_file_codes": [
        "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil; coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Primarily a smoke test for ls\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport logging\nimport hashlib\nimport sys\n\nfrom glob import glob\nfrom collections import Counter\nfrom mock import patch\n\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.distribution.dataset import Dataset\nfrom datalad.api import ls\nfrom datalad.utils import swallow_outputs, swallow_logs, chpwd\nfrom datalad.tests.utils import assert_equal, assert_in, assert_raises, assert_not_equal\nfrom datalad.tests.utils import use_cassette\nfrom datalad.tests.utils import with_tempfile\nfrom datalad.tests.utils import with_tree\nfrom datalad.tests.utils import skip_if_no_network\nfrom datalad.interface.ls import LsFormatter\nfrom datalad.interface.ls import ignored, fs_traverse, _ls_json, machinesize\nfrom os.path import exists, join as opj\nfrom os.path import relpath\nfrom os import mkdir\n\nfrom datalad.downloaders.tests.utils import get_test_providers\n\n\n@skip_if_no_network\n@use_cassette('test_ls_s3')\ndef test_ls_s3():\n    url = 's3://datalad-test0-versioned/'\n    with swallow_outputs():\n        # just to skip if no credentials\n        get_test_providers(url)\n\n    with swallow_outputs() as cmo:\n        res = ls(url)\n        assert_equal(len(res), 17)  # all the entries\n        counts = Counter(map(lambda x: x.__class__.__name__, res))\n        assert_equal(counts, {'Key': 14, 'DeleteMarker': 3})\n        assert_in('Bucket info:', cmo.out)\ntest_ls_s3.tags = ['network']\n\n\n@with_tempfile\ndef test_ls_repos(toppath):\n    # smoke test pretty much\n    GitRepo(toppath + '1', create=True)\n    AnnexRepo(toppath + '2', create=True)\n    repos = glob(toppath + '*')\n    # now make that sibling directory from which we will ls later\n    mkdir(toppath)\n    def _test(*args_):\n        #print args_\n        for args in args_:\n            for recursive in [False, True]:\n                # in both cases shouldn't fail\n                with swallow_outputs() as cmo:\n                    ls(args, recursive=recursive)\n                    assert_equal(len(cmo.out.rstrip().split('\\n')), len(args))\n                    assert_in('[annex]', cmo.out)\n                    assert_in('[git]', cmo.out)\n                    assert_in('master', cmo.out)\n                    if \"bogus\" in args:\n                        assert_in('unknown', cmo.out)\n\n    _test(repos, repos + [\"/some/bogus/file\"])\n    # check from within a sibling directory with relative paths\n    with chpwd(toppath):\n        _test([relpath(x, toppath) for x in repos])\n\n\n@with_tempfile\ndef test_ls_uninstalled(path):\n    ds = Dataset(path)\n    ds.create()\n    ds.create('sub')\n    ds.uninstall('sub', check=False)\n    with swallow_outputs() as cmo:\n        ls([path], recursive=True)\n        assert_in('not installed', cmo.out)\n\n\ndef test_machinesize():\n    assert_equal(1.0, machinesize(1))\n    for key, value in {'Byte': 0, 'Bytes': 0, 'kB': 1, 'MB': 2, 'GB': 3, 'TB': 4, 'PB': 5}.items():\n        assert_equal(1.0*(1000**value), machinesize('1 ' + key))\n    assert_raises(ValueError, machinesize, 't byte')\n\n\n@with_tree(\n    tree={'dir': {'file1.txt': '123', 'file2.txt': '456'},\n          '.hidden': {'.hidden_file': '121'}})\ndef test_ignored(topdir):\n    # create annex, git repos\n    AnnexRepo(opj(topdir, 'annexdir'), create=True)\n    GitRepo(opj(topdir, 'gitdir'), create=True)\n\n    # non-git or annex should not be ignored\n    assert_equal(ignored(topdir), False)\n    # git, annex and hidden nodes should be ignored\n    for subdir in [\"annexdir\", \"gitdir\", \".hidden\"]:\n        assert_equal(ignored(opj(topdir, subdir)), True)\n    # ignore only hidden nodes(not git or annex repos) flag should work\n    assert_equal(ignored(opj(topdir, \"annexdir\"), only_hidden=True), False)\n\n\n@with_tree(\n    tree={'dir': {'.fgit': {'ab.txt': '123'},\n                  'subdir': {'file1.txt': '124', 'file2.txt': '123'},\n                  'subgit': {'fgit.txt': '123'}},\n          'topfile.txt': '123',\n          '.hidden': {'.hidden_file': '123'}})\ndef test_fs_traverse(topdir):\n    # setup temp directory tree for testing\n    annex = AnnexRepo(topdir)\n    AnnexRepo(opj(topdir, 'annexdir'), create=True)\n    GitRepo(opj(topdir, 'gitdir'), create=True)\n    GitRepo(opj(topdir, 'dir', 'subgit'), create=True)\n    annex.add(opj(topdir, 'dir'), commit=True)\n    annex.drop(opj(topdir, 'dir', 'subdir', 'file2.txt'), options=['--force'])\n\n    # traverse file system in recursive and non-recursive modes\n    for recursive in [True, False]:\n        # test fs_traverse in display mode\n        with swallow_logs(new_level=logging.INFO) as log, swallow_outputs() as cmo:\n            fs = fs_traverse(topdir, AnnexRepo(topdir), recursive=recursive, json='display')\n            if recursive:\n                # fs_traverse logs should contain all not ignored subdirectories\n                for subdir in [opj(topdir, 'dir'), opj(topdir, 'dir', 'subdir')]:\n                    assert_in('Directory: ' + subdir, log.out)\n                # fs_traverse stdout contains subdirectory\n                assert_in(('file2.txt' and 'dir'), cmo.out)\n\n            # extract info of the top-level child directory\n            child = [item for item in fs['nodes'] if item['name'] == 'dir'][0]\n            # size of dir type child in non-recursive modes should be 0 Bytes(default) as\n            # dir type child's size currently has no metadata file for traverser to pick its size from\n            # and would require a recursive traversal w/ write to child metadata file mode\n            assert_equal(child['size']['total'], {True: '6 Bytes', False: '0 Bytes'}[recursive])\n\n    for recursive in [True, False]:\n        # run fs_traverse in write to json 'file' mode\n        fs = fs_traverse(topdir, AnnexRepo(topdir), recursive=recursive, json='file')\n        # fs_traverse should return a dictionary\n        assert_equal(isinstance(fs, dict), True)\n        # not including git and annex folders\n        assert_equal([item for item in fs['nodes'] if ('gitdir' or 'annexdir') == item['name']], [])\n        # extract info of the top-level child directory\n        child = [item for item in fs['nodes'] if item['name'] == 'dir'][0]\n        # verify node type\n        assert_equal(child['type'], 'dir')\n        # same node size on running fs_traversal in recursive followed by non-recursive mode\n        # verifies child's metadata file being used to find its size\n        # running in reverse order (non-recursive followed by recursive mode) will give (0, actual size)\n        assert_equal(child['size']['total'], '6 Bytes')\n\n        # verify subdirectory traversal if run in recursive mode\n        # In current RF 'nodes' are stripped away during recursive traversal\n        # for now... later we might reincarnate them \"differently\"\n        # TODO!\n        if False:  # recursive:\n            # sub-dictionary should not include git and hidden directory info\n            assert_equal([item for item in child['nodes'] if ('subgit' or '.fgit') == item['name']], [])\n            # extract subdirectory dictionary, else fail\n            subchild = [subitem for subitem in child[\"nodes\"] if subitem['name'] == 'subdir'][0]\n            # extract info of file1.txts, else fail\n            link = [subnode for subnode in subchild[\"nodes\"] if subnode['name'] == 'file1.txt'][0]\n            # verify node's sizes and type\n            assert_equal(link['size']['total'], '3 Bytes')\n            assert_equal(link['size']['ondisk'], link['size']['total'])\n            assert_equal(link['type'], 'link')\n            # extract info of file2.txt, else fail\n            brokenlink = [subnode for subnode in subchild[\"nodes\"] if subnode['name'] == 'file2.txt'][0]\n            # verify node's sizes and type\n            assert_equal(brokenlink['type'], 'link-broken')\n            assert_equal(brokenlink['size']['ondisk'], '0 Bytes')\n            assert_equal(brokenlink['size']['total'], '3 Bytes')\n\n\n@with_tree(\n    tree={'dir': {'.fgit': {'ab.txt': '123'},\n                  'subdir': {'file1.txt': '123', 'file2.txt': '123'},\n                  'subgit': {'fgit.txt': '123'}},\n          '.hidden': {'.hidden_file': '123'}})\ndef test_ls_json(topdir):\n    annex = AnnexRepo(topdir, create=True)\n    ds = Dataset(topdir)\n    # create some file and commit it\n    with open(opj(ds.path, 'subdsfile.txt'), 'w') as f:\n        f.write('123')\n    ds.add(path='subdsfile.txt')\n    ds.save(\"Hello!\", version_tag=1)\n    # add a subdataset\n    ds.install('subds', source=topdir)\n\n    git = GitRepo(opj(topdir, 'dir', 'subgit'), create=True)                    # create git repo\n    git.add(opj(topdir, 'dir', 'subgit', 'fgit.txt'), commit=True)              # commit to git to init git repo\n    annex.add(opj(topdir, 'dir', 'subgit'), commit=True)                        # add the non-dataset git repo to annex\n    annex.add(opj(topdir, 'dir'), commit=True)                                  # add to annex (links)\n    annex.drop(opj(topdir, 'dir', 'subdir', 'file2.txt'), options=['--force'])  # broken-link\n\n    meta_dir = opj('.git', 'datalad', 'metadata')\n    meta_path = opj(topdir, meta_dir)\n\n    def get_metahash(*path):\n        return hashlib.md5(opj(*path).encode('utf-8')).hexdigest()\n\n    for all_ in [True, False]:\n        for recursive in [True, False]:\n            for state in ['file', 'delete']:\n                with swallow_logs(), swallow_outputs():\n                    ds = _ls_json(topdir, json=state, all_=all_, recursive=recursive)\n\n                # subdataset should have its json created and deleted when all=True else not\n                subds_metahash = get_metahash('/')\n                subds_metapath = opj(topdir, 'subds', meta_dir, subds_metahash)\n                assert_equal(exists(subds_metapath), (state == 'file' and recursive))\n\n                # root should have its json file created and deleted in all cases\n                ds_metahash = get_metahash('/')\n                ds_metapath = opj(meta_path, ds_metahash)\n                assert_equal(exists(ds_metapath), state == 'file')\n\n                # children should have their metadata json's created and deleted only when recursive=True\n                child_metahash = get_metahash('dir', 'subdir')\n                child_metapath = opj(meta_path, child_metahash)\n                assert_equal(exists(child_metapath), (state == 'file' and all_))\n\n                # ignored directories should not have json files created in any case\n                for subdir in [('.hidden'), ('dir', 'subgit')]:\n                    child_metahash = get_metahash(*subdir)\n                    assert_equal(exists(opj(meta_path, child_metahash)), False)\n\n                # check if its updated in its nodes sublist too. used by web-ui json. regression test\n                assert_equal(ds['nodes'][0]['size']['total'], ds['size']['total'])\n\n                # check size of subdataset\n                subds = [item for item in ds['nodes'] if item['name'] == ('subdsfile.txt' or 'subds')][0]\n                assert_equal(subds['size']['total'], '3 Bytes')\n\n                # run non-recursive dataset traversal after subdataset metadata already created\n                # to verify sub-dataset metadata being picked up from its metadata file in such cases\n                if state == 'file' and recursive and not all_:\n                    ds = _ls_json(topdir, json='file', all_=False)\n                    subds = [item for item in ds['nodes'] if item['name'] == ('subdsfile.txt' or 'subds')][0]\n                    assert_equal(subds['size']['total'], '3 Bytes')\n\n\n@with_tempfile\ndef test_ls_noarg(toppath):\n    # smoke test pretty much\n    AnnexRepo(toppath, create=True)\n\n    # this test is pointless for now and until ls() actually returns\n    # something\n    with swallow_outputs():\n        ls_out = ls(toppath)\n        with chpwd(toppath):\n            assert_equal(ls_out, ls([]))\n            assert_equal(ls_out, ls('.'))\n\n\ndef test_ls_formatter():\n    # we will use unicode symbols only when sys.stdio supports UTF-8\n    for sysioenc, OK in [(None, \"OK\"),\n                         ('ascii', 'OK'),\n                         ('UTF-8', u\"✓\")]:\n\n        # we cannot overload sys.stdout.encoding\n        class fake_stdout(object):\n            encoding = sysioenc\n            def write(self, *args):\n                pass\n\n        with patch.object(sys, 'stdout', fake_stdout()):\n            formatter = LsFormatter()\n            assert_equal(formatter.OK, OK)\n            assert_in(OK, formatter.convert_field(True, 'X'))\n",
        "import pytest\nimport tempfile\nimport os\n\n\n@pytest.fixture\ndef path():\n    with tempfile.TemporaryDirectory() as tmpdir:\n        yield tmpdir\n\n\n@pytest.fixture(autouse=True)\ndef setup_git_config():\n    with tempfile.TemporaryDirectory() as home:\n        old_home = os.environ.get(\"HOME\")\n        os.environ[\"HOME\"] = home\n        os.system('git config --global user.name \"DataLad Tester\"')\n        os.system('git config --global user.email \"test@example.com\"')\n        yield\n        if old_home:\n            os.environ[\"HOME\"] = old_home\n"
    ],
    "test_file_names": [
        "test_1.py",
        "conftest.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Datalad installation attempts...\nVerifying installation...\n❌ Verification failed\nPython 3.9 installation failed, trying Python 3.7...\nVerifying installation...\nDatalad version: 0.9.1.dev155\n✅ Installation successful!\nSuccessfully installed Datalad using Python 3.7\n",
    "setup_res_stderr": "Using CPython 3.9.21\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 25 packages in 91ms\nInstalled 25 packages in 6.62s\n + coverage==7.6.10\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + grabbit==0.2.6\n + importlib-resources==6.4.5\n + iniconfig==2.0.0\n + nibabel==5.3.2\n + num2words==0.5.14\n + numpy==2.0.2\n + packaging==24.2\n + pandas==2.2.3\n + patsy==1.0.1\n + pluggy==1.5.0\n + pybids==0.6.5\n + pytest==8.3.4\n + pytest-cov==6.0.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.13.1\n + setuptools==75.6.0\n + six==1.17.0\n + tomli==2.2.1\n + typing-extensions==4.12.2\n + tzdata==2024.2\n + zipp==3.21.0\nerror: The build backend returned an error\n  Caused by: Call to `setuptools.build_meta:__legacy__.build_editable` failed (exit status: 1)\n\n[stderr]\nTraceback (most recent call last):\n  File \"<string>\", line 14, in <module>\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpJJORgt/lib/python3.9/site-packages/setuptools/build_meta.py\", line 483, in get_requires_for_build_editable\n    return self.get_requires_for_build_wheel(config_settings)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpJJORgt/lib/python3.9/site-packages/setuptools/build_meta.py\", line 334, in get_requires_for_build_wheel\n    return self._get_build_requires(config_settings, requirements=[])\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpJJORgt/lib/python3.9/site-packages/setuptools/build_meta.py\", line 304, in _get_build_requires\n    self.run_setup()\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpJJORgt/lib/python3.9/site-packages/setuptools/build_meta.py\", line 522, in run_setup\n    super().run_setup(setup_script=setup_script)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpJJORgt/lib/python3.9/site-packages/setuptools/build_meta.py\", line 320, in run_setup\n    exec(code, locals())\n  File \"<string>\", line 48, in <module>\nAttributeError: module 'platform' has no attribute 'dist'\n\nhint: This usually indicates a problem with the package or the build environment.\nerror: The build backend returned an error\n  Caused by: Call to `setuptools.build_meta:__legacy__.build_editable` failed (exit status: 1)\n\n[stderr]\nTraceback (most recent call last):\n  File \"<string>\", line 14, in <module>\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpF8v3PV/lib/python3.9/site-packages/setuptools/build_meta.py\", line 483, in get_requires_for_build_editable\n    return self.get_requires_for_build_wheel(config_settings)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpF8v3PV/lib/python3.9/site-packages/setuptools/build_meta.py\", line 334, in get_requires_for_build_wheel\n    return self._get_build_requires(config_settings, requirements=[])\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpF8v3PV/lib/python3.9/site-packages/setuptools/build_meta.py\", line 304, in _get_build_requires\n    self.run_setup()\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpF8v3PV/lib/python3.9/site-packages/setuptools/build_meta.py\", line 522, in run_setup\n    super().run_setup(setup_script=setup_script)\n  File \"/home/gcpuser/.cache/uv/builds-v0/.tmpF8v3PV/lib/python3.9/site-packages/setuptools/build_meta.py\", line 320, in run_setup\n    exec(code, locals())\n  File \"<string>\", line 48, in <module>\nAttributeError: module 'platform' has no attribute 'dist'\n\nhint: This usually indicates a problem with the package or the build environment.\nTraceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/datalad/__init__.py\", line 17, in <module>\n    from datalad.cmd import GitRunner\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/datalad/cmd.py\", line 34, in <module>\n    from .utils import on_windows\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/datalad/utils.py\", line 24, in <module>\n    import wrapt\nModuleNotFoundError: No module named 'wrapt'\nUsing CPython 3.7.9\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 26 packages in 313ms\nInstalled 26 packages in 5.19s\n + bids==0.0\n + coverage==7.2.7\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + fasteners==0.19\n + grabbit==0.2.6\n + importlib-metadata==6.7.0\n + iniconfig==2.0.0\n + nibabel==4.0.2\n + num2words==0.5.14\n + numpy==1.21.6\n + packaging==24.0\n + pandas==1.3.5\n + patsy==1.0.1\n + pluggy==1.2.0\n + pybids==0.6.5\n + pytest==7.4.4\n + pytest-cov==4.1.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.7.3\n + setuptools==68.0.0\n + six==1.17.0\n + tomli==2.0.1\n + typing-extensions==4.7.1\n + zipp==3.15.0\nResolved 77 packages in 1.53s\nPrepared 1 package in 1.18s\nInstalled 71 packages in 75ms\n + appdirs==1.4.4\n + attrs==24.2.0\n + automat==22.10.0\n + beautifulsoup4==4.12.3\n + boto==2.49.0\n + cachetools==5.5.0\n + certifi==2024.12.14\n + cffi==1.15.1\n + charset-normalizer==3.4.1\n + constantly==15.1.0\n + cryptography==44.0.0\n + cssselect==1.2.0\n + datalad==0.9.1.dev1 (from file:///home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb)\n + deprecated==1.2.15\n + filelock==3.12.2\n + frozendict==2.4.6\n + gitdb==4.0.12\n + gitpython==3.1.43\n + httpretty==1.1.4\n + humanize==4.6.0\n + hyperlink==21.0.0\n + idna==3.10\n + importlib-resources==5.12.0\n + incremental==22.10.0\n + iso8601==2.1.0\n + itemadapter==0.8.0\n + itemloaders==1.1.0\n + jaraco-classes==3.2.3\n + jeepney==0.8.0\n + jmespath==1.0.1\n + jsmin==3.0.1\n + keyring==24.1.1\n + keyrings-alt==4.2.0\n + lxml==5.3.0\n + mock==5.1.0\n + more-itertools==9.1.0\n + msgpack-python==0.5.6\n + multidict==6.0.5\n + nose==1.3.7\n + parsel==1.8.1\n + patool==1.12\n + protego==0.3.0\n + pyasn1==0.5.1\n + pyasn1-modules==0.3.0\n + pycparser==2.21\n + pydispatcher==2.0.7\n + pygithub==2.3.0\n + pyjwt==2.8.0\n + pyld==2.0.4\n + pynacl==1.5.0\n + pyopenssl==24.3.0\n + pyyaml==6.0.1\n + queuelib==1.6.2\n + requests==2.31.0\n + requests-file==2.1.0\n + requests-ftp==0.3.1\n + scrapy==2.9.0\n + secretstorage==3.3.3\n + service-identity==21.1.0\n + simplejson==3.19.3\n + smmap==5.0.2\n + soupsieve==2.4.1\n + tldextract==4.0.0\n + tqdm==4.67.1\n + twisted==23.8.0\n + urllib3==2.0.7\n + vcrpy==4.3.0\n + w3lib==2.1.2\n + wrapt==1.16.0\n + yarl==1.9.4\n + zope-interface==6.4.post2\nResolved 99 packages in 209ms\nInstalled 19 packages in 134ms\n + alabaster==0.7.13\n + babel==2.14.0\n + docutils==0.19\n + imagesize==1.4.1\n + jinja2==3.1.5\n + markupsafe==2.1.5\n + nose-timer==1.0.1\n + pygments==2.17.2\n + pypandoc==1.14\n + snowballstemmer==2.2.0\n + sphinx==5.3.0\n + sphinx-rtd-theme==2.0.0\n + sphinxcontrib-applehelp==1.0.2\n + sphinxcontrib-devhelp==1.0.2\n + sphinxcontrib-htmlhelp==2.0.0\n + sphinxcontrib-jquery==4.1\n + sphinxcontrib-jsmath==1.0.1\n + sphinxcontrib-qthelp==1.0.3\n + sphinxcontrib-serializinghtml==1.1.5\n",
    "new_commit_res_code": 1,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.7.9, pytest-7.4.4, pluggy-1.2.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb\nplugins: cov-4.1.0\ncollected 9 items\n\nr2e_tests/test_1.py sEF.EEEE.                                            [100%]\n\n==================================== ERRORS ====================================\n_______________________ ERROR at setup of test_ls_repos ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py, line 59\n  @with_tempfile\n  def test_ls_repos(toppath):\nE       fixture 'toppath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py:59\n________________________ ERROR at setup of test_ignored ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py, line 105\n  @with_tree(\n      tree={'dir': {'file1.txt': '123', 'file2.txt': '456'},\n            '.hidden': {'.hidden_file': '121'}})\n  def test_ignored(topdir):\nE       fixture 'topdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py:105\n______________________ ERROR at setup of test_fs_traverse ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py, line 122\n  @with_tree(\n      tree={'dir': {'.fgit': {'ab.txt': '123'},\n                    'subdir': {'file1.txt': '124', 'file2.txt': '123'},\n                    'subgit': {'fgit.txt': '123'}},\n            'topfile.txt': '123',\n            '.hidden': {'.hidden_file': '123'}})\n  def test_fs_traverse(topdir):\nE       fixture 'topdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py:122\n________________________ ERROR at setup of test_ls_json ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py, line 195\n  @with_tree(\n      tree={'dir': {'.fgit': {'ab.txt': '123'},\n                    'subdir': {'file1.txt': '123', 'file2.txt': '123'},\n                    'subgit': {'fgit.txt': '123'}},\n            '.hidden': {'.hidden_file': '123'}})\n  def test_ls_json(topdir):\nE       fixture 'topdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py:195\n_______________________ ERROR at setup of test_ls_noarg ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py, line 264\n  @with_tempfile\n  def test_ls_noarg(toppath):\nE       fixture 'toppath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py:264\n=================================== FAILURES ===================================\n_____________________________ test_ls_uninstalled ______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexdd72fizf'}\nfilename = '/tmp/datalad_temp_test_ls_uninstalledc0hoyh21'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_ls_uninstalled() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:624: TypeError\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_machinesize\nPASSED r2e_tests/test_1.py::test_ls_formatter\nSKIPPED [1] .venv/lib/python3.7/site-packages/_pytest/unittest.py:371: This test requires known credentials for datalad-test-s3\nERROR r2e_tests/test_1.py::test_ls_repos\nERROR r2e_tests/test_1.py::test_ignored\nERROR r2e_tests/test_1.py::test_fs_traverse\nERROR r2e_tests/test_1.py::test_ls_json\nERROR r2e_tests/test_1.py::test_ls_noarg\nFAILED r2e_tests/test_1.py::test_ls_uninstalled - TypeError: test_ls_uninstal...\n=============== 1 failed, 2 passed, 1 skipped, 5 errors in 1.64s ===============\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.7.9, pytest-7.4.4, pluggy-1.2.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb\nplugins: cov-4.1.0\ncollected 9 items\n\nr2e_tests/test_1.py sEF.EEEEF                                            [100%]\n\n==================================== ERRORS ====================================\n_______________________ ERROR at setup of test_ls_repos ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py, line 59\n  @with_tempfile\n  def test_ls_repos(toppath):\nE       fixture 'toppath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py:59\n________________________ ERROR at setup of test_ignored ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py, line 105\n  @with_tree(\n      tree={'dir': {'file1.txt': '123', 'file2.txt': '456'},\n            '.hidden': {'.hidden_file': '121'}})\n  def test_ignored(topdir):\nE       fixture 'topdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py:105\n______________________ ERROR at setup of test_fs_traverse ______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py, line 122\n  @with_tree(\n      tree={'dir': {'.fgit': {'ab.txt': '123'},\n                    'subdir': {'file1.txt': '124', 'file2.txt': '123'},\n                    'subgit': {'fgit.txt': '123'}},\n            'topfile.txt': '123',\n            '.hidden': {'.hidden_file': '123'}})\n  def test_fs_traverse(topdir):\nE       fixture 'topdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py:122\n________________________ ERROR at setup of test_ls_json ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py, line 195\n  @with_tree(\n      tree={'dir': {'.fgit': {'ab.txt': '123'},\n                    'subdir': {'file1.txt': '123', 'file2.txt': '123'},\n                    'subgit': {'fgit.txt': '123'}},\n            '.hidden': {'.hidden_file': '123'}})\n  def test_ls_json(topdir):\nE       fixture 'topdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py:195\n_______________________ ERROR at setup of test_ls_noarg ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py, line 264\n  @with_tempfile\n  def test_ls_noarg(toppath):\nE       fixture 'toppath' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_77ce648eb80246d9aadb5be1596a98859f4b05cb/r2e_tests/test_1.py:264\n=================================== FAILURES ===================================\n_____________________________ test_ls_uninstalled ______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexl0ftf977'}\nfilename = '/tmp/datalad_temp_test_ls_uninstalledm2jbknxo'\n\n    @wraps(t)\n    def newfunc(*arg, **kw):\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_ls_uninstalled() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:624: TypeError\n______________________________ test_ls_formatter _______________________________\n\n    def test_ls_formatter():\n        # we will use unicode symbols only when sys.stdio supports UTF-8\n        for sysioenc, OK in [(None, \"OK\"),\n                             ('ascii', 'OK'),\n                             ('UTF-8', u\"✓\")]:\n    \n            # we cannot overload sys.stdout.encoding\n            class fake_stdout(object):\n                encoding = sysioenc\n                def write(self, *args):\n                    pass\n    \n            with patch.object(sys, 'stdout', fake_stdout()):\n>               formatter = LsFormatter()\n\nr2e_tests/test_1.py:291: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\nself = <datalad.interface.ls.LsFormatter object at 0x788497e9ac90>, args = ()\nkwargs = {}, setting_encoding = None\n\n    def __init__(self, *args, **kwargs):\n        super(LsFormatter, self).__init__(*args, **kwargs)\n        for setting_encoding in (sys.getdefaultencoding(),\n                                 sys.stdout.encoding):\n            try:\n>               u\"✓\".encode(setting_encoding)\nE               TypeError: encode() argument 1 must be str, not None\n\ndatalad/interface/ls.py:387: TypeError\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_machinesize\nSKIPPED [1] .venv/lib/python3.7/site-packages/_pytest/unittest.py:371: This test requires known credentials for datalad-test-s3\nERROR r2e_tests/test_1.py::test_ls_repos\nERROR r2e_tests/test_1.py::test_ignored\nERROR r2e_tests/test_1.py::test_fs_traverse\nERROR r2e_tests/test_1.py::test_ls_json\nERROR r2e_tests/test_1.py::test_ls_noarg\nFAILED r2e_tests/test_1.py::test_ls_uninstalled - TypeError: test_ls_uninstal...\nFAILED r2e_tests/test_1.py::test_ls_formatter - TypeError: encode() argument ...\n=============== 2 failed, 1 passed, 1 skipped, 5 errors in 0.84s ===============\n",
    "old_commit_res_stderr": ""
}
datalad/interface/ls.py datalad/interface/tests/test_ls.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** TypeError Occurs When `sys.stdout.encoding` Is None in Python 2

**Description:**
When using the `ls` interface in an environment where `stdout` is piped in Python 2, `sys.stdout.encoding` returns `None`. The current implementation attempts to encode unicode characters based on `sys.getdefaultencoding`, which is `'ascii'` in Python 2. This results in a `TypeError` because the `encode` method receives `None` instead of a valid encoding string.

**Example Buggy Code:**
```python
import sys

class LsFormatter(object):
    OK = u"✓"
    
    def __init__(self):
        # sys.stdout.encoding is None when stdout is piped
        self.OK.encode(sys.stdout.encoding)

formatter = LsFormatter()
```

**Expected Behavior:**
The formatter should handle cases where `sys.stdout.encoding` is `None` gracefully, such as by disabling unicode symbols or using a safe default encoding, instead of raising an error.

**Actual Behavior:**
A `TypeError` is raised with the message: `encode() argument 1 must be str, not None`.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/interface/ls.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil; coding: utf-8 -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Helper utility to list things.  ATM list content of S3 bucket
"""

__docformat__ = 'restructuredtext'

import sys
import time
from os.path import exists, lexists, join as opj, abspath, isabs, getmtime
from os.path import curdir, isfile, islink, isdir, dirname, basename, split, realpath
from os.path import relpath
from os import listdir, lstat, remove, makedirs
import json as js
import hashlib

from six.moves.urllib.request import urlopen, Request
from six.moves.urllib.error import HTTPError

from ..cmdline.helpers import get_repo_instance
from ..utils import auto_repr
from .base import Interface
from datalad.interface.base import build_doc
from ..ui import ui
from ..utils import safe_print
from ..consts import METADATA_DIR
from ..consts import METADATA_FILENAME
from ..dochelpers import exc_str
from ..support.param import Parameter
from ..support import ansi_colors
from ..support.constraints import EnsureStr, EnsureNone
from ..distribution.dataset import Dataset

from datalad.support.annexrepo import AnnexRepo
from datalad.support.annexrepo import GitRepo

import string
import humanize
from datalad.utils import is_interactive

from logging import getLogger
lgr = getLogger('datalad.api.ls')


@build_doc
class Ls(Interface):
    """List summary information about URLs and dataset(s)

    ATM only s3:// URLs and datasets are supported

    Examples:

      $ datalad ls s3://openfmri/tarballs/ds202  # to list S3 bucket
      $ datalad ls                               # to list current dataset
    """
    # XXX prevent common args from being added to the docstring
    _no_eval_results = True

    # TODO: during big RF refactor this one away since it must not be instance's
    # attribute.  For now introduced to make `datalad ls` be relatively usable
    # in terms of speed
    _cached_subdatasets = {}

    _params_ = dict(
        loc=Parameter(
            doc="URL or path to list, e.g. s3://...",
            metavar='PATH/URL',
            nargs="*",
            constraints=EnsureStr() | EnsureNone(),
        ),
        recursive=Parameter(
            args=("-r", "--recursive"),
            action="store_true",
            doc="recurse into subdirectories",
        ),
        fast=Parameter(
            args=("-F", "--fast"),
            action="store_true",
            doc="only perform fast operations.  Would be overridden by --all",
        ),
        all_=Parameter(
            args=("-a", "--all"),
            dest='all_',
            action="store_true",
            doc="list all (versions of) entries, not e.g. only latest entries "
                "in case of S3",
        ),
        long_=Parameter(
            args=("-L", "--long"),
            dest='long_',
            action="store_true",
            doc="list more information on entries (e.g. acl, urls in s3, annex "
                "sizes etc)",
        ),
        config_file=Parameter(
            doc="""path to config file which could help the 'ls'.  E.g. for s3://
            URLs could be some ~/.s3cfg file which would provide credentials""",
            constraints=EnsureStr() | EnsureNone()
        ),
        list_content=Parameter(
            choices=(None, 'first10', 'md5', 'full'),
            doc="""list also the content or only first 10 bytes (first10), or md5
            checksum of an entry.  Might require expensive transfer and dump
            binary output to your screen.  Do not enable unless you know what you
            are after""",
            default=None
        ),
        json=Parameter(
            choices=('file', 'display', 'delete'),
            doc="""metadata json of dataset for creating web user interface.
            display: prints jsons to stdout or
            file: writes each subdir metadata to json file in subdir of dataset or
            delete: deletes all metadata json files in dataset""",
        ),
    )

    @staticmethod
    def __call__(loc, recursive=False, fast=False, all_=False, long_=False,
                 config_file=None, list_content=False, json=None):
        if isinstance(loc, list) and not len(loc):
            # nothing given, CWD assumed -- just like regular ls
            loc = '.'

        kw = dict(fast=fast, recursive=recursive, all_=all_, long_=long_)
        if isinstance(loc, list):
            return [Ls.__call__(loc_, config_file=config_file,
                                list_content=list_content, json=json, **kw)
                    for loc_ in loc]

        # TODO: do some clever handling of kwargs as to remember what were defaults
        # and what any particular implementation actually needs, and then issuing
        # warning if some custom value/option was specified which doesn't apply to the
        # given url

        # rename to not angry Python gods who took all_ good words
        kw['long_'] = kw.pop('long_')

        loc_type = "unknown"
        if loc.startswith('s3://'):
            return _ls_s3(loc, config_file=config_file, list_content=list_content,
                          **kw)
        elif lexists(loc):
            if isdir(loc):
                ds = Dataset(loc)
                if ds.is_installed():
                    return _ls_json(loc, json=json, **kw) if json else _ls_dataset(loc, **kw)
                    loc_type = False
                else:
                    loc_type = "dir"  # we know that so far for sure
                    # it might have been an uninstalled dataset within super-dataset
                    superds = ds.get_superdataset()
                    if superds:
                        try:
                            subdatasets = Ls._cached_subdatasets[superds.path]
                        except KeyError:
                            subdatasets = Ls._cached_subdatasets[superds.path] \
                                = superds.subdatasets(result_xfm='relpaths')
                        if relpath(ds.path, superds.path) in subdatasets:
                            loc_type = "not installed"
            else:
                loc_type = "file"
                # could list properties -- under annex or git, either clean/dirty
                # etc
                # repo = get_repo_instance(dirname(loc))

        if loc_type:
            #raise ValueError("ATM supporting only s3:// URLs and paths to local datasets")
            # TODO: unify all_ the output here -- _ls functions should just return something
            # to be displayed
            ui.message(
                "{}  {}".format(
                    ansi_colors.color_word(loc, ansi_colors.DATASET),
                    ansi_colors.color_word(
                        loc_type,
                        ansi_colors.RED
                        if loc_type in {'unknown', 'not installed'}
                        else ansi_colors.BLUE)
                )
            )


#
# Dataset listing
#

@auto_repr
class AbsentRepoModel(object):
    """Just a base for those where repo wasn't installed yet"""

    def __init__(self, path):
        self.path = path
        self.repo = None

    @property
    def type(self):
        return "N/A"


@auto_repr
class GitModel(object):
    """A base class for models which have some .repo available"""

    __slots__ = ['_branch', 'repo', '_path']

    def __init__(self, repo):
        self.repo = repo
        # lazy evaluation variables
        self._branch = None
        self._path = None

    @property
    def path(self):
        return self.repo.path if self._path is None else self._path

    @path.setter
    def path(self, v):
        self._path = v

    @property
    def branch(self):
        if self._branch is None:
            try:
                self._branch = self.repo.get_active_branch()
            except:  # MIH: InvalidGitRepositoryError?
                return None
        return self._branch

    @property
    def clean(self):
        return not self.repo.dirty

    @property
    def describe(self):
        return self.repo.describe(tags=True)

    @property
    def date(self):
        """Date of the last commit
        """
        return self.repo.get_committed_date()

    @property
    def count_objects(self):
        return self.repo.count_objects

    @property
    def git_local_size(self):
        count_objects = self.count_objects
        return count_objects['size'] if count_objects else None

    @property
    def type(self):
        return {False: 'git', True: 'annex'}[isinstance(self.repo, AnnexRepo)]


@auto_repr
class AnnexModel(GitModel):

    __slots__ = ['_info'] + GitModel.__slots__

    def __init__(self, *args, **kwargs):
        super(AnnexModel, self).__init__(*args, **kwargs)
        self._info = None

    @property
    def info(self):
        if self._info is None and self.type == 'annex':
            self._info = self.repo.repo_info()
        return self._info

    @property
    def annex_worktree_size(self):
        info = self.info
        return info['size of annexed files in working tree'] if info else 0.0

    @property
    def annex_local_size(self):
        info = self.info
        return info['local annex size'] if info else 0.0


@auto_repr
class FsModel(AnnexModel):

    __slots__ = AnnexModel.__slots__

    def __init__(self, path, *args, **kwargs):
        super(FsModel, self).__init__(*args, **kwargs)
        self._path = path

    @property
    def path(self):
        return self._path

    @property
    def symlink(self):
        """if symlink returns path the symlink points to else returns None"""
        if islink(self._path):                    # if symlink
            target_path = realpath(self._path)    # find link target
            # convert to absolute path if not
            return target_path if exists(target_path) else None
        return None

    @property
    def date(self):
        """Date of last modification"""
        if self.type_ is not ['git', 'annex']:
            return lstat(self._path).st_mtime
        else:
            super(self.__class__, self).date

    @property
    def size(self):
        """Size of the node computed based on its type"""
        type_ = self.type_
        sizes = {'total': 0.0, 'ondisk': 0.0, 'git': 0.0, 'annex': 0.0, 'annex_worktree': 0.0}

        if type_ in ['file', 'link', 'link-broken']:
            # if node is under annex, ask annex for node size, ondisk_size
            if isinstance(self.repo, AnnexRepo) and self.repo.is_under_annex(self._path):
                size = self.repo.info(self._path, batch=True)['size']
                ondisk_size = size \
                    if self.repo.file_has_content(self._path) \
                    else 0
            # else ask fs for node size (= ondisk_size)
            else:
                size = ondisk_size = 0 \
                    if type_ == 'link-broken' \
                    else lstat(self.symlink or self._path).st_size

            sizes.update({'total': size, 'ondisk': ondisk_size})

        if self.repo.path == self._path:
            sizes.update({'git': self.git_local_size,
                          'annex': self.annex_local_size,
                          'annex_worktree': self.annex_worktree_size})
        return sizes

    @property
    def type_(self):
        """outputs the node type

        Types: link, link-broken, file, dir, annex-repo, git-repo"""
        if islink(self.path):
            return 'link' if self.symlink else 'link-broken'
        elif isfile(self.path):
            return 'file'
        elif exists(opj(self.path, ".git", "annex")):
            return 'annex'
        elif exists(opj(self.path, ".git")):
            return 'git'
        elif isdir(self.path):
            return 'dir'
        else:
            return None


class LsFormatter(string.Formatter):
    # condition by interactive
    if is_interactive():
        BLUE = ansi_colors.COLOR_SEQ % ansi_colors.BLUE
        RED = ansi_colors.COLOR_SEQ % ansi_colors.RED
        GREEN = ansi_colors.COLOR_SEQ % ansi_colors.GREEN
        RESET = ansi_colors.RESET_SEQ
        DATASET = ansi_colors.COLOR_SEQ % ansi_colors.UNDERLINE
    else:
        BLUE = RED = GREEN = RESET = DATASET = u""

    # TODO: we might want to just ignore and force utf8 while explicitly .encode()'ing output!
    # unicode versions which look better but which blow during tests etc
    # Those might be reset by the constructor
    OK = u"✓"
    NOK = u"✗"
    NONE = u"✗"

    def __init__(self, *args, **kwargs):
        super(LsFormatter, self).__init__(*args, **kwargs)
        for setting_encoding in (sys.getdefaultencoding(),
                                 sys.stdout.encoding):
            try:
                u"✓".encode(setting_encoding)
            except UnicodeEncodeError:
                lgr.debug("encoding %s found to not support unicode, resetting to safe alternatives", setting_encoding)
                self.OK = 'OK'   # u"✓"
                self.NOK = 'X'  # u"✗"
                self.NONE = '-'  # u"✗"
                break


    def convert_field(self, value, conversion):
        #print("%r->%r" % (value, conversion))
        if conversion == 'D':  # Date
            if value is not None:
                return time.strftime(u"%Y-%m-%d/%H:%M:%S", time.localtime(value))
            else:
                return u'-'
        elif conversion == 'S':  # Human size
            #return value
            if value is not None:
                return humanize.naturalsize(value)
            else:
                return u'-'
        elif conversion == 'X':  # colored bool
            chr, col = (self.OK, self.GREEN) if value else (self.NOK, self.RED)
            return u"%s%s%s" % (col, chr, self.RESET)
        elif conversion == 'N':  # colored Red - if None
            if value is None:
                # return "%s✖%s" % (self.RED, self.RESET)
                return u"%s%s%s" % (self.RED, self.NONE, self.RESET)
            return value
        elif conversion in {'B', 'R', 'U'}:
            return u"%s%s%s" % ({'B': self.BLUE, 'R': self.RED, 'U': self.DATASET}[conversion], value, self.RESET)

        return super(LsFormatter, self).convert_field(value, conversion)

    def format_field(self, value, format_spec):
        # TODO: move all the "coloring" into formatting, so we could correctly indent
        # given the format and only then color it up
        # print "> %r, %r" % (value, format_spec)
        return super(LsFormatter, self).format_field(value, format_spec)


def format_ds_model(formatter, ds_model, format_str, format_exc):
    try:
        #print("WORKING ON %s" % ds_model.path)
        if not exists(ds_model.path) or not ds_model.repo:
            return formatter.format(format_exc, ds=ds_model, msg=u"not installed")
        ds_formatted = formatter.format(format_str, ds=ds_model)
        #print("FINISHED ON %s" % ds_model.path)
        return ds_formatted
    except Exception as exc:
        return formatter.format(format_exc, ds=ds_model, msg=exc_str(exc))

# from joblib import Parallel, delayed


def _ls_dataset(loc, fast=False, recursive=False, all_=False, long_=False):
    isabs_loc = isabs(loc)
    topdir = '' if isabs_loc else abspath(curdir)

    topds = Dataset(loc)
    dss = [topds] + (
        [Dataset(opj(loc, sm))
         for sm in topds.subdatasets(recursive=recursive, result_xfm='relpaths')]
        if recursive else [])

    dsms = []
    for ds in dss:
        if not ds.is_installed():
            dsm = AbsentRepoModel(ds.path)
        elif isinstance(ds.repo, AnnexRepo):
            dsm = AnnexModel(ds.repo)
        elif isinstance(ds.repo, GitRepo):
            dsm = GitModel(ds.repo)
        else:
            raise RuntimeError("Got some dataset which don't know how to handle %s"
                               % ds)
        dsms.append(dsm)

    # adjust path strings
    for ds_model in dsms:
        #path = ds_model.path[len(topdir) + 1 if topdir else 0:]
        path = relpath(ds_model.path, topdir) if topdir else ds_model.path
        if not path:
            path = '.'
        ds_model.path = path
    dsms = sorted(dsms, key=lambda m: m.path)

    maxpath = max(len(ds_model.path) for ds_model in dsms)
    path_fmt = u"{ds.path!U:<%d}" % (maxpath + (11 if is_interactive() else 0))  # + to accommodate ansi codes
    pathtype_fmt = path_fmt + u"  [{ds.type}]"
    full_fmt = pathtype_fmt + u"  {ds.branch!N}  {ds.describe!N} {ds.date!D}"
    if (not fast) or long_:
        full_fmt += u"  {ds.clean!X}"

    fmts = {
        AbsentRepoModel: pathtype_fmt,
        GitModel: full_fmt,
        AnnexModel: full_fmt
    }
    if long_:
        fmts[AnnexModel] += u"  {ds.annex_local_size!S}/{ds.annex_worktree_size!S}"

    formatter = LsFormatter()
    # weird problems happen in the parallel run -- TODO - figure it out
    # for out in Parallel(n_jobs=1)(
    #         delayed(format_ds_model)(formatter, dsm, full_fmt, format_exc=path_fmt + "  {msg!R}")
    #         for dsm in dss):
    #     print(out)
    for dsm in dsms:
        fmt = fmts[dsm.__class__]
        ds_str = format_ds_model(formatter, dsm, fmt, format_exc=path_fmt + u"  {msg!R}")
        safe_print(ds_str)
        # workaround for explosion of git cat-file --batch processes
        # https://github.com/datalad/datalad/issues/1888
        if dsm.repo is not None:
            dsm.repo.repo.close()
            del dsm.repo
            dsm.repo = None


def machinesize(humansize):
    """convert human-size string to machine-size"""
    try:
        size_str, size_unit = humansize.split(" ")
    except AttributeError:
        return float(humansize)
    unit_converter = {'Byte': 0, 'Bytes': 0, 'kB': 1, 'MB': 2, 'GB': 3, 'TB': 4, 'PB': 5}
    machinesize = float(size_str) * (1000 ** unit_converter[size_unit])
    return machinesize


def leaf_name(path):
    """takes a relative or absolute path and returns name of node at that location"""
    head, tail = split(abspath(path))
    return tail or basename(head)


def ignored(path, only_hidden=False):
    """if path is in the ignorelist return True

    ignore list includes hidden files and git or annex maintained folders
    when only_hidden set, only ignores hidden files and folders not git or annex maintained folders
    """
    if isdir(opj(path, ".git")) and not only_hidden:
        return True
    return '.' == leaf_name(path)[0] or leaf_name(path) == 'index.html'


def metadata_locator(fs_metadata=None, path=None, ds_path=None, metadata_path=None):
    """path to metadata file of node associated with the fs_metadata dictionary

    Parameters
    ----------
    fs_metadata: dict
      Metadata json of a node
    path: str
      Path to directory of metadata to be rendered
    ds_path: str
      Path to dataset root
    metadata_path: str
      Path to metadata root. Calculated relative to ds_path

    Returns
    -------
    str
      path to metadata of current node
    """

    # use implicit paths unless paths explicitly specified
    # Note: usage of ds_path as if it was the Repo's path. Therefore use
    # realpath, since we switched to have symlinks resolved in repos but not in
    # datasets
    ds_path = realpath(ds_path) if ds_path else fs_metadata['repo']
    path = path or fs_metadata['path']
    metadata_path = metadata_path or '.git/datalad/metadata'
    # directory metadata directory tree location
    metadata_dir = opj(ds_path, metadata_path)
    # relative path of current directory wrt dataset root
    dir_path = relpath(path, ds_path) if isabs(path) else path
    # normalize to / -- TODO, switch to '.' which is now actually the name since path is relative in web meta?
    if dir_path in ('.', None, ''):
        dir_path = '/'
    # create md5 hash of current directory's relative path
    metadata_hash = hashlib.md5(dir_path.encode('utf-8')).hexdigest()
    # construct final path to metadata file
    metadata_file = opj(metadata_dir, metadata_hash)

    return metadata_file


def fs_extract(nodepath, repo, basepath='/'):
    """extract required info of nodepath with its associated parent repository and returns it as a dictionary

    Parameters
    ----------
    nodepath : str
        Full path to the location we are exploring (must be a directory within
        `repo`
    repo : GitRepo
        Is the repository nodepath belongs to
    """
    # Create FsModel from filesystem nodepath and its associated parent repository
    node = FsModel(nodepath, repo)
    pretty_size = {stype: humanize.naturalsize(svalue) for stype, svalue in node.size.items()}
    pretty_date = time.strftime(u"%Y-%m-%d %H:%M:%S", time.localtime(node.date))
    name = leaf_name(node._path) if leaf_name(node._path) != "" else leaf_name(node.repo.path)
    rec = {
        "name": name, "path": relpath(node._path, basepath),
        "type": node.type_, "size": pretty_size, "date": pretty_date,
    }
    # if there is meta-data for the dataset (done by aggregate-metadata)
    # we include it
    metadata_path = opj(nodepath, METADATA_DIR, METADATA_FILENAME)
    if exists(metadata_path):
        # might need flattening!  TODO: flatten when aggregating?  why wasn't done?
        metadata = js.load(open(metadata_path))
        # might be too heavy to carry around, so will do basic flattening manually
        # and in a basic fashion
        # import jsonld
        metadata_reduced = metadata[0]
        for m in metadata[1:]:
            metadata_reduced.update(m)
        # but verify that they all had the same id
        if metadata:
            metaid = metadata[0]['@id']
            assert all(m['@id'] == metaid for m in metadata)
        rec["metadata"] = metadata_reduced
    return rec


def fs_render(fs_metadata, json=None, **kwargs):
    """takes node to render and based on json option passed renders to file, stdout or deletes json at root

    Parameters
    ----------
    fs_metadata: dict
      Metadata json to be rendered
    json: str ('file', 'display', 'delete')
      Render to file, stdout or delete json
    """

    metadata_file = metadata_locator(fs_metadata, **kwargs)

    if json == 'file':
        # create metadata_root directory if it doesn't exist
        metadata_dir = dirname(metadata_file)
        if not exists(metadata_dir):
            makedirs(metadata_dir)
        # write directory metadata to json
        with open(metadata_file, 'w') as f:
            js.dump(fs_metadata, f)

    # else if json flag set to delete, remove .dir.json of current directory
    elif json == 'delete' and exists(metadata_file):
        remove(metadata_file)

    # else dump json to stdout
    elif json == 'display':
        safe_print(js.dumps(fs_metadata) + '\n')


def fs_traverse(path, repo, parent=None, render=True, recursive=False, json=None, basepath=None):
    """Traverse path through its nodes and returns a dictionary of relevant attributes attached to each node

    Parameters
    ----------
    path: str
      Path to the directory to be traversed
    repo: AnnexRepo or GitRepo
      Repo object the directory belongs too
    parent: dict
      Extracted info about parent directory
    recursive: bool
      Recurse into subdirectories (note that subdatasets are not traversed)
    render: bool
       To render from within function or not. Set to false if results to be manipulated before final render

    Returns
    -------
    list of dict
      extracts and returns a (recursive) list of directory info at path
      does not traverse into annex, git or hidden directories
    """
    fs = fs_extract(path, repo, basepath=basepath or path)
    if isdir(path):                     # if node is a directory
        children = [fs.copy()]          # store its info in its children dict too  (Yarik is not sure why, but I guess for .?)
        # ATM seems some pieces still rely on having this duplication, so left as is
        # TODO: strip away
        for node in listdir(path):
            nodepath = opj(path, node)

            # TODO:  it might be a subdir which is non-initialized submodule!
            # if not ignored, append child node info to current nodes dictionary
            if not ignored(nodepath):
                # if recursive, create info dictionary (within) each child node too
                if recursive:
                    subdir = fs_traverse(nodepath,
                                         repo,
                                         parent=None,  # children[0],
                                         recursive=recursive,
                                         json=json,
                                         basepath=basepath or path)
                    subdir.pop('nodes', None)
                else:
                    # read child metadata from its metadata file if it exists
                    subdir_json = metadata_locator(path=node, ds_path=basepath or path)
                    if exists(subdir_json):
                        with open(subdir_json) as data_file:
                            subdir = js.load(data_file)
                            subdir.pop('nodes', None)
                    # else extract whatever information you can about the child
                    else:
                        # Yarik: this one is way too lean...
                        subdir = fs_extract(nodepath,
                                            repo,
                                            basepath=basepath or path)
                # append child metadata to list
                children.extend([subdir])

        # sum sizes of all 1st level children
        children_size = {}
        for node in children[1:]:
            for size_type, child_size in node['size'].items():
                children_size[size_type] = children_size.get(size_type, 0) + machinesize(child_size)

        # update current node sizes to the humanized aggregate children size
        fs['size'] = children[0]['size'] = \
            {size_type: humanize.naturalsize(child_size)
             for size_type, child_size in children_size.items()}

        children[0]['name'] = '.'       # replace current node name with '.' to emulate unix syntax
        if parent:
            parent['name'] = '..'       # replace parent node name with '..' to emulate unix syntax
            children.insert(1, parent)  # insert parent info after current node info in children dict

        fs['nodes'] = children          # add children info to main fs dictionary
        if render:                      # render directory node at location(path)
            fs_render(fs, json=json, ds_path=basepath or path)
            lgr.info('Directory: %s' % path)

    return fs


def ds_traverse(rootds, parent=None, json=None, recursive=False, all_=False,
                long_=False):
    """Hierarchical dataset traverser

    Parameters
    ----------
    rootds: Dataset
      Root dataset to be traversed
    parent: Dataset
      Parent dataset of the current rootds
    recursive: bool
       Recurse into subdirectories of the current dataset
    all_: bool
       Recurse into subdatasets of the root dataset

    Returns
    -------
    list of dict
      extracts and returns a (recursive) list of dataset(s) info at path
    """
    # extract parent info to pass to traverser
    fsparent = fs_extract(parent.path, parent.repo, basepath=rootds.path) if parent else None

    # (recursively) traverse file tree of current dataset
    fs = fs_traverse(rootds.path, rootds.repo,
                     render=False, parent=fsparent, recursive=all_,
                     json=json)
    size_list = [fs['size']]

    # (recursively) traverse each subdataset
    children = []
    # yoh: was in return results branch returning full datasets:
    # for subds in rootds.subdatasets(result_xfm='datasets'):
    # but since rpath is needed/used, decided to return relpaths
    for subds_rpath in rootds.subdatasets(result_xfm='relpaths'):

        subds_path = opj(rootds.path, subds_rpath)
        subds = Dataset(subds_path)
        subds_json = metadata_locator(path='.', ds_path=subds_path)

        def handle_not_installed():
            # for now just traverse as fs
            lgr.warning("%s is either not installed or lacks meta-data", subds)
            subfs = fs_extract(subds_path, rootds, basepath=rootds.path)
            # but add a custom type that it is a not installed subds
            subfs['type'] = 'uninitialized'
            # we need to kick it out from 'children'
            # TODO:  this is inefficient and cruel -- "ignored" should be made
            # smarted to ignore submodules for the repo
            if fs['nodes']:
                fs['nodes'] = [c for c in fs['nodes'] if c['path'] != subds_rpath]
            return subfs

        if not subds.is_installed():
            subfs = handle_not_installed()
        elif recursive:
            subfs = ds_traverse(subds,
                                json=json,
                                recursive=recursive,
                                all_=all_,
                                parent=rootds)
            subfs.pop('nodes', None)
            size_list.append(subfs['size'])
        # else just pick the data from metadata_file of each subdataset
        else:
            lgr.info(subds.path)
            if exists(subds_json):
                with open(subds_json) as data_file:
                    subfs = js.load(data_file)
                    subfs.pop('nodes', None)    # remove children
                    subfs['path'] = subds_rpath # reassign the path
                    size_list.append(subfs['size'])
            else:
                # the same drill as if not installed
                lgr.warning("%s is installed but no meta-data yet", subds)
                subfs = handle_not_installed()

        children.extend([subfs])

    # sum sizes of all 1st level children dataset
    children_size = {}
    for subdataset_size in size_list:
        for size_type, subds_size in subdataset_size.items():
            children_size[size_type] = children_size.get(size_type, 0) + machinesize(subds_size)

    # update current dataset sizes to the humanized aggregate subdataset sizes
    fs['size'] = {size_type: humanize.naturalsize(size)
                  for size_type, size in children_size.items()}
    fs['nodes'][0]['size'] = fs['size']  # update self's updated size in nodes sublist too!

    # add dataset specific entries to its dict
    rootds_model = GitModel(rootds.repo)
    fs['tags'] = rootds_model.describe
    fs['branch'] = rootds_model.branch
    index_file = opj(rootds.path, '.git', 'index')
    fs['index-mtime'] = time.strftime(
        u"%Y-%m-%d %H:%M:%S",
        time.localtime(getmtime(index_file))) if exists(index_file) else ''

    # append children datasets info to current dataset
    fs['nodes'].extend(children)

    # render current dataset
    lgr.info('Dataset: %s' % rootds.path)
    fs_render(fs, json=json, ds_path=rootds.path)
    return fs


def _ls_json(loc, fast=False, **kwargs):
    # hierarchically traverse file tree of (sub-)dataset(s) under path passed(loc)
    return ds_traverse(Dataset(loc), parent=None, **kwargs)


#
# S3 listing
#
def _ls_s3(loc, fast=False, recursive=False, all_=False, long_=False,
           config_file=None, list_content=False):
    """List S3 bucket content"""
    if loc.startswith('s3://'):
        bucket_prefix = loc[5:]
    else:
        raise ValueError("passed location should be an s3:// url")

    import boto
    from hashlib import md5
    from boto.s3.key import Key
    from boto.s3.prefix import Prefix
    from boto.exception import S3ResponseError
    from ..support.configparserinc import SafeConfigParser  # provides PY2,3 imports

    if '/' in bucket_prefix:
        bucket_name, prefix = bucket_prefix.split('/', 1)
    else:
        bucket_name, prefix = bucket_prefix, None

    if prefix and '?' in prefix:
        ui.message("We do not care about URL options ATM, they get stripped")
        prefix = prefix[:prefix.index('?')]

    ui.message("Connecting to bucket: %s" % bucket_name)
    if config_file:
        config = SafeConfigParser()
        config.read(config_file)
        access_key = config.get('default', 'access_key')
        secret_key = config.get('default', 'secret_key')

        # TODO: remove duplication -- reuse logic within downloaders/s3.py to get connected
        conn = boto.connect_s3(access_key, secret_key)
        try:
            bucket = conn.get_bucket(bucket_name)
        except S3ResponseError as e:
            ui.message("E: Cannot access bucket %s by name" % bucket_name)
            all_buckets = conn.get_all_buckets()
            all_bucket_names = [b.name for b in all_buckets]
            ui.message("I: Found following buckets %s" % ', '.join(all_bucket_names))
            if bucket_name in all_bucket_names:
                bucket = all_buckets[all_bucket_names.index(bucket_name)]
            else:
                raise RuntimeError("E: no bucket named %s thus exiting" % bucket_name)
    else:
        # TODO: expose credentials
        # We don't need any provider here really but only credentials
        from datalad.downloaders.providers import Providers
        providers = Providers.from_config_files()
        provider = providers.get_provider(loc)

        if not provider:
            raise ValueError(
                "Don't know how to deal with this url %s -- no provider defined for %s. "
                "Define a new provider (DOCS: TODO) or specify just s3cmd config file instead for now."
                % loc
            )
        downloader = provider.get_downloader(loc)

        # should authenticate etc, and when ready we will ask for a bucket ;)
        bucket = downloader.access(lambda url: downloader.bucket, loc)

    info = []
    for iname, imeth in [
        ("Versioning", bucket.get_versioning_status),
        ("   Website", bucket.get_website_endpoint),
        ("       ACL", bucket.get_acl),
    ]:
        try:
            ival = imeth()
        except Exception as e:
            ival = str(e).split('\n')[0]
        info.append(" {iname}: {ival}".format(**locals()))
    ui.message("Bucket info:\n %s" % '\n '.join(info))

    kwargs = {} if recursive else {'delimiter': '/'}

    ACCESS_METHODS = [
        bucket.list_versions,
        bucket.list
    ]

    prefix_all_versions = None
    for acc in ACCESS_METHODS:
        try:
            prefix_all_versions = list(acc(prefix, **kwargs))
            break
        except Exception as exc:
            lgr.debug("Failed to access via %s: %s", acc, exc_str(exc))

    if not prefix_all_versions:
        ui.error("No output was provided for prefix %r" % prefix)
    else:
        max_length = max((len(e.name) for e in prefix_all_versions))
        max_size_length = max((len(str(getattr(e, 'size', 0))) for e in prefix_all_versions))

    results = []
    for e in prefix_all_versions:
        results.append(e)
        if isinstance(e, Prefix):
            ui.message("%s" % (e.name, ),)
            continue

        base_msg = ("%%-%ds %%s" % max_length) % (e.name, e.last_modified)
        if isinstance(e, Key):
            if not (e.is_latest or all_):
                # Skip this one
                continue
            ui.message(base_msg + " %%%dd" % max_size_length % e.size, cr=' ')
            # OPT: delayed import
            from ..support.s3 import get_key_url
            url = get_key_url(e, schema='http')
            try:
                _ = urlopen(Request(url))
                urlok = "OK"
            except HTTPError as err:
                urlok = "E: %s" % err.code

            try:
                acl = e.get_acl()
            except S3ResponseError as exc:
                acl = exc.code if exc.code in ('AccessDenied',) else str(exc)

            content = ""
            if list_content:
                # IO intensive, make an option finally!
                try:
                    # _ = e.next()[:5]  if we are able to fetch the content
                    kwargs = dict(version_id=e.version_id)
                    if list_content in {'full', 'first10'}:
                        if list_content in 'first10':
                            kwargs['headers'] = {'Range': 'bytes=0-9'}
                        content = repr(e.get_contents_as_string(**kwargs))
                    elif list_content == 'md5':
                        digest = md5()
                        digest.update(e.get_contents_as_string(**kwargs))
                        content = digest.hexdigest()
                    else:
                        raise ValueError(list_content)
                    # content = "[S3: OK]"
                except S3ResponseError as err:
                    content = str(err)
                finally:
                    content = " " + content
            ui.message(
                "ver:%-32s  acl:%s  %s [%s]%s"
                % (getattr(e, 'version_id', None),
                   acl, url, urlok, content)
                if long_ else ''
            )
        else:
            ui.message(base_msg + " " + str(type(e)).split('.')[-1].rstrip("\"'>"))
    return results

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]
**Title:** TypeError Occurs When `sys.stdout.encoding` Is None in Python 2

**Description:**
When using the `ls` interface in an environment where `stdout` is piped in Python 2, `sys.stdout.encoding` returns `None`. The current implementation attempts to encode unicode characters based on `sys.getdefaultencoding`, which is `'ascii'` in Python 2. This results in a `TypeError` because the `encode` method receives `None` instead of a valid encoding string.

**Example Buggy Code:**
```python
import sys

class LsFormatter(object):
    OK = u"✓"
    
    def __init__(self):
        # sys.stdout.encoding is None when stdout is piped
        self.OK.encode(sys.stdout.encoding)

formatter = LsFormatter()
```

**Expected Behavior:**
The formatter should handle cases where `sys.stdout.encoding` is `None` gracefully, such as by disabling unicode symbols or using a safe default encoding, instead of raising an error.

**Actual Behavior:**
A `TypeError` is raised with the message: `encode() argument 1 must be str, not None`.

[/ISSUE]
{
    "test_machinesize": "PASSED",
    "test_ls_formatter": "PASSED",
    "test_ls_repos": "ERROR",
    "test_ignored": "ERROR",
    "test_fs_traverse": "ERROR",
    "test_ls_json": "ERROR",
    "test_ls_noarg": "ERROR",
    "test_ls_uninstalled": "FAILED"
}
diff --git a/datalad/interface/ls.py b/datalad/interface/ls.py
index e979d255c..523fa0e71 100644
--- a/datalad/interface/ls.py
+++ b/datalad/interface/ls.py
@@ -375,23 +375,25 @@ class LsFormatter(string.Formatter):
     # TODO: we might want to just ignore and force utf8 while explicitly .encode()'ing output!
     # unicode versions which look better but which blow during tests etc
     # Those might be reset by the constructor
-    OK = u"✓"
-    NOK = u"✗"
-    NONE = u"✗"
+    OK = 'OK'   # u"✓"
+    NOK = 'X'  # u"✗"
+    NONE = '-'  # u"✗"
 
     def __init__(self, *args, **kwargs):
         super(LsFormatter, self).__init__(*args, **kwargs)
-        for setting_encoding in (sys.getdefaultencoding(),
-                                 sys.stdout.encoding):
+        if sys.stdout.encoding is None:
+            lgr.debug("encoding not set, using safe alternatives")
+        else:
             try:
-                u"✓".encode(setting_encoding)
+                u"✓".encode(sys.stdout.encoding)
             except UnicodeEncodeError:
-                lgr.debug("encoding %s found to not support unicode, resetting to safe alternatives", setting_encoding)
-                self.OK = 'OK'   # u"✓"
-                self.NOK = 'X'  # u"✗"
-                self.NONE = '-'  # u"✗"
-                break
-
+                lgr.debug("encoding %s does not support unicode, "
+                          "using safe alternatives",
+                          sys.stdout.encoding)
+            else:
+                self.OK = u"✓"
+                self.NOK = u"✗"
+                self.NONE = u"✗"
 
     def convert_field(self, value, conversion):
         #print("%r->%r" % (value, conversion))

{"datalad/interface/ls.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil; coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Helper utility to list things.  ATM list content of S3 bucket\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport sys\nimport time\nfrom os.path import exists, lexists, join as opj, abspath, isabs, getmtime\nfrom os.path import curdir, isfile, islink, isdir, dirname, basename, split, realpath\nfrom os.path import relpath\nfrom os import listdir, lstat, remove, makedirs\nimport json as js\nimport hashlib\n\nfrom six.moves.urllib.request import urlopen, Request\nfrom six.moves.urllib.error import HTTPError\n\nfrom ..cmdline.helpers import get_repo_instance\nfrom ..utils import auto_repr\nfrom .base import Interface\nfrom datalad.interface.base import build_doc\nfrom ..ui import ui\nfrom ..utils import safe_print\nfrom ..consts import METADATA_DIR\nfrom ..consts import METADATA_FILENAME\nfrom ..dochelpers import exc_str\nfrom ..support.param import Parameter\nfrom ..support import ansi_colors\nfrom ..support.constraints import EnsureStr, EnsureNone\nfrom ..distribution.dataset import Dataset\n\nfrom datalad.support.annexrepo import AnnexRepo\nfrom datalad.support.annexrepo import GitRepo\n\nimport string\nimport humanize\nfrom datalad.utils import is_interactive\n\nfrom logging import getLogger\nlgr = getLogger('datalad.api.ls')\n\n\n@build_doc\nclass Ls(Interface):\n    \"\"\"List summary information about URLs and dataset(s)\n\n    ATM only s3:// URLs and datasets are supported\n\n    Examples:\n\n      $ datalad ls s3://openfmri/tarballs/ds202  # to list S3 bucket\n      $ datalad ls                               # to list current dataset\n    \"\"\"\n    # XXX prevent common args from being added to the docstring\n    _no_eval_results = True\n\n    # TODO: during big RF refactor this one away since it must not be instance's\n    # attribute.  For now introduced to make `datalad ls` be relatively usable\n    # in terms of speed\n    _cached_subdatasets = {}\n\n    _params_ = dict(\n        loc=Parameter(\n            doc=\"URL or path to list, e.g. s3://...\",\n            metavar='PATH/URL',\n            nargs=\"*\",\n            constraints=EnsureStr() | EnsureNone(),\n        ),\n        recursive=Parameter(\n            args=(\"-r\", \"--recursive\"),\n            action=\"store_true\",\n            doc=\"recurse into subdirectories\",\n        ),\n        fast=Parameter(\n            args=(\"-F\", \"--fast\"),\n            action=\"store_true\",\n            doc=\"only perform fast operations.  Would be overridden by --all\",\n        ),\n        all_=Parameter(\n            args=(\"-a\", \"--all\"),\n            dest='all_',\n            action=\"store_true\",\n            doc=\"list all (versions of) entries, not e.g. only latest entries \"\n                \"in case of S3\",\n        ),\n        long_=Parameter(\n            args=(\"-L\", \"--long\"),\n            dest='long_',\n            action=\"store_true\",\n            doc=\"list more information on entries (e.g. acl, urls in s3, annex \"\n                \"sizes etc)\",\n        ),\n        config_file=Parameter(\n            doc=\"\"\"path to config file which could help the 'ls'.  E.g. for s3://\n            URLs could be some ~/.s3cfg file which would provide credentials\"\"\",\n            constraints=EnsureStr() | EnsureNone()\n        ),\n        list_content=Parameter(\n            choices=(None, 'first10', 'md5', 'full'),\n            doc=\"\"\"list also the content or only first 10 bytes (first10), or md5\n            checksum of an entry.  Might require expensive transfer and dump\n            binary output to your screen.  Do not enable unless you know what you\n            are after\"\"\",\n            default=None\n        ),\n        json=Parameter(\n            choices=('file', 'display', 'delete'),\n            doc=\"\"\"metadata json of dataset for creating web user interface.\n            display: prints jsons to stdout or\n            file: writes each subdir metadata to json file in subdir of dataset or\n            delete: deletes all metadata json files in dataset\"\"\",\n        ),\n    )\n\n    @staticmethod\n    def __call__(loc, recursive=False, fast=False, all_=False, long_=False,\n                 config_file=None, list_content=False, json=None):\n        if isinstance(loc, list) and not len(loc):\n            # nothing given, CWD assumed -- just like regular ls\n            loc = '.'\n\n        kw = dict(fast=fast, recursive=recursive, all_=all_, long_=long_)\n        if isinstance(loc, list):\n            return [Ls.__call__(loc_, config_file=config_file,\n                                list_content=list_content, json=json, **kw)\n                    for loc_ in loc]\n\n        # TODO: do some clever handling of kwargs as to remember what were defaults\n        # and what any particular implementation actually needs, and then issuing\n        # warning if some custom value/option was specified which doesn't apply to the\n        # given url\n\n        # rename to not angry Python gods who took all_ good words\n        kw['long_'] = kw.pop('long_')\n\n        loc_type = \"unknown\"\n        if loc.startswith('s3://'):\n            return _ls_s3(loc, config_file=config_file, list_content=list_content,\n                          **kw)\n        elif lexists(loc):\n            if isdir(loc):\n                ds = Dataset(loc)\n                if ds.is_installed():\n                    return _ls_json(loc, json=json, **kw) if json else _ls_dataset(loc, **kw)\n                    loc_type = False\n                else:\n                    loc_type = \"dir\"  # we know that so far for sure\n                    # it might have been an uninstalled dataset within super-dataset\n                    superds = ds.get_superdataset()\n                    if superds:\n                        try:\n                            subdatasets = Ls._cached_subdatasets[superds.path]\n                        except KeyError:\n                            subdatasets = Ls._cached_subdatasets[superds.path] \\\n                                = superds.subdatasets(result_xfm='relpaths')\n                        if relpath(ds.path, superds.path) in subdatasets:\n                            loc_type = \"not installed\"\n            else:\n                loc_type = \"file\"\n                # could list properties -- under annex or git, either clean/dirty\n                # etc\n                # repo = get_repo_instance(dirname(loc))\n\n        if loc_type:\n            #raise ValueError(\"ATM supporting only s3:// URLs and paths to local datasets\")\n            # TODO: unify all_ the output here -- _ls functions should just return something\n            # to be displayed\n            ui.message(\n                \"{}  {}\".format(\n                    ansi_colors.color_word(loc, ansi_colors.DATASET),\n                    ansi_colors.color_word(\n                        loc_type,\n                        ansi_colors.RED\n                        if loc_type in {'unknown', 'not installed'}\n                        else ansi_colors.BLUE)\n                )\n            )\n\n\n#\n# Dataset listing\n#\n\n@auto_repr\nclass AbsentRepoModel(object):\n    \"\"\"Just a base for those where repo wasn't installed yet\"\"\"\n\n    def __init__(self, path):\n        self.path = path\n        self.repo = None\n\n    @property\n    def type(self):\n        return \"N/A\"\n\n\n@auto_repr\nclass GitModel(object):\n    \"\"\"A base class for models which have some .repo available\"\"\"\n\n    __slots__ = ['_branch', 'repo', '_path']\n\n    def __init__(self, repo):\n        self.repo = repo\n        # lazy evaluation variables\n        self._branch = None\n        self._path = None\n\n    @property\n    def path(self):\n        return self.repo.path if self._path is None else self._path\n\n    @path.setter\n    def path(self, v):\n        self._path = v\n\n    @property\n    def branch(self):\n        if self._branch is None:\n            try:\n                self._branch = self.repo.get_active_branch()\n            except:  # MIH: InvalidGitRepositoryError?\n                return None\n        return self._branch\n\n    @property\n    def clean(self):\n        return not self.repo.dirty\n\n    @property\n    def describe(self):\n        return self.repo.describe(tags=True)\n\n    @property\n    def date(self):\n        \"\"\"Date of the last commit\n        \"\"\"\n        return self.repo.get_committed_date()\n\n    @property\n    def count_objects(self):\n        return self.repo.count_objects\n\n    @property\n    def git_local_size(self):\n        count_objects = self.count_objects\n        return count_objects['size'] if count_objects else None\n\n    @property\n    def type(self):\n        return {False: 'git', True: 'annex'}[isinstance(self.repo, AnnexRepo)]\n\n\n@auto_repr\nclass AnnexModel(GitModel):\n\n    __slots__ = ['_info'] + GitModel.__slots__\n\n    def __init__(self, *args, **kwargs):\n        super(AnnexModel, self).__init__(*args, **kwargs)\n        self._info = None\n\n    @property\n    def info(self):\n        if self._info is None and self.type == 'annex':\n            self._info = self.repo.repo_info()\n        return self._info\n\n    @property\n    def annex_worktree_size(self):\n        info = self.info\n        return info['size of annexed files in working tree'] if info else 0.0\n\n    @property\n    def annex_local_size(self):\n        info = self.info\n        return info['local annex size'] if info else 0.0\n\n\n@auto_repr\nclass FsModel(AnnexModel):\n\n    __slots__ = AnnexModel.__slots__\n\n    def __init__(self, path, *args, **kwargs):\n        super(FsModel, self).__init__(*args, **kwargs)\n        self._path = path\n\n    @property\n    def path(self):\n        return self._path\n\n    @property\n    def symlink(self):\n        \"\"\"if symlink returns path the symlink points to else returns None\"\"\"\n        if islink(self._path):                    # if symlink\n            target_path = realpath(self._path)    # find link target\n            # convert to absolute path if not\n            return target_path if exists(target_path) else None\n        return None\n\n    @property\n    def date(self):\n        \"\"\"Date of last modification\"\"\"\n        if self.type_ is not ['git', 'annex']:\n            return lstat(self._path).st_mtime\n        else:\n            super(self.__class__, self).date\n\n    @property\n    def size(self):\n        \"\"\"Size of the node computed based on its type\"\"\"\n        type_ = self.type_\n        sizes = {'total': 0.0, 'ondisk': 0.0, 'git': 0.0, 'annex': 0.0, 'annex_worktree': 0.0}\n\n        if type_ in ['file', 'link', 'link-broken']:\n            # if node is under annex, ask annex for node size, ondisk_size\n            if isinstance(self.repo, AnnexRepo) and self.repo.is_under_annex(self._path):\n                size = self.repo.info(self._path, batch=True)['size']\n                ondisk_size = size \\\n                    if self.repo.file_has_content(self._path) \\\n                    else 0\n            # else ask fs for node size (= ondisk_size)\n            else:\n                size = ondisk_size = 0 \\\n                    if type_ == 'link-broken' \\\n                    else lstat(self.symlink or self._path).st_size\n\n            sizes.update({'total': size, 'ondisk': ondisk_size})\n\n        if self.repo.path == self._path:\n            sizes.update({'git': self.git_local_size,\n                          'annex': self.annex_local_size,\n                          'annex_worktree': self.annex_worktree_size})\n        return sizes\n\n    @property\n    def type_(self):\n        \"\"\"outputs the node type\n\n        Types: link, link-broken, file, dir, annex-repo, git-repo\"\"\"\n        if islink(self.path):\n            return 'link' if self.symlink else 'link-broken'\n        elif isfile(self.path):\n            return 'file'\n        elif exists(opj(self.path, \".git\", \"annex\")):\n            return 'annex'\n        elif exists(opj(self.path, \".git\")):\n            return 'git'\n        elif isdir(self.path):\n            return 'dir'\n        else:\n            return None\n\n\nclass LsFormatter(string.Formatter):\n    # condition by interactive\n    if is_interactive():\n        BLUE = ansi_colors.COLOR_SEQ % ansi_colors.BLUE\n        RED = ansi_colors.COLOR_SEQ % ansi_colors.RED\n        GREEN = ansi_colors.COLOR_SEQ % ansi_colors.GREEN\n        RESET = ansi_colors.RESET_SEQ\n        DATASET = ansi_colors.COLOR_SEQ % ansi_colors.UNDERLINE\n    else:\n        BLUE = RED = GREEN = RESET = DATASET = u\"\"\n\n    # TODO: we might want to just ignore and force utf8 while explicitly .encode()'ing output!\n    # unicode versions which look better but which blow during tests etc\n    # Those might be reset by the constructor\n    OK = u\"\u2713\"\n    NOK = u\"\u2717\"\n    NONE = u\"\u2717\"\n\n    def __init__(self, *args, **kwargs):\n        super(LsFormatter, self).__init__(*args, **kwargs)\n        for setting_encoding in (sys.getdefaultencoding(),\n                                 sys.stdout.encoding):\n            try:\n                u\"\u2713\".encode(setting_encoding)\n            except UnicodeEncodeError:\n                lgr.debug(\"encoding %s found to not support unicode, resetting to safe alternatives\", setting_encoding)\n                self.OK = 'OK'   # u\"\u2713\"\n                self.NOK = 'X'  # u\"\u2717\"\n                self.NONE = '-'  # u\"\u2717\"\n                break\n\n\n    def convert_field(self, value, conversion):\n        #print(\"%r->%r\" % (value, conversion))\n        if conversion == 'D':  # Date\n            if value is not None:\n                return time.strftime(u\"%Y-%m-%d/%H:%M:%S\", time.localtime(value))\n            else:\n                return u'-'\n        elif conversion == 'S':  # Human size\n            #return value\n            if value is not None:\n                return humanize.naturalsize(value)\n            else:\n                return u'-'\n        elif conversion == 'X':  # colored bool\n            chr, col = (self.OK, self.GREEN) if value else (self.NOK, self.RED)\n            return u\"%s%s%s\" % (col, chr, self.RESET)\n        elif conversion == 'N':  # colored Red - if None\n            if value is None:\n                # return \"%s\u2716%s\" % (self.RED, self.RESET)\n                return u\"%s%s%s\" % (self.RED, self.NONE, self.RESET)\n            return value\n        elif conversion in {'B', 'R', 'U'}:\n            return u\"%s%s%s\" % ({'B': self.BLUE, 'R': self.RED, 'U': self.DATASET}[conversion], value, self.RESET)\n\n        return super(LsFormatter, self).convert_field(value, conversion)\n\n    def format_field(self, value, format_spec):\n        # TODO: move all the \"coloring\" into formatting, so we could correctly indent\n        # given the format and only then color it up\n        # print \"> %r, %r\" % (value, format_spec)\n        return super(LsFormatter, self).format_field(value, format_spec)\n\n\ndef format_ds_model(formatter, ds_model, format_str, format_exc):\n    try:\n        #print(\"WORKING ON %s\" % ds_model.path)\n        if not exists(ds_model.path) or not ds_model.repo:\n            return formatter.format(format_exc, ds=ds_model, msg=u\"not installed\")\n        ds_formatted = formatter.format(format_str, ds=ds_model)\n        #print(\"FINISHED ON %s\" % ds_model.path)\n        return ds_formatted\n    except Exception as exc:\n        return formatter.format(format_exc, ds=ds_model, msg=exc_str(exc))\n\n# from joblib import Parallel, delayed\n\n\ndef _ls_dataset(loc, fast=False, recursive=False, all_=False, long_=False):\n    isabs_loc = isabs(loc)\n    topdir = '' if isabs_loc else abspath(curdir)\n\n    topds = Dataset(loc)\n    dss = [topds] + (\n        [Dataset(opj(loc, sm))\n         for sm in topds.subdatasets(recursive=recursive, result_xfm='relpaths')]\n        if recursive else [])\n\n    dsms = []\n    for ds in dss:\n        if not ds.is_installed():\n            dsm = AbsentRepoModel(ds.path)\n        elif isinstance(ds.repo, AnnexRepo):\n            dsm = AnnexModel(ds.repo)\n        elif isinstance(ds.repo, GitRepo):\n            dsm = GitModel(ds.repo)\n        else:\n            raise RuntimeError(\"Got some dataset which don't know how to handle %s\"\n                               % ds)\n        dsms.append(dsm)\n\n    # adjust path strings\n    for ds_model in dsms:\n        #path = ds_model.path[len(topdir) + 1 if topdir else 0:]\n        path = relpath(ds_model.path, topdir) if topdir else ds_model.path\n        if not path:\n            path = '.'\n        ds_model.path = path\n    dsms = sorted(dsms, key=lambda m: m.path)\n\n    maxpath = max(len(ds_model.path) for ds_model in dsms)\n    path_fmt = u\"{ds.path!U:<%d}\" % (maxpath + (11 if is_interactive() else 0))  # + to accommodate ansi codes\n    pathtype_fmt = path_fmt + u\"  [{ds.type}]\"\n    full_fmt = pathtype_fmt + u\"  {ds.branch!N}  {ds.describe!N} {ds.date!D}\"\n    if (not fast) or long_:\n        full_fmt += u\"  {ds.clean!X}\"\n\n    fmts = {\n        AbsentRepoModel: pathtype_fmt,\n        GitModel: full_fmt,\n        AnnexModel: full_fmt\n    }\n    if long_:\n        fmts[AnnexModel] += u\"  {ds.annex_local_size!S}/{ds.annex_worktree_size!S}\"\n\n    formatter = LsFormatter()\n    # weird problems happen in the parallel run -- TODO - figure it out\n    # for out in Parallel(n_jobs=1)(\n    #         delayed(format_ds_model)(formatter, dsm, full_fmt, format_exc=path_fmt + \"  {msg!R}\")\n    #         for dsm in dss):\n    #     print(out)\n    for dsm in dsms:\n        fmt = fmts[dsm.__class__]\n        ds_str = format_ds_model(formatter, dsm, fmt, format_exc=path_fmt + u\"  {msg!R}\")\n        safe_print(ds_str)\n        # workaround for explosion of git cat-file --batch processes\n        # https://github.com/datalad/datalad/issues/1888\n        if dsm.repo is not None:\n            dsm.repo.repo.close()\n            del dsm.repo\n            dsm.repo = None\n\n\ndef machinesize(humansize):\n    \"\"\"convert human-size string to machine-size\"\"\"\n    try:\n        size_str, size_unit = humansize.split(\" \")\n    except AttributeError:\n        return float(humansize)\n    unit_converter = {'Byte': 0, 'Bytes': 0, 'kB': 1, 'MB': 2, 'GB': 3, 'TB': 4, 'PB': 5}\n    machinesize = float(size_str) * (1000 ** unit_converter[size_unit])\n    return machinesize\n\n\ndef leaf_name(path):\n    \"\"\"takes a relative or absolute path and returns name of node at that location\"\"\"\n    head, tail = split(abspath(path))\n    return tail or basename(head)\n\n\ndef ignored(path, only_hidden=False):\n    \"\"\"if path is in the ignorelist return True\n\n    ignore list includes hidden files and git or annex maintained folders\n    when only_hidden set, only ignores hidden files and folders not git or annex maintained folders\n    \"\"\"\n    if isdir(opj(path, \".git\")) and not only_hidden:\n        return True\n    return '.' == leaf_name(path)[0] or leaf_name(path) == 'index.html'\n\n\ndef metadata_locator(fs_metadata=None, path=None, ds_path=None, metadata_path=None):\n    \"\"\"path to metadata file of node associated with the fs_metadata dictionary\n\n    Parameters\n    ----------\n    fs_metadata: dict\n      Metadata json of a node\n    path: str\n      Path to directory of metadata to be rendered\n    ds_path: str\n      Path to dataset root\n    metadata_path: str\n      Path to metadata root. Calculated relative to ds_path\n\n    Returns\n    -------\n    str\n      path to metadata of current node\n    \"\"\"\n\n    # use implicit paths unless paths explicitly specified\n    # Note: usage of ds_path as if it was the Repo's path. Therefore use\n    # realpath, since we switched to have symlinks resolved in repos but not in\n    # datasets\n    ds_path = realpath(ds_path) if ds_path else fs_metadata['repo']\n    path = path or fs_metadata['path']\n    metadata_path = metadata_path or '.git/datalad/metadata'\n    # directory metadata directory tree location\n    metadata_dir = opj(ds_path, metadata_path)\n    # relative path of current directory wrt dataset root\n    dir_path = relpath(path, ds_path) if isabs(path) else path\n    # normalize to / -- TODO, switch to '.' which is now actually the name since path is relative in web meta?\n    if dir_path in ('.', None, ''):\n        dir_path = '/'\n    # create md5 hash of current directory's relative path\n    metadata_hash = hashlib.md5(dir_path.encode('utf-8')).hexdigest()\n    # construct final path to metadata file\n    metadata_file = opj(metadata_dir, metadata_hash)\n\n    return metadata_file\n\n\ndef fs_extract(nodepath, repo, basepath='/'):\n    \"\"\"extract required info of nodepath with its associated parent repository and returns it as a dictionary\n\n    Parameters\n    ----------\n    nodepath : str\n        Full path to the location we are exploring (must be a directory within\n        `repo`\n    repo : GitRepo\n        Is the repository nodepath belongs to\n    \"\"\"\n    # Create FsModel from filesystem nodepath and its associated parent repository\n    node = FsModel(nodepath, repo)\n    pretty_size = {stype: humanize.naturalsize(svalue) for stype, svalue in node.size.items()}\n    pretty_date = time.strftime(u\"%Y-%m-%d %H:%M:%S\", time.localtime(node.date))\n    name = leaf_name(node._path) if leaf_name(node._path) != \"\" else leaf_name(node.repo.path)\n    rec = {\n        \"name\": name, \"path\": relpath(node._path, basepath),\n        \"type\": node.type_, \"size\": pretty_size, \"date\": pretty_date,\n    }\n    # if there is meta-data for the dataset (done by aggregate-metadata)\n    # we include it\n    metadata_path = opj(nodepath, METADATA_DIR, METADATA_FILENAME)\n    if exists(metadata_path):\n        # might need flattening!  TODO: flatten when aggregating?  why wasn't done?\n        metadata = js.load(open(metadata_path))\n        # might be too heavy to carry around, so will do basic flattening manually\n        # and in a basic fashion\n        # import jsonld\n        metadata_reduced = metadata[0]\n        for m in metadata[1:]:\n            metadata_reduced.update(m)\n        # but verify that they all had the same id\n        if metadata:\n            metaid = metadata[0]['@id']\n            assert all(m['@id'] == metaid for m in metadata)\n        rec[\"metadata\"] = metadata_reduced\n    return rec\n\n\ndef fs_render(fs_metadata, json=None, **kwargs):\n    \"\"\"takes node to render and based on json option passed renders to file, stdout or deletes json at root\n\n    Parameters\n    ----------\n    fs_metadata: dict\n      Metadata json to be rendered\n    json: str ('file', 'display', 'delete')\n      Render to file, stdout or delete json\n    \"\"\"\n\n    metadata_file = metadata_locator(fs_metadata, **kwargs)\n\n    if json == 'file':\n        # create metadata_root directory if it doesn't exist\n        metadata_dir = dirname(metadata_file)\n        if not exists(metadata_dir):\n            makedirs(metadata_dir)\n        # write directory metadata to json\n        with open(metadata_file, 'w') as f:\n            js.dump(fs_metadata, f)\n\n    # else if json flag set to delete, remove .dir.json of current directory\n    elif json == 'delete' and exists(metadata_file):\n        remove(metadata_file)\n\n    # else dump json to stdout\n    elif json == 'display':\n        safe_print(js.dumps(fs_metadata) + '\\n')\n\n\ndef fs_traverse(path, repo, parent=None, render=True, recursive=False, json=None, basepath=None):\n    \"\"\"Traverse path through its nodes and returns a dictionary of relevant attributes attached to each node\n\n    Parameters\n    ----------\n    path: str\n      Path to the directory to be traversed\n    repo: AnnexRepo or GitRepo\n      Repo object the directory belongs too\n    parent: dict\n      Extracted info about parent directory\n    recursive: bool\n      Recurse into subdirectories (note that subdatasets are not traversed)\n    render: bool\n       To render from within function or not. Set to false if results to be manipulated before final render\n\n    Returns\n    -------\n    list of dict\n      extracts and returns a (recursive) list of directory info at path\n      does not traverse into annex, git or hidden directories\n    \"\"\"\n    fs = fs_extract(path, repo, basepath=basepath or path)\n    if isdir(path):                     # if node is a directory\n        children = [fs.copy()]          # store its info in its children dict too  (Yarik is not sure why, but I guess for .?)\n        # ATM seems some pieces still rely on having this duplication, so left as is\n        # TODO: strip away\n        for node in listdir(path):\n            nodepath = opj(path, node)\n\n            # TODO:  it might be a subdir which is non-initialized submodule!\n            # if not ignored, append child node info to current nodes dictionary\n            if not ignored(nodepath):\n                # if recursive, create info dictionary (within) each child node too\n                if recursive:\n                    subdir = fs_traverse(nodepath,\n                                         repo,\n                                         parent=None,  # children[0],\n                                         recursive=recursive,\n                                         json=json,\n                                         basepath=basepath or path)\n                    subdir.pop('nodes', None)\n                else:\n                    # read child metadata from its metadata file if it exists\n                    subdir_json = metadata_locator(path=node, ds_path=basepath or path)\n                    if exists(subdir_json):\n                        with open(subdir_json) as data_file:\n                            subdir = js.load(data_file)\n                            subdir.pop('nodes', None)\n                    # else extract whatever information you can about the child\n                    else:\n                        # Yarik: this one is way too lean...\n                        subdir = fs_extract(nodepath,\n                                            repo,\n                                            basepath=basepath or path)\n                # append child metadata to list\n                children.extend([subdir])\n\n        # sum sizes of all 1st level children\n        children_size = {}\n        for node in children[1:]:\n            for size_type, child_size in node['size'].items():\n                children_size[size_type] = children_size.get(size_type, 0) + machinesize(child_size)\n\n        # update current node sizes to the humanized aggregate children size\n        fs['size'] = children[0]['size'] = \\\n            {size_type: humanize.naturalsize(child_size)\n             for size_type, child_size in children_size.items()}\n\n        children[0]['name'] = '.'       # replace current node name with '.' to emulate unix syntax\n        if parent:\n            parent['name'] = '..'       # replace parent node name with '..' to emulate unix syntax\n            children.insert(1, parent)  # insert parent info after current node info in children dict\n\n        fs['nodes'] = children          # add children info to main fs dictionary\n        if render:                      # render directory node at location(path)\n            fs_render(fs, json=json, ds_path=basepath or path)\n            lgr.info('Directory: %s' % path)\n\n    return fs\n\n\ndef ds_traverse(rootds, parent=None, json=None, recursive=False, all_=False,\n                long_=False):\n    \"\"\"Hierarchical dataset traverser\n\n    Parameters\n    ----------\n    rootds: Dataset\n      Root dataset to be traversed\n    parent: Dataset\n      Parent dataset of the current rootds\n    recursive: bool\n       Recurse into subdirectories of the current dataset\n    all_: bool\n       Recurse into subdatasets of the root dataset\n\n    Returns\n    -------\n    list of dict\n      extracts and returns a (recursive) list of dataset(s) info at path\n    \"\"\"\n    # extract parent info to pass to traverser\n    fsparent = fs_extract(parent.path, parent.repo, basepath=rootds.path) if parent else None\n\n    # (recursively) traverse file tree of current dataset\n    fs = fs_traverse(rootds.path, rootds.repo,\n                     render=False, parent=fsparent, recursive=all_,\n                     json=json)\n    size_list = [fs['size']]\n\n    # (recursively) traverse each subdataset\n    children = []\n    # yoh: was in return results branch returning full datasets:\n    # for subds in rootds.subdatasets(result_xfm='datasets'):\n    # but since rpath is needed/used, decided to return relpaths\n    for subds_rpath in rootds.subdatasets(result_xfm='relpaths'):\n\n        subds_path = opj(rootds.path, subds_rpath)\n        subds = Dataset(subds_path)\n        subds_json = metadata_locator(path='.', ds_path=subds_path)\n\n        def handle_not_installed():\n            # for now just traverse as fs\n            lgr.warning(\"%s is either not installed or lacks meta-data\", subds)\n            subfs = fs_extract(subds_path, rootds, basepath=rootds.path)\n            # but add a custom type that it is a not installed subds\n            subfs['type'] = 'uninitialized'\n            # we need to kick it out from 'children'\n            # TODO:  this is inefficient and cruel -- \"ignored\" should be made\n            # smarted to ignore submodules for the repo\n            if fs['nodes']:\n                fs['nodes'] = [c for c in fs['nodes'] if c['path'] != subds_rpath]\n            return subfs\n\n        if not subds.is_installed():\n            subfs = handle_not_installed()\n        elif recursive:\n            subfs = ds_traverse(subds,\n                                json=json,\n                                recursive=recursive,\n                                all_=all_,\n                                parent=rootds)\n            subfs.pop('nodes', None)\n            size_list.append(subfs['size'])\n        # else just pick the data from metadata_file of each subdataset\n        else:\n            lgr.info(subds.path)\n            if exists(subds_json):\n                with open(subds_json) as data_file:\n                    subfs = js.load(data_file)\n                    subfs.pop('nodes', None)    # remove children\n                    subfs['path'] = subds_rpath # reassign the path\n                    size_list.append(subfs['size'])\n            else:\n                # the same drill as if not installed\n                lgr.warning(\"%s is installed but no meta-data yet\", subds)\n                subfs = handle_not_installed()\n\n        children.extend([subfs])\n\n    # sum sizes of all 1st level children dataset\n    children_size = {}\n    for subdataset_size in size_list:\n        for size_type, subds_size in subdataset_size.items():\n            children_size[size_type] = children_size.get(size_type, 0) + machinesize(subds_size)\n\n    # update current dataset sizes to the humanized aggregate subdataset sizes\n    fs['size'] = {size_type: humanize.naturalsize(size)\n                  for size_type, size in children_size.items()}\n    fs['nodes'][0]['size'] = fs['size']  # update self's updated size in nodes sublist too!\n\n    # add dataset specific entries to its dict\n    rootds_model = GitModel(rootds.repo)\n    fs['tags'] = rootds_model.describe\n    fs['branch'] = rootds_model.branch\n    index_file = opj(rootds.path, '.git', 'index')\n    fs['index-mtime'] = time.strftime(\n        u\"%Y-%m-%d %H:%M:%S\",\n        time.localtime(getmtime(index_file))) if exists(index_file) else ''\n\n    # append children datasets info to current dataset\n    fs['nodes'].extend(children)\n\n    # render current dataset\n    lgr.info('Dataset: %s' % rootds.path)\n    fs_render(fs, json=json, ds_path=rootds.path)\n    return fs\n\n\ndef _ls_json(loc, fast=False, **kwargs):\n    # hierarchically traverse file tree of (sub-)dataset(s) under path passed(loc)\n    return ds_traverse(Dataset(loc), parent=None, **kwargs)\n\n\n#\n# S3 listing\n#\ndef _ls_s3(loc, fast=False, recursive=False, all_=False, long_=False,\n           config_file=None, list_content=False):\n    \"\"\"List S3 bucket content\"\"\"\n    if loc.startswith('s3://'):\n        bucket_prefix = loc[5:]\n    else:\n        raise ValueError(\"passed location should be an s3:// url\")\n\n    import boto\n    from hashlib import md5\n    from boto.s3.key import Key\n    from boto.s3.prefix import Prefix\n    from boto.exception import S3ResponseError\n    from ..support.configparserinc import SafeConfigParser  # provides PY2,3 imports\n\n    if '/' in bucket_prefix:\n        bucket_name, prefix = bucket_prefix.split('/', 1)\n    else:\n        bucket_name, prefix = bucket_prefix, None\n\n    if prefix and '?' in prefix:\n        ui.message(\"We do not care about URL options ATM, they get stripped\")\n        prefix = prefix[:prefix.index('?')]\n\n    ui.message(\"Connecting to bucket: %s\" % bucket_name)\n    if config_file:\n        config = SafeConfigParser()\n        config.read(config_file)\n        access_key = config.get('default', 'access_key')\n        secret_key = config.get('default', 'secret_key')\n\n        # TODO: remove duplication -- reuse logic within downloaders/s3.py to get connected\n        conn = boto.connect_s3(access_key, secret_key)\n        try:\n            bucket = conn.get_bucket(bucket_name)\n        except S3ResponseError as e:\n            ui.message(\"E: Cannot access bucket %s by name\" % bucket_name)\n            all_buckets = conn.get_all_buckets()\n            all_bucket_names = [b.name for b in all_buckets]\n            ui.message(\"I: Found following buckets %s\" % ', '.join(all_bucket_names))\n            if bucket_name in all_bucket_names:\n                bucket = all_buckets[all_bucket_names.index(bucket_name)]\n            else:\n                raise RuntimeError(\"E: no bucket named %s thus exiting\" % bucket_name)\n    else:\n        # TODO: expose credentials\n        # We don't need any provider here really but only credentials\n        from datalad.downloaders.providers import Providers\n        providers = Providers.from_config_files()\n        provider = providers.get_provider(loc)\n\n        if not provider:\n            raise ValueError(\n                \"Don't know how to deal with this url %s -- no provider defined for %s. \"\n                \"Define a new provider (DOCS: TODO) or specify just s3cmd config file instead for now.\"\n                % loc\n            )\n        downloader = provider.get_downloader(loc)\n\n        # should authenticate etc, and when ready we will ask for a bucket ;)\n        bucket = downloader.access(lambda url: downloader.bucket, loc)\n\n    info = []\n    for iname, imeth in [\n        (\"Versioning\", bucket.get_versioning_status),\n        (\"   Website\", bucket.get_website_endpoint),\n        (\"       ACL\", bucket.get_acl),\n    ]:\n        try:\n            ival = imeth()\n        except Exception as e:\n            ival = str(e).split('\\n')[0]\n        info.append(\" {iname}: {ival}\".format(**locals()))\n    ui.message(\"Bucket info:\\n %s\" % '\\n '.join(info))\n\n    kwargs = {} if recursive else {'delimiter': '/'}\n\n    ACCESS_METHODS = [\n        bucket.list_versions,\n        bucket.list\n    ]\n\n    prefix_all_versions = None\n    for acc in ACCESS_METHODS:\n        try:\n            prefix_all_versions = list(acc(prefix, **kwargs))\n            break\n        except Exception as exc:\n            lgr.debug(\"Failed to access via %s: %s\", acc, exc_str(exc))\n\n    if not prefix_all_versions:\n        ui.error(\"No output was provided for prefix %r\" % prefix)\n    else:\n        max_length = max((len(e.name) for e in prefix_all_versions))\n        max_size_length = max((len(str(getattr(e, 'size', 0))) for e in prefix_all_versions))\n\n    results = []\n    for e in prefix_all_versions:\n        results.append(e)\n        if isinstance(e, Prefix):\n            ui.message(\"%s\" % (e.name, ),)\n            continue\n\n        base_msg = (\"%%-%ds %%s\" % max_length) % (e.name, e.last_modified)\n        if isinstance(e, Key):\n            if not (e.is_latest or all_):\n                # Skip this one\n                continue\n            ui.message(base_msg + \" %%%dd\" % max_size_length % e.size, cr=' ')\n            # OPT: delayed import\n            from ..support.s3 import get_key_url\n            url = get_key_url(e, schema='http')\n            try:\n                _ = urlopen(Request(url))\n                urlok = \"OK\"\n            except HTTPError as err:\n                urlok = \"E: %s\" % err.code\n\n            try:\n                acl = e.get_acl()\n            except S3ResponseError as exc:\n                acl = exc.code if exc.code in ('AccessDenied',) else str(exc)\n\n            content = \"\"\n            if list_content:\n                # IO intensive, make an option finally!\n                try:\n                    # _ = e.next()[:5]  if we are able to fetch the content\n                    kwargs = dict(version_id=e.version_id)\n                    if list_content in {'full', 'first10'}:\n                        if list_content in 'first10':\n                            kwargs['headers'] = {'Range': 'bytes=0-9'}\n                        content = repr(e.get_contents_as_string(**kwargs))\n                    elif list_content == 'md5':\n                        digest = md5()\n                        digest.update(e.get_contents_as_string(**kwargs))\n                        content = digest.hexdigest()\n                    else:\n                        raise ValueError(list_content)\n                    # content = \"[S3: OK]\"\n                except S3ResponseError as err:\n                    content = str(err)\n                finally:\n                    content = \" \" + content\n            ui.message(\n                \"ver:%-32s  acl:%s  %s [%s]%s\"\n                % (getattr(e, 'version_id', None),\n                   acl, url, urlok, content)\n                if long_ else ''\n            )\n        else:\n            ui.message(base_msg + \" \" + str(type(e)).split('.')[-1].rstrip(\"\\\"'>\"))\n    return results\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** TypeError Occurs When `sys.stdout.encoding` Is None in Python 2

**Description:**
When using the `ls` interface in an environment where `stdout` is piped in Python 2, `sys.stdout.encoding` returns `None`. The current implementation attempts to encode unicode characters based on `sys.getdefaultencoding`, which is `'ascii'` in Python 2. This results in a `TypeError` because the `encode` method receives `None` instead of a valid encoding string.

**Example Buggy Code:**
```python
import sys

class LsFormatter(object):
    OK = u"✓"
    
    def __init__(self):
        # sys.stdout.encoding is None when stdout is piped
        self.OK.encode(sys.stdout.encoding)

formatter = LsFormatter()
```

**Expected Behavior:**
The formatter should handle cases where `sys.stdout.encoding` is `None` gracefully, such as by disabling unicode symbols or using a safe default encoding, instead of raising an error.

**Actual Behavior:**
A `TypeError` is raised with the message: `encode() argument 1 must be str, not None`.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/interface/ls.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil; coding: utf-8 -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Helper utility to list things.  ATM list content of S3 bucket
"""

__docformat__ = 'restructuredtext'

import sys
import time
from os.path import exists, lexists, join as opj, abspath, isabs, getmtime
from os.path import curdir, isfile, islink, isdir, dirname, basename, split, realpath
from os.path import relpath
from os import listdir, lstat, remove, makedirs
import json as js
import hashlib

from six.moves.urllib.request import urlopen, Request
from six.moves.urllib.error import HTTPError

from ..cmdline.helpers import get_repo_instance
from ..utils import auto_repr
from .base import Interface
from datalad.interface.base import build_doc
from ..ui import ui
from ..utils import safe_print
from ..consts import METADATA_DIR
from ..consts import METADATA_FILENAME
from ..dochelpers import exc_str
from ..support.param import Parameter
from ..support import ansi_colors
from ..support.constraints import EnsureStr, EnsureNone
from ..distribution.dataset import Dataset

from datalad.support.annexrepo import AnnexRepo
from datalad.support.annexrepo import GitRepo

import string
import humanize
from datalad.utils import is_interactive

from logging import getLogger
lgr = getLogger('datalad.api.ls')


@build_doc
class Ls(Interface):
    """List summary information about URLs and dataset(s)

    ATM only s3:// URLs and datasets are supported

    Examples:

      $ datalad ls s3://openfmri/tarballs/ds202  # to list S3 bucket
      $ datalad ls                               # to list current dataset
    """
    # XXX prevent common args from being added to the docstring
    _no_eval_results = True

    # TODO: during big RF refactor this one away since it must not be instance's
    # attribute.  For now introduced to make `datalad ls` be relatively usable
    # in terms of speed
    _cached_subdatasets = {}

    _params_ = dict(
        loc=Parameter(
            doc="URL or path to list, e.g. s3://...",
            metavar='PATH/URL',
            nargs="*",
            constraints=EnsureStr() | EnsureNone(),
        ),
        recursive=Parameter(
            args=("-r", "--recursive"),
            action="store_true",
            doc="recurse into subdirectories",
        ),
        fast=Parameter(
            args=("-F", "--fast"),
            action="store_true",
            doc="only perform fast operations.  Would be overridden by --all",
        ),
        all_=Parameter(
            args=("-a", "--all"),
            dest='all_',
            action="store_true",
            doc="list all (versions of) entries, not e.g. only latest entries "
                "in case of S3",
        ),
        long_=Parameter(
            args=("-L", "--long"),
            dest='long_',
            action="store_true",
            doc="list more information on entries (e.g. acl, urls in s3, annex "
                "sizes etc)",
        ),
        config_file=Parameter(
            doc="""path to config file which could help the 'ls'.  E.g. for s3://
            URLs could be some ~/.s3cfg file which would provide credentials""",
            constraints=EnsureStr() | EnsureNone()
        ),
        list_content=Parameter(
            choices=(None, 'first10', 'md5', 'full'),
            doc="""list also the content or only first 10 bytes (first10), or md5
            checksum of an entry.  Might require expensive transfer and dump
            binary output to your screen.  Do not enable unless you know what you
            are after""",
            default=None
        ),
        json=Parameter(
            choices=('file', 'display', 'delete'),
            doc="""metadata json of dataset for creating web user interface.
            display: prints jsons to stdout or
            file: writes each subdir metadata to json file in subdir of dataset or
            delete: deletes all metadata json files in dataset""",
        ),
    )

    @staticmethod
    def __call__(loc, recursive=False, fast=False, all_=False, long_=False,
                 config_file=None, list_content=False, json=None):
        if isinstance(loc, list) and not len(loc):
            # nothing given, CWD assumed -- just like regular ls
            loc = '.'

        kw = dict(fast=fast, recursive=recursive, all_=all_, long_=long_)
        if isinstance(loc, list):
            return [Ls.__call__(loc_, config_file=config_file,
                                list_content=list_content, json=json, **kw)
                    for loc_ in loc]

        # TODO: do some clever handling of kwargs as to remember what were defaults
        # and what any particular implementation actually needs, and then issuing
        # warning if some custom value/option was specified which doesn't apply to the
        # given url

        # rename to not angry Python gods who took all_ good words
        kw['long_'] = kw.pop('long_')

        loc_type = "unknown"
        if loc.startswith('s3://'):
            return _ls_s3(loc, config_file=config_file, list_content=list_content,
                          **kw)
        elif lexists(loc):
            if isdir(loc):
                ds = Dataset(loc)
                if ds.is_installed():
                    return _ls_json(loc, json=json, **kw) if json else _ls_dataset(loc, **kw)
                    loc_type = False
                else:
                    loc_type = "dir"  # we know that so far for sure
                    # it might have been an uninstalled dataset within super-dataset
                    superds = ds.get_superdataset()
                    if superds:
                        try:
                            subdatasets = Ls._cached_subdatasets[superds.path]
                        except KeyError:
                            subdatasets = Ls._cached_subdatasets[superds.path] \
                                = superds.subdatasets(result_xfm='relpaths')
                        if relpath(ds.path, superds.path) in subdatasets:
                            loc_type = "not installed"
            else:
                loc_type = "file"
                # could list properties -- under annex or git, either clean/dirty
                # etc
                # repo = get_repo_instance(dirname(loc))

        if loc_type:
            #raise ValueError("ATM supporting only s3:// URLs and paths to local datasets")
            # TODO: unify all_ the output here -- _ls functions should just return something
            # to be displayed
            ui.message(
                "{}  {}".format(
                    ansi_colors.color_word(loc, ansi_colors.DATASET),
                    ansi_colors.color_word(
                        loc_type,
                        ansi_colors.RED
                        if loc_type in {'unknown', 'not installed'}
                        else ansi_colors.BLUE)
                )
            )


#
# Dataset listing
#

@auto_repr
class AbsentRepoModel(object):
    """Just a base for those where repo wasn't installed yet"""

    def __init__(self, path):
        self.path = path
        self.repo = None

    @property
    def type(self):
        return "N/A"


@auto_repr
class GitModel(object):
    """A base class for models which have some .repo available"""

    __slots__ = ['_branch', 'repo', '_path']

    def __init__(self, repo):
        self.repo = repo
        # lazy evaluation variables
        self._branch = None
        self._path = None

    @property
    def path(self):
        return self.repo.path if self._path is None else self._path

    @path.setter
    def path(self, v):
        self._path = v

    @property
    def branch(self):
        if self._branch is None:
            try:
                self._branch = self.repo.get_active_branch()
            except:  # MIH: InvalidGitRepositoryError?
                return None
        return self._branch

    @property
    def clean(self):
        return not self.repo.dirty

    @property
    def describe(self):
        return self.repo.describe(tags=True)

    @property
    def date(self):
        """Date of the last commit
        """
        return self.repo.get_committed_date()

    @property
    def count_objects(self):
        return self.repo.count_objects

    @property
    def git_local_size(self):
        count_objects = self.count_objects
        return count_objects['size'] if count_objects else None

    @property
    def type(self):
        return {False: 'git', True: 'annex'}[isinstance(self.repo, AnnexRepo)]


@auto_repr
class AnnexModel(GitModel):

    __slots__ = ['_info'] + GitModel.__slots__

    def __init__(self, *args, **kwargs):
        super(AnnexModel, self).__init__(*args, **kwargs)
        self._info = None

    @property
    def info(self):
        if self._info is None and self.type == 'annex':
            self._info = self.repo.repo_info()
        return self._info

    @property
    def annex_worktree_size(self):
        info = self.info
        return info['size of annexed files in working tree'] if info else 0.0

    @property
    def annex_local_size(self):
        info = self.info
        return info['local annex size'] if info else 0.0


@auto_repr
class FsModel(AnnexModel):

    __slots__ = AnnexModel.__slots__

    def __init__(self, path, *args, **kwargs):
        super(FsModel, self).__init__(*args, **kwargs)
        self._path = path

    @property
    def path(self):
        return self._path

    @property
    def symlink(self):
        """if symlink returns path the symlink points to else returns None"""
        if islink(self._path):                    # if symlink
            target_path = realpath(self._path)    # find link target
            # convert to absolute path if not
            return target_path if exists(target_path) else None
        return None

    @property
    def date(self):
        """Date of last modification"""
        if self.type_ is not ['git', 'annex']:
            return lstat(self._path).st_mtime
        else:
            super(self.__class__, self).date

    @property
    def size(self):
        """Size of the node computed based on its type"""
        type_ = self.type_
        sizes = {'total': 0.0, 'ondisk': 0.0, 'git': 0.0, 'annex': 0.0, 'annex_worktree': 0.0}

        if type_ in ['file', 'link', 'link-broken']:
            # if node is under annex, ask annex for node size, ondisk_size
            if isinstance(self.repo, AnnexRepo) and self.repo.is_under_annex(self._path):
                size = self.repo.info(self._path, batch=True)['size']
                ondisk_size = size \
                    if self.repo.file_has_content(self._path) \
                    else 0
            # else ask fs for node size (= ondisk_size)
            else:
                size = ondisk_size = 0 \
                    if type_ == 'link-broken' \
                    else lstat(self.symlink or self._path).st_size

            sizes.update({'total': size, 'ondisk': ondisk_size})

        if self.repo.path == self._path:
            sizes.update({'git': self.git_local_size,
                          'annex': self.annex_local_size,
                          'annex_worktree': self.annex_worktree_size})
        return sizes

    @property
    def type_(self):
        """outputs the node type

        Types: link, link-broken, file, dir, annex-repo, git-repo"""
        if islink(self.path):
            return 'link' if self.symlink else 'link-broken'
        elif isfile(self.path):
            return 'file'
        elif exists(opj(self.path, ".git", "annex")):
            return 'annex'
        elif exists(opj(self.path, ".git")):
            return 'git'
        elif isdir(self.path):
            return 'dir'
        else:
            return None


class LsFormatter(string.Formatter):
    # condition by interactive
    if is_interactive():
        BLUE = ansi_colors.COLOR_SEQ % ansi_colors.BLUE
        RED = ansi_colors.COLOR_SEQ % ansi_colors.RED
        GREEN = ansi_colors.COLOR_SEQ % ansi_colors.GREEN
        RESET = ansi_colors.RESET_SEQ
        DATASET = ansi_colors.COLOR_SEQ % ansi_colors.UNDERLINE
    else:
        BLUE = RED = GREEN = RESET = DATASET = u""

    # TODO: we might want to just ignore and force utf8 while explicitly .encode()'ing output!
    # unicode versions which look better but which blow during tests etc
    # Those might be reset by the constructor
    OK = u"✓"
    NOK = u"✗"
    NONE = u"✗"

    def __init__(self, *args, **kwargs):
        super(LsFormatter, self).__init__(*args, **kwargs)
        for setting_encoding in (sys.getdefaultencoding(),
                                 sys.stdout.encoding):
            try:
                u"✓".encode(setting_encoding)
            except UnicodeEncodeError:
                lgr.debug("encoding %s found to not support unicode, resetting to safe alternatives", setting_encoding)
                self.OK = 'OK'   # u"✓"
                self.NOK = 'X'  # u"✗"
                self.NONE = '-'  # u"✗"
                break


    def convert_field(self, value, conversion):
        #print("%r->%r" % (value, conversion))
        if conversion == 'D':  # Date
            if value is not None:
                return time.strftime(u"%Y-%m-%d/%H:%M:%S", time.localtime(value))
            else:
                return u'-'
        elif conversion == 'S':  # Human size
            #return value
            if value is not None:
                return humanize.naturalsize(value)
            else:
                return u'-'
        elif conversion == 'X':  # colored bool
            chr, col = (self.OK, self.GREEN) if value else (self.NOK, self.RED)
            return u"%s%s%s" % (col, chr, self.RESET)
        elif conversion == 'N':  # colored Red - if None
            if value is None:
                # return "%s✖%s" % (self.RED, self.RESET)
                return u"%s%s%s" % (self.RED, self.NONE, self.RESET)
            return value
        elif conversion in {'B', 'R', 'U'}:
            return u"%s%s%s" % ({'B': self.BLUE, 'R': self.RED, 'U': self.DATASET}[conversion], value, self.RESET)

        return super(LsFormatter, self).convert_field(value, conversion)

    def format_field(self, value, format_spec):
        # TODO: move all the "coloring" into formatting, so we could correctly indent
        # given the format and only then color it up
        # print "> %r, %r" % (value, format_spec)
        return super(LsFormatter, self).format_field(value, format_spec)


def format_ds_model(formatter, ds_model, format_str, format_exc):
    try:
        #print("WORKING ON %s" % ds_model.path)
        if not exists(ds_model.path) or not ds_model.repo:
            return formatter.format(format_exc, ds=ds_model, msg=u"not installed")
        ds_formatted = formatter.format(format_str, ds=ds_model)
        #print("FINISHED ON %s" % ds_model.path)
        return ds_formatted
    except Exception as exc:
        return formatter.format(format_exc, ds=ds_model, msg=exc_str(exc))

# from joblib import Parallel, delayed


def _ls_dataset(loc, fast=False, recursive=False, all_=False, long_=False):
    isabs_loc = isabs(loc)
    topdir = '' if isabs_loc else abspath(curdir)

    topds = Dataset(loc)
    dss = [topds] + (
        [Dataset(opj(loc, sm))
         for sm in topds.subdatasets(recursive=recursive, result_xfm='relpaths')]
        if recursive else [])

    dsms = []
    for ds in dss:
        if not ds.is_installed():
            dsm = AbsentRepoModel(ds.path)
        elif isinstance(ds.repo, AnnexRepo):
            dsm = AnnexModel(ds.repo)
        elif isinstance(ds.repo, GitRepo):
            dsm = GitModel(ds.repo)
        else:
            raise RuntimeError("Got some dataset which don't know how to handle %s"
                               % ds)
        dsms.append(dsm)

    # adjust path strings
    for ds_model in dsms:
        #path = ds_model.path[len(topdir) + 1 if topdir else 0:]
        path = relpath(ds_model.path, topdir) if topdir else ds_model.path
        if not path:
            path = '.'
        ds_model.path = path
    dsms = sorted(dsms, key=lambda m: m.path)

    maxpath = max(len(ds_model.path) for ds_model in dsms)
    path_fmt = u"{ds.path!U:<%d}" % (maxpath + (11 if is_interactive() else 0))  # + to accommodate ansi codes
    pathtype_fmt = path_fmt + u"  [{ds.type}]"
    full_fmt = pathtype_fmt + u"  {ds.branch!N}  {ds.describe!N} {ds.date!D}"
    if (not fast) or long_:
        full_fmt += u"  {ds.clean!X}"

    fmts = {
        AbsentRepoModel: pathtype_fmt,
        GitModel: full_fmt,
        AnnexModel: full_fmt
    }
    if long_:
        fmts[AnnexModel] += u"  {ds.annex_local_size!S}/{ds.annex_worktree_size!S}"

    formatter = LsFormatter()
    # weird problems happen in the parallel run -- TODO - figure it out
    # for out in Parallel(n_jobs=1)(
    #         delayed(format_ds_model)(formatter, dsm, full_fmt, format_exc=path_fmt + "  {msg!R}")
    #         for dsm in dss):
    #     print(out)
    for dsm in dsms:
        fmt = fmts[dsm.__class__]
        ds_str = format_ds_model(formatter, dsm, fmt, format_exc=path_fmt + u"  {msg!R}")
        safe_print(ds_str)
        # workaround for explosion of git cat-file --batch processes
        # https://github.com/datalad/datalad/issues/1888
        if dsm.repo is not None:
            dsm.repo.repo.close()
            del dsm.repo
            dsm.repo = None


def machinesize(humansize):
    """convert human-size string to machine-size"""
    try:
        size_str, size_unit = humansize.split(" ")
    except AttributeError:
        return float(humansize)
    unit_converter = {'Byte': 0, 'Bytes': 0, 'kB': 1, 'MB': 2, 'GB': 3, 'TB': 4, 'PB': 5}
    machinesize = float(size_str) * (1000 ** unit_converter[size_unit])
    return machinesize


def leaf_name(path):
    """takes a relative or absolute path and returns name of node at that location"""
    head, tail = split(abspath(path))
    return tail or basename(head)


def ignored(path, only_hidden=False):
    """if path is in the ignorelist return True

    ignore list includes hidden files and git or annex maintained folders
    when only_hidden set, only ignores hidden files and folders not git or annex maintained folders
    """
    if isdir(opj(path, ".git")) and not only_hidden:
        return True
    return '.' == leaf_name(path)[0] or leaf_name(path) == 'index.html'


def metadata_locator(fs_metadata=None, path=None, ds_path=None, metadata_path=None):
    """path to metadata file of node associated with the fs_metadata dictionary

    Parameters
    ----------
    fs_metadata: dict
      Metadata json of a node
    path: str
      Path to directory of metadata to be rendered
    ds_path: str
      Path to dataset root
    metadata_path: str
      Path to metadata root. Calculated relative to ds_path

    Returns
    -------
    str
      path to metadata of current node
    """

    # use implicit paths unless paths explicitly specified
    # Note: usage of ds_path as if it was the Repo's path. Therefore use
    # realpath, since we switched to have symlinks resolved in repos but not in
    # datasets
    ds_path = realpath(ds_path) if ds_path else fs_metadata['repo']
    path = path or fs_metadata['path']
    metadata_path = metadata_path or '.git/datalad/metadata'
    # directory metadata directory tree location
    metadata_dir = opj(ds_path, metadata_path)
    # relative path of current directory wrt dataset root
    dir_path = relpath(path, ds_path) if isabs(path) else path
    # normalize to / -- TODO, switch to '.' which is now actually the name since path is relative in web meta?
    if dir_path in ('.', None, ''):
        dir_path = '/'
    # create md5 hash of current directory's relative path
    metadata_hash = hashlib.md5(dir_path.encode('utf-8')).hexdigest()
    # construct final path to metadata file
    metadata_file = opj(metadata_dir, metadata_hash)

    return metadata_file


def fs_extract(nodepath, repo, basepath='/'):
    """extract required info of nodepath with its associated parent repository and returns it as a dictionary

    Parameters
    ----------
    nodepath : str
        Full path to the location we are exploring (must be a directory within
        `repo`
    repo : GitRepo
        Is the repository nodepath belongs to
    """
    # Create FsModel from filesystem nodepath and its associated parent repository
    node = FsModel(nodepath, repo)
    pretty_size = {stype: humanize.naturalsize(svalue) for stype, svalue in node.size.items()}
    pretty_date = time.strftime(u"%Y-%m-%d %H:%M:%S", time.localtime(node.date))
    name = leaf_name(node._path) if leaf_name(node._path) != "" else leaf_name(node.repo.path)
    rec = {
        "name": name, "path": relpath(node._path, basepath),
        "type": node.type_, "size": pretty_size, "date": pretty_date,
    }
    # if there is meta-data for the dataset (done by aggregate-metadata)
    # we include it
    metadata_path = opj(nodepath, METADATA_DIR, METADATA_FILENAME)
    if exists(metadata_path):
        # might need flattening!  TODO: flatten when aggregating?  why wasn't done?
        metadata = js.load(open(metadata_path))
        # might be too heavy to carry around, so will do basic flattening manually
        # and in a basic fashion
        # import jsonld
        metadata_reduced = metadata[0]
        for m in metadata[1:]:
            metadata_reduced.update(m)
        # but verify that they all had the same id
        if metadata:
            metaid = metadata[0]['@id']
            assert all(m['@id'] == metaid for m in metadata)
        rec["metadata"] = metadata_reduced
    return rec


def fs_render(fs_metadata, json=None, **kwargs):
    """takes node to render and based on json option passed renders to file, stdout or deletes json at root

    Parameters
    ----------
    fs_metadata: dict
      Metadata json to be rendered
    json: str ('file', 'display', 'delete')
      Render to file, stdout or delete json
    """

    metadata_file = metadata_locator(fs_metadata, **kwargs)

    if json == 'file':
        # create metadata_root directory if it doesn't exist
        metadata_dir = dirname(metadata_file)
        if not exists(metadata_dir):
            makedirs(metadata_dir)
        # write directory metadata to json
        with open(metadata_file, 'w') as f:
            js.dump(fs_metadata, f)

    # else if json flag set to delete, remove .dir.json of current directory
    elif json == 'delete' and exists(metadata_file):
        remove(metadata_file)

    # else dump json to stdout
    elif json == 'display':
        safe_print(js.dumps(fs_metadata) + '\n')


def fs_traverse(path, repo, parent=None, render=True, recursive=False, json=None, basepath=None):
    """Traverse path through its nodes and returns a dictionary of relevant attributes attached to each node

    Parameters
    ----------
    path: str
      Path to the directory to be traversed
    repo: AnnexRepo or GitRepo
      Repo object the directory belongs too
    parent: dict
      Extracted info about parent directory
    recursive: bool
      Recurse into subdirectories (note that subdatasets are not traversed)
    render: bool
       To render from within function or not. Set to false if results to be manipulated before final render

    Returns
    -------
    list of dict
      extracts and returns a (recursive) list of directory info at path
      does not traverse into annex, git or hidden directories
    """
    fs = fs_extract(path, repo, basepath=basepath or path)
    if isdir(path):                     # if node is a directory
        children = [fs.copy()]          # store its info in its children dict too  (Yarik is not sure why, but I guess for .?)
        # ATM seems some pieces still rely on having this duplication, so left as is
        # TODO: strip away
        for node in listdir(path):
            nodepath = opj(path, node)

            # TODO:  it might be a subdir which is non-initialized submodule!
            # if not ignored, append child node info to current nodes dictionary
            if not ignored(nodepath):
                # if recursive, create info dictionary (within) each child node too
                if recursive:
                    subdir = fs_traverse(nodepath,
                                         repo,
                                         parent=None,  # children[0],
                                         recursive=recursive,
                                         json=json,
                                         basepath=basepath or path)
                    subdir.pop('nodes', None)
                else:
                    # read child metadata from its metadata file if it exists
                    subdir_json = metadata_locator(path=node, ds_path=basepath or path)
                    if exists(subdir_json):
                        with open(subdir_json) as data_file:
                            subdir = js.load(data_file)
                            subdir.pop('nodes', None)
                    # else extract whatever information you can about the child
                    else:
                        # Yarik: this one is way too lean...
                        subdir = fs_extract(nodepath,
                                            repo,
                                            basepath=basepath or path)
                # append child metadata to list
                children.extend([subdir])

        # sum sizes of all 1st level children
        children_size = {}
        for node in children[1:]:
            for size_type, child_size in node['size'].items():
                children_size[size_type] = children_size.get(size_type, 0) + machinesize(child_size)

        # update current node sizes to the humanized aggregate children size
        fs['size'] = children[0]['size'] = \
            {size_type: humanize.naturalsize(child_size)
             for size_type, child_size in children_size.items()}

        children[0]['name'] = '.'       # replace current node name with '.' to emulate unix syntax
        if parent:
            parent['name'] = '..'       # replace parent node name with '..' to emulate unix syntax
            children.insert(1, parent)  # insert parent info after current node info in children dict

        fs['nodes'] = children          # add children info to main fs dictionary
        if render:                      # render directory node at location(path)
            fs_render(fs, json=json, ds_path=basepath or path)
            lgr.info('Directory: %s' % path)

    return fs


def ds_traverse(rootds, parent=None, json=None, recursive=False, all_=False,
                long_=False):
    """Hierarchical dataset traverser

    Parameters
    ----------
    rootds: Dataset
      Root dataset to be traversed
    parent: Dataset
      Parent dataset of the current rootds
    recursive: bool
       Recurse into subdirectories of the current dataset
    all_: bool
       Recurse into subdatasets of the root dataset

    Returns
    -------
    list of dict
      extracts and returns a (recursive) list of dataset(s) info at path
    """
    # extract parent info to pass to traverser
    fsparent = fs_extract(parent.path, parent.repo, basepath=rootds.path) if parent else None

    # (recursively) traverse file tree of current dataset
    fs = fs_traverse(rootds.path, rootds.repo,
                     render=False, parent=fsparent, recursive=all_,
                     json=json)
    size_list = [fs['size']]

    # (recursively) traverse each subdataset
    children = []
    # yoh: was in return results branch returning full datasets:
    # for subds in rootds.subdatasets(result_xfm='datasets'):
    # but since rpath is needed/used, decided to return relpaths
    for subds_rpath in rootds.subdatasets(result_xfm='relpaths'):

        subds_path = opj(rootds.path, subds_rpath)
        subds = Dataset(subds_path)
        subds_json = metadata_locator(path='.', ds_path=subds_path)

        def handle_not_installed():
            # for now just traverse as fs
            lgr.warning("%s is either not installed or lacks meta-data", subds)
            subfs = fs_extract(subds_path, rootds, basepath=rootds.path)
            # but add a custom type that it is a not installed subds
            subfs['type'] = 'uninitialized'
            # we need to kick it out from 'children'
            # TODO:  this is inefficient and cruel -- "ignored" should be made
            # smarted to ignore submodules for the repo
            if fs['nodes']:
                fs['nodes'] = [c for c in fs['nodes'] if c['path'] != subds_rpath]
            return subfs

        if not subds.is_installed():
            subfs = handle_not_installed()
        elif recursive:
            subfs = ds_traverse(subds,
                                json=json,
                                recursive=recursive,
                                all_=all_,
                                parent=rootds)
            subfs.pop('nodes', None)
            size_list.append(subfs['size'])
        # else just pick the data from metadata_file of each subdataset
        else:
            lgr.info(subds.path)
            if exists(subds_json):
                with open(subds_json) as data_file:
                    subfs = js.load(data_file)
                    subfs.pop('nodes', None)    # remove children
                    subfs['path'] = subds_rpath # reassign the path
                    size_list.append(subfs['size'])
            else:
                # the same drill as if not installed
                lgr.warning("%s is installed but no meta-data yet", subds)
                subfs = handle_not_installed()

        children.extend([subfs])

    # sum sizes of all 1st level children dataset
    children_size = {}
    for subdataset_size in size_list:
        for size_type, subds_size in subdataset_size.items():
            children_size[size_type] = children_size.get(size_type, 0) + machinesize(subds_size)

    # update current dataset sizes to the humanized aggregate subdataset sizes
    fs['size'] = {size_type: humanize.naturalsize(size)
                  for size_type, size in children_size.items()}
    fs['nodes'][0]['size'] = fs['size']  # update self's updated size in nodes sublist too!

    # add dataset specific entries to its dict
    rootds_model = GitModel(rootds.repo)
    fs['tags'] = rootds_model.describe
    fs['branch'] = rootds_model.branch
    index_file = opj(rootds.path, '.git', 'index')
    fs['index-mtime'] = time.strftime(
        u"%Y-%m-%d %H:%M:%S",
        time.localtime(getmtime(index_file))) if exists(index_file) else ''

    # append children datasets info to current dataset
    fs['nodes'].extend(children)

    # render current dataset
    lgr.info('Dataset: %s' % rootds.path)
    fs_render(fs, json=json, ds_path=rootds.path)
    return fs


def _ls_json(loc, fast=False, **kwargs):
    # hierarchically traverse file tree of (sub-)dataset(s) under path passed(loc)
    return ds_traverse(Dataset(loc), parent=None, **kwargs)


#
# S3 listing
#
def _ls_s3(loc, fast=False, recursive=False, all_=False, long_=False,
           config_file=None, list_content=False):
    """List S3 bucket content"""
    if loc.startswith('s3://'):
        bucket_prefix = loc[5:]
    else:
        raise ValueError("passed location should be an s3:// url")

    import boto
    from hashlib import md5
    from boto.s3.key import Key
    from boto.s3.prefix import Prefix
    from boto.exception import S3ResponseError
    from ..support.configparserinc import SafeConfigParser  # provides PY2,3 imports

    if '/' in bucket_prefix:
        bucket_name, prefix = bucket_prefix.split('/', 1)
    else:
        bucket_name, prefix = bucket_prefix, None

    if prefix and '?' in prefix:
        ui.message("We do not care about URL options ATM, they get stripped")
        prefix = prefix[:prefix.index('?')]

    ui.message("Connecting to bucket: %s" % bucket_name)
    if config_file:
        config = SafeConfigParser()
        config.read(config_file)
        access_key = config.get('default', 'access_key')
        secret_key = config.get('default', 'secret_key')

        # TODO: remove duplication -- reuse logic within downloaders/s3.py to get connected
        conn = boto.connect_s3(access_key, secret_key)
        try:
            bucket = conn.get_bucket(bucket_name)
        except S3ResponseError as e:
            ui.message("E: Cannot access bucket %s by name" % bucket_name)
            all_buckets = conn.get_all_buckets()
            all_bucket_names = [b.name for b in all_buckets]
            ui.message("I: Found following buckets %s" % ', '.join(all_bucket_names))
            if bucket_name in all_bucket_names:
                bucket = all_buckets[all_bucket_names.index(bucket_name)]
            else:
                raise RuntimeError("E: no bucket named %s thus exiting" % bucket_name)
    else:
        # TODO: expose credentials
        # We don't need any provider here really but only credentials
        from datalad.downloaders.providers import Providers
        providers = Providers.from_config_files()
        provider = providers.get_provider(loc)

        if not provider:
            raise ValueError(
                "Don't know how to deal with this url %s -- no provider defined for %s. "
                "Define a new provider (DOCS: TODO) or specify just s3cmd config file instead for now."
                % loc
            )
        downloader = provider.get_downloader(loc)

        # should authenticate etc, and when ready we will ask for a bucket ;)
        bucket = downloader.access(lambda url: downloader.bucket, loc)

    info = []
    for iname, imeth in [
        ("Versioning", bucket.get_versioning_status),
        ("   Website", bucket.get_website_endpoint),
        ("       ACL", bucket.get_acl),
    ]:
        try:
            ival = imeth()
        except Exception as e:
            ival = str(e).split('\n')[0]
        info.append(" {iname}: {ival}".format(**locals()))
    ui.message("Bucket info:\n %s" % '\n '.join(info))

    kwargs = {} if recursive else {'delimiter': '/'}

    ACCESS_METHODS = [
        bucket.list_versions,
        bucket.list
    ]

    prefix_all_versions = None
    for acc in ACCESS_METHODS:
        try:
            prefix_all_versions = list(acc(prefix, **kwargs))
            break
        except Exception as exc:
            lgr.debug("Failed to access via %s: %s", acc, exc_str(exc))

    if not prefix_all_versions:
        ui.error("No output was provided for prefix %r" % prefix)
    else:
        max_length = max((len(e.name) for e in prefix_all_versions))
        max_size_length = max((len(str(getattr(e, 'size', 0))) for e in prefix_all_versions))

    results = []
    for e in prefix_all_versions:
        results.append(e)
        if isinstance(e, Prefix):
            ui.message("%s" % (e.name, ),)
            continue

        base_msg = ("%%-%ds %%s" % max_length) % (e.name, e.last_modified)
        if isinstance(e, Key):
            if not (e.is_latest or all_):
                # Skip this one
                continue
            ui.message(base_msg + " %%%dd" % max_size_length % e.size, cr=' ')
            # OPT: delayed import
            from ..support.s3 import get_key_url
            url = get_key_url(e, schema='http')
            try:
                _ = urlopen(Request(url))
                urlok = "OK"
            except HTTPError as err:
                urlok = "E: %s" % err.code

            try:
                acl = e.get_acl()
            except S3ResponseError as exc:
                acl = exc.code if exc.code in ('AccessDenied',) else str(exc)

            content = ""
            if list_content:
                # IO intensive, make an option finally!
                try:
                    # _ = e.next()[:5]  if we are able to fetch the content
                    kwargs = dict(version_id=e.version_id)
                    if list_content in {'full', 'first10'}:
                        if list_content in 'first10':
                            kwargs['headers'] = {'Range': 'bytes=0-9'}
                        content = repr(e.get_contents_as_string(**kwargs))
                    elif list_content == 'md5':
                        digest = md5()
                        digest.update(e.get_contents_as_string(**kwargs))
                        content = digest.hexdigest()
                    else:
                        raise ValueError(list_content)
                    # content = "[S3: OK]"
                except S3ResponseError as err:
                    content = str(err)
                finally:
                    content = " " + content
            ui.message(
                "ver:%-32s  acl:%s  %s [%s]%s"
                % (getattr(e, 'version_id', None),
                   acl, url, urlok, content)
                if long_ else ''
            )
        else:
            ui.message(base_msg + " " + str(type(e)).split('.')[-1].rstrip("\"'>"))
    return results

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user