We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
securedrop-admin (setup) fails in Tails 4.0-rc1
## Description

Running `./securedrop-admin setup` in tails 4.0-rc1 (upgraded from 3.16) returns the following error:

```
INFO: Virtualenv already exists, not creating
INFO: Checking Python dependencies for securedrop-admin
ERROR: Failed to install pip dependencies. Check network connection and try again.
```
This was done in VMs, will try to reproduce on hardware.

## Steps to Reproduce

1. Upgrade Tails device from 3.16 to 4.0 (Boot into 4.0-rc1 drive and clone to upgrade 3.16 drive)
2. Boot into newly-upgraded tails drive
3. Verify that the internet is working (tor is bootstrapped, you can reach an external website over tor)
4. check out `1.1.0~rc2` tag
5. Run `./securedrop-admin setup` in ~/Persistent/securedrop
6. Observe error

## Expected Behavior

Securedrop-admin should run and the dependencies should be installed.

## Actual Behavior

Securedrop-admin fails and returns an error, the dependencies are not installed

## Comments

Per https://github.com/freedomofpress/securedrop/pull/4852/files#diff-b5e536cc161fcc0d62e661b4d6eae381R70-R73

When running the commands locally, I get
* `lsb_release --id --short` returns `Debian`
* `uname -a` returns `Linux amnesia 5.3.0-trunk-amd64 #1 SMF Debian 5.3.2-1~exp1 (2019-10-02) x86_64 GNU/Linux`

When i run ./securedrop-admin with no parameter, I get:
```
amnesia@amnesia:~/Persistent/securedrop$ ./securedrop-admin help
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'

Current thread 0x00007cf687450740 (most recent call first):
Aborted
```
securedrop-admin (setup) fails in Tails 4.0-rc1
## Description

Running `./securedrop-admin setup` in tails 4.0-rc1 (upgraded from 3.16) returns the following error:

```
INFO: Virtualenv already exists, not creating
INFO: Checking Python dependencies for securedrop-admin
ERROR: Failed to install pip dependencies. Check network connection and try again.
```
This was done in VMs, will try to reproduce on hardware.

## Steps to Reproduce

1. Upgrade Tails device from 3.16 to 4.0 (Boot into 4.0-rc1 drive and clone to upgrade 3.16 drive)
2. Boot into newly-upgraded tails drive
3. Verify that the internet is working (tor is bootstrapped, you can reach an external website over tor)
4. check out `1.1.0~rc2` tag
5. Run `./securedrop-admin setup` in ~/Persistent/securedrop
6. Observe error

## Expected Behavior

Securedrop-admin should run and the dependencies should be installed.

## Actual Behavior

Securedrop-admin fails and returns an error, the dependencies are not installed

## Comments

Per https://github.com/freedomofpress/securedrop/pull/4852/files#diff-b5e536cc161fcc0d62e661b4d6eae381R70-R73

When running the commands locally, I get
* `lsb_release --id --short` returns `Debian`
* `uname -a` returns `Linux amnesia 5.3.0-trunk-amd64 #1 SMF Debian 5.3.2-1~exp1 (2019-10-02) x86_64 GNU/Linux`

When i run ./securedrop-admin with no parameter, I get:
```
amnesia@amnesia:~/Persistent/securedrop$ ./securedrop-admin help
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'

Current thread 0x00007cf687450740 (most recent call first):
Aborted
```

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `admin/bootstrap.py`
Content:
```
1 # -*- mode: python; coding: utf-8 -*-
2 #
3 # Copyright (C) 2013-2018 Freedom of the Press Foundation & al
4 # Copyright (C) 2018 Loic Dachary <loic@dachary.org>
5 #
6 #    This program is free software: you can redistribute it and/or modify
7 #    it under the terms of the GNU General Public License as published by
8 #    the Free Software Foundation, either version 3 of the License, or
9 #    (at your option) any later version.
10 #
11 #    This program is distributed in the hope that it will be useful,
12 #    but WITHOUT ANY WARRANTY; without even the implied warranty of
13 #    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
14 #    GNU General Public License for more details.
15 #
16 #    You should have received a copy of the GNU General Public License
17 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
18 #
19 
20 import argparse
21 import logging
22 import os
23 import subprocess
24 import sys
25 
26 sdlog = logging.getLogger(__name__)
27 
28 DIR = os.path.dirname(os.path.realpath(__file__))
29 VENV_DIR = os.path.join(DIR, ".venv3")
30 
31 
32 def setup_logger(verbose=False):
33     """ Configure logging handler """
34     # Set default level on parent
35     sdlog.setLevel(logging.DEBUG)
36     level = logging.DEBUG if verbose else logging.INFO
37 
38     stdout = logging.StreamHandler(sys.stdout)
39     stdout.setFormatter(logging.Formatter('%(levelname)s: %(message)s'))
40     stdout.setLevel(level)
41     sdlog.addHandler(stdout)
42 
43 
44 def run_command(command):
45     """
46     Wrapper function to display stdout for running command,
47     similar to how shelling out in a Bash script displays rolling output.
48 
49     Yields a list of the stdout from the `command`, and raises a
50     CalledProcessError if `command` returns non-zero.
51     """
52     popen = subprocess.Popen(command,
53                              stdout=subprocess.PIPE,
54                              stderr=subprocess.STDOUT)
55     for stdout_line in iter(popen.stdout.readline, b""):
56         yield stdout_line
57     popen.stdout.close()
58     return_code = popen.wait()
59     if return_code:
60         raise subprocess.CalledProcessError(return_code, command)
61 
62 
63 def is_tails():
64     try:
65         id = subprocess.check_output('lsb_release --id --short',
66                                      shell=True).strip()
67     except subprocess.CalledProcessError:
68         id = None
69 
70     # dirty hack to unreliably detect Tails 4.0~beta2
71     if id == 'Debian':
72         if os.uname()[1] == 'amnesia':
73             id = 'Tails'
74 
75     return id == 'Tails'
76 
77 
78 def maybe_torify():
79     if is_tails():
80         return ['torify']
81     else:
82         return []
83 
84 
85 def install_apt_dependencies(args):
86     """
87     Install apt dependencies in Tails. In order to install Ansible in
88     a virtualenv, first there are a number of Python prerequisites.
89     """
90     sdlog.info("Installing SecureDrop Admin dependencies")
91     sdlog.info(("You'll be prompted for the temporary Tails admin password,"
92                 " which was set on Tails login screen"))
93 
94     apt_command = ['sudo', 'su', '-c',
95                    "apt-get update && \
96                    apt-get -q -o=Dpkg::Use-Pty=0 install -y \
97                    python3-virtualenv \
98                    python3-yaml \
99                    python3-pip \
100                    ccontrol \
101                    virtualenv \
102                    libffi-dev \
103                    libssl-dev \
104                    libpython3-dev",
105                    ]
106 
107     try:
108         # Print command results in real-time, to keep Admin apprised
109         # of progress during long-running command.
110         for output_line in run_command(apt_command):
111             print(output_line.decode('utf-8').rstrip())
112     except subprocess.CalledProcessError:
113         # Tails supports apt persistence, which was used by SecureDrop
114         # under Tails 2.x. If updates are being applied, don't try to pile
115         # on with more apt requests.
116         sdlog.error(("Failed to install apt dependencies. Check network"
117                      " connection and try again."))
118         raise
119 
120 
121 def envsetup(args):
122     """Installs Admin tooling required for managing SecureDrop. Specifically:
123 
124         * updates apt-cache
125         * installs apt packages for Python virtualenv
126         * creates virtualenv
127         * installs pip packages inside virtualenv
128 
129     The virtualenv is created within the Persistence volume in Tails, so that
130     Ansible is available to the Admin on subsequent boots without requiring
131     installation of packages again.
132     """
133     # virtualenv doesnt exist? Install dependencies and create
134     if not os.path.exists(VENV_DIR):
135 
136         install_apt_dependencies(args)
137 
138         # Technically you can create a virtualenv from within python
139         # but pip can only be run over tor on tails, and debugging that
140         # along with instaling a third-party dependency is not worth
141         # the effort here.
142         sdlog.info("Setting up virtualenv")
143         try:
144             sdlog.debug(subprocess.check_output(
145                 maybe_torify() + ['virtualenv', '--python=python3', VENV_DIR],
146                 stderr=subprocess.STDOUT))
147         except subprocess.CalledProcessError as e:
148             sdlog.debug(e.output)
149             sdlog.error(("Unable to create virtualenv. Check network settings"
150                          " and try again."))
151             raise
152     else:
153         sdlog.info("Virtualenv already exists, not creating")
154 
155     install_pip_dependencies(args)
156     if os.path.exists(os.path.join(DIR, 'setup.py')):
157         install_pip_self(args)
158 
159     sdlog.info("Finished installing SecureDrop dependencies")
160 
161 
162 def install_pip_self(args):
163     pip_install_cmd = [
164         os.path.join(VENV_DIR, 'bin', 'pip3'),
165         'install', '-e', DIR
166     ]
167     try:
168         subprocess.check_output(maybe_torify() + pip_install_cmd,
169                                 stderr=subprocess.STDOUT)
170     except subprocess.CalledProcessError as e:
171         sdlog.debug(e.output)
172         sdlog.error("Unable to install self, run with -v for more information")
173         raise
174 
175 
176 def install_pip_dependencies(args, pip_install_cmd=[
177         os.path.join(VENV_DIR, 'bin', 'pip3'),
178         'install',
179         # Specify requirements file.
180         '-r', os.path.join(DIR, 'requirements.txt'),
181         '--require-hashes',
182         # Make sure to upgrade packages only if necessary.
183         '-U', '--upgrade-strategy', 'only-if-needed',
184 ]):
185     """
186     Install Python dependencies via pip into virtualenv.
187     """
188 
189     sdlog.info("Checking Python dependencies for securedrop-admin")
190     try:
191         pip_output = subprocess.check_output(maybe_torify() + pip_install_cmd,
192                                              stderr=subprocess.STDOUT)
193     except subprocess.CalledProcessError as e:
194         sdlog.debug(e.output)
195         sdlog.error(("Failed to install pip dependencies. Check network"
196                      " connection and try again."))
197         raise
198 
199     sdlog.debug(pip_output)
200     if "Successfully installed" in str(pip_output):
201         sdlog.info("Python dependencies for securedrop-admin upgraded")
202     else:
203         sdlog.info("Python dependencies for securedrop-admin are up-to-date")
204 
205 
206 def parse_argv(argv):
207     parser = argparse.ArgumentParser()
208     parser.add_argument('-v', action='store_true', default=False,
209                         help="Increase verbosity on output")
210     parser.set_defaults(func=envsetup)
211 
212     return parser.parse_args(argv)
213 
214 
215 if __name__ == "__main__":
216     args = parse_argv(sys.argv[1:])
217     setup_logger(args.v)
218     if args.v:
219         args.func(args)
220     else:
221         try:
222             args.func(args)
223         except Exception:
224             sys.exit(1)
225         else:
226             sys.exit(0)
227
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/admin/bootstrap.py b/admin/bootstrap.py
--- a/admin/bootstrap.py
+++ b/admin/bootstrap.py
@@ -20,6 +20,7 @@
 import argparse
 import logging
 import os
+import shutil
 import subprocess
 import sys
 
@@ -68,13 +69,48 @@
         id = None
 
     # dirty hack to unreliably detect Tails 4.0~beta2
-    if id == 'Debian':
+    if id == b'Debian':
         if os.uname()[1] == 'amnesia':
             id = 'Tails'
 
     return id == 'Tails'
 
 
+def clean_up_tails3_venv(virtualenv_dir=VENV_DIR):
+    """
+    Tails 3.x, based on debian stretch uses libpython3.5, whereas Tails 4.x is
+    based on Debian Buster and uses libpython3.7. This means that the Tails 3.x
+    virtualenv will not work under Tails 4.x, and will need to be destroyed and
+    rebuilt. We can detect if the version of libpython is 3.5 in the
+    admin/.venv3/ folder, and delete it if that's the case. This will ensure a
+    smooth upgrade from Tails 3.x to Tails 4.x.
+    """
+    if is_tails():
+        try:
+            dist = subprocess.check_output('lsb_release --codename --short',
+                                           shell=True).strip()
+        except subprocess.CalledProcessError:
+            dist = None
+
+        # tails4 is based on buster
+        if dist == b'buster':
+            python_lib_path = os.path.join(virtualenv_dir, "lib/python3.5")
+            if os.path.exists(os.path.join(python_lib_path)):
+                sdlog.info(
+                    "Tails 3 Python 3 virtualenv detected. "
+                    "Removing it."
+                )
+                shutil.rmtree(virtualenv_dir)
+                sdlog.info("Tails 3 Python 3 virtualenv deleted.")
+
+
+def checkenv(args):
+    clean_up_tails3_venv(VENV_DIR)
+    if not os.path.exists(os.path.join(VENV_DIR, "bin/activate")):
+        sdlog.error('Please run "securedrop-admin setup".')
+        sys.exit(1)
+
+
 def maybe_torify():
     if is_tails():
         return ['torify']
@@ -130,6 +166,9 @@
     Ansible is available to the Admin on subsequent boots without requiring
     installation of packages again.
     """
+    # clean up tails 3.x venv when migrating to tails 4.x
+    clean_up_tails3_venv(VENV_DIR)
+
     # virtualenv doesnt exist? Install dependencies and create
     if not os.path.exists(VENV_DIR):
 
@@ -209,18 +248,30 @@
                         help="Increase verbosity on output")
     parser.set_defaults(func=envsetup)
 
+    subparsers = parser.add_subparsers()
+
+    envsetup_parser = subparsers.add_parser(
+        'envsetup',
+        help='Set up the admin virtualenv.'
+    )
+    envsetup_parser.set_defaults(func=envsetup)
+
+    checkenv_parser = subparsers.add_parser(
+        'checkenv',
+        help='Check that the admin virtualenv is properly set up.'
+    )
+    checkenv_parser.set_defaults(func=checkenv)
+
     return parser.parse_args(argv)
 
 
 if __name__ == "__main__":
     args = parse_argv(sys.argv[1:])
     setup_logger(args.v)
-    if args.v:
+
+    try:
         args.func(args)
+    except Exception:
+        sys.exit(1)
     else:
-        try:
-            args.func(args)
-        except Exception:
-            sys.exit(1)
-        else:
-            sys.exit(0)
+        sys.exit(0)

{"golden_diff": "diff --git a/admin/bootstrap.py b/admin/bootstrap.py\n--- a/admin/bootstrap.py\n+++ b/admin/bootstrap.py\n@@ -20,6 +20,7 @@\n import argparse\n import logging\n import os\n+import shutil\n import subprocess\n import sys\n \n@@ -68,13 +69,48 @@\n         id = None\n \n     # dirty hack to unreliably detect Tails 4.0~beta2\n-    if id == 'Debian':\n+    if id == b'Debian':\n         if os.uname()[1] == 'amnesia':\n             id = 'Tails'\n \n     return id == 'Tails'\n \n \n+def clean_up_tails3_venv(virtualenv_dir=VENV_DIR):\n+    \"\"\"\n+    Tails 3.x, based on debian stretch uses libpython3.5, whereas Tails 4.x is\n+    based on Debian Buster and uses libpython3.7. This means that the Tails 3.x\n+    virtualenv will not work under Tails 4.x, and will need to be destroyed and\n+    rebuilt. We can detect if the version of libpython is 3.5 in the\n+    admin/.venv3/ folder, and delete it if that's the case. This will ensure a\n+    smooth upgrade from Tails 3.x to Tails 4.x.\n+    \"\"\"\n+    if is_tails():\n+        try:\n+            dist = subprocess.check_output('lsb_release --codename --short',\n+                                           shell=True).strip()\n+        except subprocess.CalledProcessError:\n+            dist = None\n+\n+        # tails4 is based on buster\n+        if dist == b'buster':\n+            python_lib_path = os.path.join(virtualenv_dir, \"lib/python3.5\")\n+            if os.path.exists(os.path.join(python_lib_path)):\n+                sdlog.info(\n+                    \"Tails 3 Python 3 virtualenv detected. \"\n+                    \"Removing it.\"\n+                )\n+                shutil.rmtree(virtualenv_dir)\n+                sdlog.info(\"Tails 3 Python 3 virtualenv deleted.\")\n+\n+\n+def checkenv(args):\n+    clean_up_tails3_venv(VENV_DIR)\n+    if not os.path.exists(os.path.join(VENV_DIR, \"bin/activate\")):\n+        sdlog.error('Please run \"securedrop-admin setup\".')\n+        sys.exit(1)\n+\n+\n def maybe_torify():\n     if is_tails():\n         return ['torify']\n@@ -130,6 +166,9 @@\n     Ansible is available to the Admin on subsequent boots without requiring\n     installation of packages again.\n     \"\"\"\n+    # clean up tails 3.x venv when migrating to tails 4.x\n+    clean_up_tails3_venv(VENV_DIR)\n+\n     # virtualenv doesnt exist? Install dependencies and create\n     if not os.path.exists(VENV_DIR):\n \n@@ -209,18 +248,30 @@\n                         help=\"Increase verbosity on output\")\n     parser.set_defaults(func=envsetup)\n \n+    subparsers = parser.add_subparsers()\n+\n+    envsetup_parser = subparsers.add_parser(\n+        'envsetup',\n+        help='Set up the admin virtualenv.'\n+    )\n+    envsetup_parser.set_defaults(func=envsetup)\n+\n+    checkenv_parser = subparsers.add_parser(\n+        'checkenv',\n+        help='Check that the admin virtualenv is properly set up.'\n+    )\n+    checkenv_parser.set_defaults(func=checkenv)\n+\n     return parser.parse_args(argv)\n \n \n if __name__ == \"__main__\":\n     args = parse_argv(sys.argv[1:])\n     setup_logger(args.v)\n-    if args.v:\n+\n+    try:\n         args.func(args)\n+    except Exception:\n+        sys.exit(1)\n     else:\n-        try:\n-            args.func(args)\n-        except Exception:\n-            sys.exit(1)\n-        else:\n-            sys.exit(0)\n+        sys.exit(0)\n", "issue": "securedrop-admin (setup) fails in Tails 4.0-rc1\n## Description\r\n\r\nRunning `./securedrop-admin setup` in tails 4.0-rc1 (upgraded from 3.16) returns the following error:\r\n\r\n```\r\nINFO: Virtualenv already exists, not creating\r\nINFO: Checking Python dependencies for securedrop-admin\r\nERROR: Failed to install pip dependencies. Check network connection and try again.\r\n```\r\nThis was done in VMs, will try to reproduce on hardware.\r\n\r\n## Steps to Reproduce\r\n\r\n1. Upgrade Tails device from 3.16 to 4.0 (Boot into 4.0-rc1 drive and clone to upgrade 3.16 drive)\r\n2. Boot into newly-upgraded tails drive\r\n3. Verify that the internet is working (tor is bootstrapped, you can reach an external website over tor)\r\n4. check out `1.1.0~rc2` tag\r\n5. Run `./securedrop-admin setup` in ~/Persistent/securedrop\r\n6. Observe error\r\n\r\n## Expected Behavior\r\n\r\nSecuredrop-admin should run and the dependencies should be installed.\r\n\r\n## Actual Behavior\r\n\r\nSecuredrop-admin fails and returns an error, the dependencies are not installed\r\n\r\n## Comments\r\n\r\nPer https://github.com/freedomofpress/securedrop/pull/4852/files#diff-b5e536cc161fcc0d62e661b4d6eae381R70-R73\r\n\r\nWhen running the commands locally, I get\r\n* `lsb_release --id --short` returns `Debian`\r\n* `uname -a` returns `Linux amnesia 5.3.0-trunk-amd64 #1 SMF Debian 5.3.2-1~exp1 (2019-10-02) x86_64 GNU/Linux`\r\n\r\nWhen i run ./securedrop-admin with no parameter, I get:\r\n```\r\namnesia@amnesia:~/Persistent/securedrop$ ./securedrop-admin help\r\nCould not find platform independent libraries <prefix>\r\nCould not find platform dependent libraries <exec_prefix>\r\nConsider setting $PYTHONHOME to <prefix>[:<exec_prefix>]\r\nFatal Python error: Py_Initialize: Unable to get the locale encoding\r\nImportError: No module named 'encodings'\r\n\r\nCurrent thread 0x00007cf687450740 (most recent call first):\r\nAborted\r\n```\nsecuredrop-admin (setup) fails in Tails 4.0-rc1\n## Description\r\n\r\nRunning `./securedrop-admin setup` in tails 4.0-rc1 (upgraded from 3.16) returns the following error:\r\n\r\n```\r\nINFO: Virtualenv already exists, not creating\r\nINFO: Checking Python dependencies for securedrop-admin\r\nERROR: Failed to install pip dependencies. Check network connection and try again.\r\n```\r\nThis was done in VMs, will try to reproduce on hardware.\r\n\r\n## Steps to Reproduce\r\n\r\n1. Upgrade Tails device from 3.16 to 4.0 (Boot into 4.0-rc1 drive and clone to upgrade 3.16 drive)\r\n2. Boot into newly-upgraded tails drive\r\n3. Verify that the internet is working (tor is bootstrapped, you can reach an external website over tor)\r\n4. check out `1.1.0~rc2` tag\r\n5. Run `./securedrop-admin setup` in ~/Persistent/securedrop\r\n6. Observe error\r\n\r\n## Expected Behavior\r\n\r\nSecuredrop-admin should run and the dependencies should be installed.\r\n\r\n## Actual Behavior\r\n\r\nSecuredrop-admin fails and returns an error, the dependencies are not installed\r\n\r\n## Comments\r\n\r\nPer https://github.com/freedomofpress/securedrop/pull/4852/files#diff-b5e536cc161fcc0d62e661b4d6eae381R70-R73\r\n\r\nWhen running the commands locally, I get\r\n* `lsb_release --id --short` returns `Debian`\r\n* `uname -a` returns `Linux amnesia 5.3.0-trunk-amd64 #1 SMF Debian 5.3.2-1~exp1 (2019-10-02) x86_64 GNU/Linux`\r\n\r\nWhen i run ./securedrop-admin with no parameter, I get:\r\n```\r\namnesia@amnesia:~/Persistent/securedrop$ ./securedrop-admin help\r\nCould not find platform independent libraries <prefix>\r\nCould not find platform dependent libraries <exec_prefix>\r\nConsider setting $PYTHONHOME to <prefix>[:<exec_prefix>]\r\nFatal Python error: Py_Initialize: Unable to get the locale encoding\r\nImportError: No module named 'encodings'\r\n\r\nCurrent thread 0x00007cf687450740 (most recent call first):\r\nAborted\r\n```\n", "before_files": [{"content": "# -*- mode: python; coding: utf-8 -*-\n#\n# Copyright (C) 2013-2018 Freedom of the Press Foundation & al\n# Copyright (C) 2018 Loic Dachary <loic@dachary.org>\n#\n#    This program is free software: you can redistribute it and/or modify\n#    it under the terms of the GNU General Public License as published by\n#    the Free Software Foundation, either version 3 of the License, or\n#    (at your option) any later version.\n#\n#    This program is distributed in the hope that it will be useful,\n#    but WITHOUT ANY WARRANTY; without even the implied warranty of\n#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#    GNU General Public License for more details.\n#\n#    You should have received a copy of the GNU General Public License\n#    along with this program.  If not, see <http://www.gnu.org/licenses/>.\n#\n\nimport argparse\nimport logging\nimport os\nimport subprocess\nimport sys\n\nsdlog = logging.getLogger(__name__)\n\nDIR = os.path.dirname(os.path.realpath(__file__))\nVENV_DIR = os.path.join(DIR, \".venv3\")\n\n\ndef setup_logger(verbose=False):\n    \"\"\" Configure logging handler \"\"\"\n    # Set default level on parent\n    sdlog.setLevel(logging.DEBUG)\n    level = logging.DEBUG if verbose else logging.INFO\n\n    stdout = logging.StreamHandler(sys.stdout)\n    stdout.setFormatter(logging.Formatter('%(levelname)s: %(message)s'))\n    stdout.setLevel(level)\n    sdlog.addHandler(stdout)\n\n\ndef run_command(command):\n    \"\"\"\n    Wrapper function to display stdout for running command,\n    similar to how shelling out in a Bash script displays rolling output.\n\n    Yields a list of the stdout from the `command`, and raises a\n    CalledProcessError if `command` returns non-zero.\n    \"\"\"\n    popen = subprocess.Popen(command,\n                             stdout=subprocess.PIPE,\n                             stderr=subprocess.STDOUT)\n    for stdout_line in iter(popen.stdout.readline, b\"\"):\n        yield stdout_line\n    popen.stdout.close()\n    return_code = popen.wait()\n    if return_code:\n        raise subprocess.CalledProcessError(return_code, command)\n\n\ndef is_tails():\n    try:\n        id = subprocess.check_output('lsb_release --id --short',\n                                     shell=True).strip()\n    except subprocess.CalledProcessError:\n        id = None\n\n    # dirty hack to unreliably detect Tails 4.0~beta2\n    if id == 'Debian':\n        if os.uname()[1] == 'amnesia':\n            id = 'Tails'\n\n    return id == 'Tails'\n\n\ndef maybe_torify():\n    if is_tails():\n        return ['torify']\n    else:\n        return []\n\n\ndef install_apt_dependencies(args):\n    \"\"\"\n    Install apt dependencies in Tails. In order to install Ansible in\n    a virtualenv, first there are a number of Python prerequisites.\n    \"\"\"\n    sdlog.info(\"Installing SecureDrop Admin dependencies\")\n    sdlog.info((\"You'll be prompted for the temporary Tails admin password,\"\n                \" which was set on Tails login screen\"))\n\n    apt_command = ['sudo', 'su', '-c',\n                   \"apt-get update && \\\n                   apt-get -q -o=Dpkg::Use-Pty=0 install -y \\\n                   python3-virtualenv \\\n                   python3-yaml \\\n                   python3-pip \\\n                   ccontrol \\\n                   virtualenv \\\n                   libffi-dev \\\n                   libssl-dev \\\n                   libpython3-dev\",\n                   ]\n\n    try:\n        # Print command results in real-time, to keep Admin apprised\n        # of progress during long-running command.\n        for output_line in run_command(apt_command):\n            print(output_line.decode('utf-8').rstrip())\n    except subprocess.CalledProcessError:\n        # Tails supports apt persistence, which was used by SecureDrop\n        # under Tails 2.x. If updates are being applied, don't try to pile\n        # on with more apt requests.\n        sdlog.error((\"Failed to install apt dependencies. Check network\"\n                     \" connection and try again.\"))\n        raise\n\n\ndef envsetup(args):\n    \"\"\"Installs Admin tooling required for managing SecureDrop. Specifically:\n\n        * updates apt-cache\n        * installs apt packages for Python virtualenv\n        * creates virtualenv\n        * installs pip packages inside virtualenv\n\n    The virtualenv is created within the Persistence volume in Tails, so that\n    Ansible is available to the Admin on subsequent boots without requiring\n    installation of packages again.\n    \"\"\"\n    # virtualenv doesnt exist? Install dependencies and create\n    if not os.path.exists(VENV_DIR):\n\n        install_apt_dependencies(args)\n\n        # Technically you can create a virtualenv from within python\n        # but pip can only be run over tor on tails, and debugging that\n        # along with instaling a third-party dependency is not worth\n        # the effort here.\n        sdlog.info(\"Setting up virtualenv\")\n        try:\n            sdlog.debug(subprocess.check_output(\n                maybe_torify() + ['virtualenv', '--python=python3', VENV_DIR],\n                stderr=subprocess.STDOUT))\n        except subprocess.CalledProcessError as e:\n            sdlog.debug(e.output)\n            sdlog.error((\"Unable to create virtualenv. Check network settings\"\n                         \" and try again.\"))\n            raise\n    else:\n        sdlog.info(\"Virtualenv already exists, not creating\")\n\n    install_pip_dependencies(args)\n    if os.path.exists(os.path.join(DIR, 'setup.py')):\n        install_pip_self(args)\n\n    sdlog.info(\"Finished installing SecureDrop dependencies\")\n\n\ndef install_pip_self(args):\n    pip_install_cmd = [\n        os.path.join(VENV_DIR, 'bin', 'pip3'),\n        'install', '-e', DIR\n    ]\n    try:\n        subprocess.check_output(maybe_torify() + pip_install_cmd,\n                                stderr=subprocess.STDOUT)\n    except subprocess.CalledProcessError as e:\n        sdlog.debug(e.output)\n        sdlog.error(\"Unable to install self, run with -v for more information\")\n        raise\n\n\ndef install_pip_dependencies(args, pip_install_cmd=[\n        os.path.join(VENV_DIR, 'bin', 'pip3'),\n        'install',\n        # Specify requirements file.\n        '-r', os.path.join(DIR, 'requirements.txt'),\n        '--require-hashes',\n        # Make sure to upgrade packages only if necessary.\n        '-U', '--upgrade-strategy', 'only-if-needed',\n]):\n    \"\"\"\n    Install Python dependencies via pip into virtualenv.\n    \"\"\"\n\n    sdlog.info(\"Checking Python dependencies for securedrop-admin\")\n    try:\n        pip_output = subprocess.check_output(maybe_torify() + pip_install_cmd,\n                                             stderr=subprocess.STDOUT)\n    except subprocess.CalledProcessError as e:\n        sdlog.debug(e.output)\n        sdlog.error((\"Failed to install pip dependencies. Check network\"\n                     \" connection and try again.\"))\n        raise\n\n    sdlog.debug(pip_output)\n    if \"Successfully installed\" in str(pip_output):\n        sdlog.info(\"Python dependencies for securedrop-admin upgraded\")\n    else:\n        sdlog.info(\"Python dependencies for securedrop-admin are up-to-date\")\n\n\ndef parse_argv(argv):\n    parser = argparse.ArgumentParser()\n    parser.add_argument('-v', action='store_true', default=False,\n                        help=\"Increase verbosity on output\")\n    parser.set_defaults(func=envsetup)\n\n    return parser.parse_args(argv)\n\n\nif __name__ == \"__main__\":\n    args = parse_argv(sys.argv[1:])\n    setup_logger(args.v)\n    if args.v:\n        args.func(args)\n    else:\n        try:\n            args.func(args)\n        except Exception:\n            sys.exit(1)\n        else:\n            sys.exit(0)\n", "path": "admin/bootstrap.py"}], "after_files": [{"content": "# -*- mode: python; coding: utf-8 -*-\n#\n# Copyright (C) 2013-2018 Freedom of the Press Foundation & al\n# Copyright (C) 2018 Loic Dachary <loic@dachary.org>\n#\n#    This program is free software: you can redistribute it and/or modify\n#    it under the terms of the GNU General Public License as published by\n#    the Free Software Foundation, either version 3 of the License, or\n#    (at your option) any later version.\n#\n#    This program is distributed in the hope that it will be useful,\n#    but WITHOUT ANY WARRANTY; without even the implied warranty of\n#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#    GNU General Public License for more details.\n#\n#    You should have received a copy of the GNU General Public License\n#    along with this program.  If not, see <http://www.gnu.org/licenses/>.\n#\n\nimport argparse\nimport logging\nimport os\nimport shutil\nimport subprocess\nimport sys\n\nsdlog = logging.getLogger(__name__)\n\nDIR = os.path.dirname(os.path.realpath(__file__))\nVENV_DIR = os.path.join(DIR, \".venv3\")\n\n\ndef setup_logger(verbose=False):\n    \"\"\" Configure logging handler \"\"\"\n    # Set default level on parent\n    sdlog.setLevel(logging.DEBUG)\n    level = logging.DEBUG if verbose else logging.INFO\n\n    stdout = logging.StreamHandler(sys.stdout)\n    stdout.setFormatter(logging.Formatter('%(levelname)s: %(message)s'))\n    stdout.setLevel(level)\n    sdlog.addHandler(stdout)\n\n\ndef run_command(command):\n    \"\"\"\n    Wrapper function to display stdout for running command,\n    similar to how shelling out in a Bash script displays rolling output.\n\n    Yields a list of the stdout from the `command`, and raises a\n    CalledProcessError if `command` returns non-zero.\n    \"\"\"\n    popen = subprocess.Popen(command,\n                             stdout=subprocess.PIPE,\n                             stderr=subprocess.STDOUT)\n    for stdout_line in iter(popen.stdout.readline, b\"\"):\n        yield stdout_line\n    popen.stdout.close()\n    return_code = popen.wait()\n    if return_code:\n        raise subprocess.CalledProcessError(return_code, command)\n\n\ndef is_tails():\n    try:\n        id = subprocess.check_output('lsb_release --id --short',\n                                     shell=True).strip()\n    except subprocess.CalledProcessError:\n        id = None\n\n    # dirty hack to unreliably detect Tails 4.0~beta2\n    if id == b'Debian':\n        if os.uname()[1] == 'amnesia':\n            id = 'Tails'\n\n    return id == 'Tails'\n\n\ndef clean_up_tails3_venv(virtualenv_dir=VENV_DIR):\n    \"\"\"\n    Tails 3.x, based on debian stretch uses libpython3.5, whereas Tails 4.x is\n    based on Debian Buster and uses libpython3.7. This means that the Tails 3.x\n    virtualenv will not work under Tails 4.x, and will need to be destroyed and\n    rebuilt. We can detect if the version of libpython is 3.5 in the\n    admin/.venv3/ folder, and delete it if that's the case. This will ensure a\n    smooth upgrade from Tails 3.x to Tails 4.x.\n    \"\"\"\n    if is_tails():\n        try:\n            dist = subprocess.check_output('lsb_release --codename --short',\n                                           shell=True).strip()\n        except subprocess.CalledProcessError:\n            dist = None\n\n        # tails4 is based on buster\n        if dist == b'buster':\n            python_lib_path = os.path.join(virtualenv_dir, \"lib/python3.5\")\n            if os.path.exists(os.path.join(python_lib_path)):\n                sdlog.info(\n                    \"Tails 3 Python 3 virtualenv detected. \"\n                    \"Removing it.\"\n                )\n                shutil.rmtree(virtualenv_dir)\n                sdlog.info(\"Tails 3 Python 3 virtualenv deleted.\")\n\n\ndef checkenv(args):\n    clean_up_tails3_venv(VENV_DIR)\n    if not os.path.exists(os.path.join(VENV_DIR, \"bin/activate\")):\n        sdlog.error('Please run \"securedrop-admin setup\".')\n        sys.exit(1)\n\n\ndef maybe_torify():\n    if is_tails():\n        return ['torify']\n    else:\n        return []\n\n\ndef install_apt_dependencies(args):\n    \"\"\"\n    Install apt dependencies in Tails. In order to install Ansible in\n    a virtualenv, first there are a number of Python prerequisites.\n    \"\"\"\n    sdlog.info(\"Installing SecureDrop Admin dependencies\")\n    sdlog.info((\"You'll be prompted for the temporary Tails admin password,\"\n                \" which was set on Tails login screen\"))\n\n    apt_command = ['sudo', 'su', '-c',\n                   \"apt-get update && \\\n                   apt-get -q -o=Dpkg::Use-Pty=0 install -y \\\n                   python3-virtualenv \\\n                   python3-yaml \\\n                   python3-pip \\\n                   ccontrol \\\n                   virtualenv \\\n                   libffi-dev \\\n                   libssl-dev \\\n                   libpython3-dev\",\n                   ]\n\n    try:\n        # Print command results in real-time, to keep Admin apprised\n        # of progress during long-running command.\n        for output_line in run_command(apt_command):\n            print(output_line.decode('utf-8').rstrip())\n    except subprocess.CalledProcessError:\n        # Tails supports apt persistence, which was used by SecureDrop\n        # under Tails 2.x. If updates are being applied, don't try to pile\n        # on with more apt requests.\n        sdlog.error((\"Failed to install apt dependencies. Check network\"\n                     \" connection and try again.\"))\n        raise\n\n\ndef envsetup(args):\n    \"\"\"Installs Admin tooling required for managing SecureDrop. Specifically:\n\n        * updates apt-cache\n        * installs apt packages for Python virtualenv\n        * creates virtualenv\n        * installs pip packages inside virtualenv\n\n    The virtualenv is created within the Persistence volume in Tails, so that\n    Ansible is available to the Admin on subsequent boots without requiring\n    installation of packages again.\n    \"\"\"\n    # clean up tails 3.x venv when migrating to tails 4.x\n    clean_up_tails3_venv(VENV_DIR)\n\n    # virtualenv doesnt exist? Install dependencies and create\n    if not os.path.exists(VENV_DIR):\n\n        install_apt_dependencies(args)\n\n        # Technically you can create a virtualenv from within python\n        # but pip can only be run over tor on tails, and debugging that\n        # along with instaling a third-party dependency is not worth\n        # the effort here.\n        sdlog.info(\"Setting up virtualenv\")\n        try:\n            sdlog.debug(subprocess.check_output(\n                maybe_torify() + ['virtualenv', '--python=python3', VENV_DIR],\n                stderr=subprocess.STDOUT))\n        except subprocess.CalledProcessError as e:\n            sdlog.debug(e.output)\n            sdlog.error((\"Unable to create virtualenv. Check network settings\"\n                         \" and try again.\"))\n            raise\n    else:\n        sdlog.info(\"Virtualenv already exists, not creating\")\n\n    install_pip_dependencies(args)\n    if os.path.exists(os.path.join(DIR, 'setup.py')):\n        install_pip_self(args)\n\n    sdlog.info(\"Finished installing SecureDrop dependencies\")\n\n\ndef install_pip_self(args):\n    pip_install_cmd = [\n        os.path.join(VENV_DIR, 'bin', 'pip3'),\n        'install', '-e', DIR\n    ]\n    try:\n        subprocess.check_output(maybe_torify() + pip_install_cmd,\n                                stderr=subprocess.STDOUT)\n    except subprocess.CalledProcessError as e:\n        sdlog.debug(e.output)\n        sdlog.error(\"Unable to install self, run with -v for more information\")\n        raise\n\n\ndef install_pip_dependencies(args, pip_install_cmd=[\n        os.path.join(VENV_DIR, 'bin', 'pip3'),\n        'install',\n        # Specify requirements file.\n        '-r', os.path.join(DIR, 'requirements.txt'),\n        '--require-hashes',\n        # Make sure to upgrade packages only if necessary.\n        '-U', '--upgrade-strategy', 'only-if-needed',\n]):\n    \"\"\"\n    Install Python dependencies via pip into virtualenv.\n    \"\"\"\n\n    sdlog.info(\"Checking Python dependencies for securedrop-admin\")\n    try:\n        pip_output = subprocess.check_output(maybe_torify() + pip_install_cmd,\n                                             stderr=subprocess.STDOUT)\n    except subprocess.CalledProcessError as e:\n        sdlog.debug(e.output)\n        sdlog.error((\"Failed to install pip dependencies. Check network\"\n                     \" connection and try again.\"))\n        raise\n\n    sdlog.debug(pip_output)\n    if \"Successfully installed\" in str(pip_output):\n        sdlog.info(\"Python dependencies for securedrop-admin upgraded\")\n    else:\n        sdlog.info(\"Python dependencies for securedrop-admin are up-to-date\")\n\n\ndef parse_argv(argv):\n    parser = argparse.ArgumentParser()\n    parser.add_argument('-v', action='store_true', default=False,\n                        help=\"Increase verbosity on output\")\n    parser.set_defaults(func=envsetup)\n\n    subparsers = parser.add_subparsers()\n\n    envsetup_parser = subparsers.add_parser(\n        'envsetup',\n        help='Set up the admin virtualenv.'\n    )\n    envsetup_parser.set_defaults(func=envsetup)\n\n    checkenv_parser = subparsers.add_parser(\n        'checkenv',\n        help='Check that the admin virtualenv is properly set up.'\n    )\n    checkenv_parser.set_defaults(func=checkenv)\n\n    return parser.parse_args(argv)\n\n\nif __name__ == \"__main__\":\n    args = parse_argv(sys.argv[1:])\n    setup_logger(args.v)\n\n    try:\n        args.func(args)\n    except Exception:\n        sys.exit(1)\n    else:\n        sys.exit(0)\n", "path": "admin/bootstrap.py"}]}