Mac OS 11.4 Big Sur - fixing python cryptography package in /System/Library/Frameworks/Python.framework/Versions/2.7

Question

I bought a MacBook Pro 13 M1 (16 GB RAM, 1 TB SSD) and I'm trying to get it setup for work and part of that is using python 2 (yes I know it's end of life, I still need to use it till we finish porting to go). I have some python packages I'd like to remove from my system python2 (I'm sure I must've run sudo pip install on these at some point, which I know was stupid, but what is done is done) and now that I'm on Big Sur I can't remove these packages. I tried disabling SIP until I can get them removed (csrutil disable from recovery), but now I'm getting errors that I can't remove files from a read only filesystem

sudo pip uninstall cryptography                                                                                                                                                                                            2  14:34
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
WARNING: The directory '/Users/darren/Library/Caches/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Found existing installation: cryptography 3.3.2
Uninstalling cryptography-3.3.2:
  Would remove:
    /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography
    /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography-3.3.2-py2.7.egg-info
Proceed (y/n)? y
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/pip/_internal/commands/uninstall.py", line 90, in run
    auto_confirm=options.yes, verbose=self.verbosity > 0,
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/pip/_internal/req/req_install.py", line 686, in uninstall
    uninstalled_pathset.remove(auto_confirm, verbose)
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/pip/_internal/req/req_uninstall.py", line 403, in remove
    moved.stash(path)
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/pip/_internal/req/req_uninstall.py", line 292, in stash
    renames(path, new_path)
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/pip/_internal/utils/misc.py", line 355, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 323, in move
    rmtree(src)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 270, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 275, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 273, in rmtree
    os.remove(fullname)
OSError: [Errno 30] Read-only file system: '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/_der.py'

but I can't remount root as read/write and get this error when I try:

sudo mount -uw /
mount_apfs: volume could not be mounted: Permission denied
mount: / failed with 66

How do I remove these files from /System/Library if it's read only? Do I have to format my system and start from scratch now that mac os put these files in a read only filesystem or is there a way to remove them?

Edit:

So I have broken packages in /System/Library that are preventing fabric from running so I can't get our stuff up and running. The main one being the cryptography package. So I either need to fix these packages so fabric can run or remove them or something.

The other part that makes this such a pain is that I'm trying to run python 2 on an M1 Mac which means either using the built in python or using rosetta with an x86 python2 as I have found no method for installing arm python 2 on an M1 Mac (there isn't an installer from python.org or brew since python2 was end of life when the M1 Macs shipped).

Edit2:

Maybe these packages that I'm having issues with did ship with Mac OS as the package's folder's created timestamp is "Jan 1 01:00:00 2020" just like every other file & folder in that same folder. If so I guess I need to fix them somehow. When I try to run fab from the command line I get this error in the cryptography package:

fab
/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.hazmat.backends import default_backend
Traceback (most recent call last):
  File "/Users/darren/Library/Python/2.7/bin/fab", line 5, in <module>
    from fabric.main import program
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/fabric/__init__.py", line 3, in <module>
    from .connection import Config, Connection
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/fabric/connection.py", line 16, in <module>
    from paramiko.agent import AgentRequestHandler
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/__init__.py", line 22, in <module>
    from paramiko.transport import SecurityOptions, Transport
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/transport.py", line 129, in <module>
    class Transport(threading.Thread, ClosingContextManager):
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/transport.py", line 190, in Transport
    if KexCurve25519.is_available():
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/kex_curve25519.py", line 30, in is_available
    X25519PrivateKey.generate()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/primitives/asymmetric/x25519.py", line 39, in generate
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/backends/openssl/backend.py", line 117, in <module>
    from cryptography.hazmat.bindings.openssl import binding
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/bindings/openssl/binding.py", line 14, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: dlopen(/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/bindings/_openssl.so, 2): Symbol not found: _DTLS_client_method
  Referenced from: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/cryptography/hazmat/bindings/_openssl.so
  Expected in: flat namespace

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/transport.py", line 120, in _join_lingering_threads
    for thr in _active_threads:
TypeError: 'NoneType' object is not iterable
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/Users/darren/Library/Python/2.7/lib/python/site-packages/paramiko/transport.py", line 120, in _join_lingering_threads
    for thr in _active_threads:
TypeError: 'NoneType' object is not iterable

Has anyone else had this problem or have a fix for it?

Answer 1

I was able to install python2 for Apple Silicon/M1 from nix and then I had to manually install pip which let me install cryptography & cffi (I had libffi installed from brew). Here are the steps I used, I'm not sure if they're all necessary, but it's what I did:

• Install Nix
  • see https://nixos.org/manual/nix/stable/#sect-macos-installation for more details, but if you have a mac with a T2 chip or M1 this will still be encrypted
  • sh <(curl -L https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume --daemon
• Install python2 from nix
  • nix-env -i python-2.7.18
  • verify that you're using this python in a new terminal session (you'll need to start a new shell to get the updated path)
    • run which python
    • if it doesn't return something like /Users/user/.nix-profile/bin/python you'll want to add ~/.nix-profile/bin to the front of your path, start a new shell, and run which python again
• Install pip
  • See https://pip.pypa.io/en/stable/installing/#installing-with-get-pip-py for more details, but if you curl the get-pip.py in this url and run it with python 2 it'll tell you to instead download and use the get-pip.py from https://bootstrap.pypa.io/pip/2.7/get-pip.py
  • curl https://bootstrap.pypa.io/pip/2.7/get-pip.py -o get-pip.py
  • python ./get-pip.py
• Use brew to install libiff and openssl@1.1
  • I had done this step previously and I'm not 100% sure it's necessary, but I believe it is
  • brew install libiff openssl@1.1
  • Add the following to your .bashrc, .zshrc, .profile, or wherever you set your environment variables:

export LDFLAGS="-L$(brew --prefix openssl@1.1)/lib"
export CFLAGS="-I$(brew --prefix openssl@1.1)/include"

• use pip to install cryptography
  • pip install cryptography

After this I had a functional python with cryptography installed and I was then able to install fabric (via pip install fabric==1.14.1 since 1.14.1 is the version I need for this old project, use what you need)

Edit: Thanks to @CharlesDuffy for all of his help and as he correctly mentions in the comments bellow, you could (and probably should) install all of the dependencies and cryptography from nix instead of using a combination of brew and nix. I had brew and the dependancies installed before I had Nix installed so I haven't tested it out, but it should work.