Delete unused packages from requirements file

Question

Is there any easy way to delete no-more-using packages from requirements file?

I wrote a bash script for this task but, it doesn't work as I expected. Because, some packages are not used following their PyPI project names. For example;

dj-database-url

package is used as

dj_database_url

My project has many packages in its own requirements file, so, searching them one-by-one is too messy, error-prone and takes too much time. As I searched, IDEs don't have this property, yet.

Answer 1

You can use Code Inspection in PyCharm.

1. Delete the contents of your requirements.txt but keep the empty file.
2. Load your project in,
3. PyCharm go to Code -> Inspect code....
4. Choose Whole project option in dialog and click OK. In inspection results panel locate Package requirements section under Python (note that this section will be showed only if there is any requirements.txt or setup.py file). The section will contain one of the following messages:

• Package requirement '<package>' is not satisfied if there is any package that is listed in requirements.txt but not used in any .py file.
• Package '<package>' is not listed in project requirements if there is any package that is used in .py files, but not listed in requirements.txt.

You are interested in the second inspection. You can add all used packages to requirements.txt by right clicking the Package requirements section and selecting Apply Fix 'Add requirements '<package>' to requirements.txt'. Note that it will show only one package name, but it will actually add all used packages to requirements.txt if called for section.

If you want, you can add them one by one, just right click the inspection corresponding to certain package and choose Apply Fix 'Add requirements '<package>' to requirements.txt', repeat for each inspection of this kind.

After that you can create clean virtual environment and install packages from new requirements.txt.

Also note that PyCharm has import optimisation feature, see Optimize imports.... It can be useful to use this feature before any other steps listed above.

Answer 2

The best bet is to use a (fresh) python venv/virtual-env with no packages, or only those you definitely know you need, test your package - installing missing packages with pip as you hit problems which should be quite quick for most software then use the pip freeze command to list the packages you really need. Better you you could use pip wheel to create a wheel with the packages in.
The other approach would be to:

1. Use pylint to check each file for unused imports and delete them, (you should be doing this anyway),
2. Run your tests to make sure that it was right,
3. Use a tool like snakefood or snakefood3 to generate your new list of dependencies

Note that for any dependency checking to work well it is advisable to avoid conditional import and import within functions.

Also note that to be sure you have everything then it is a good idea to build a new venv/virtual-env and install from your dependencies list then re-test your code.

Answer 3

You can find obsolete dependencies by using deptry, a command line utility that checks for various issues with a project's dependencies, such as unused, missing or transitive dependencies.

Add it to your project with

pip install deptry

and then run

deptry .

Example output:

Scanning 2 files...

requirements.txt: DEP002 'pandas' defined as a dependency but not used in the codebase
Found 1 dependency issue.

Note that for the best results, you should be using a virtual environment for your project, see e.g. here.

Disclaimer: I am the author of deptry.

Answer 4

In pycharm go to Tools -> Sync Python Requirements. There's a 'Remove unused requirements' checkbox.

Answer 5

I've used with success pip-check-reqs.

With command pip-extra-reqs your_directory it will check for all unused dependencies in your_directory

Install it with pip install pip-check-reqs.