A Short History of Python Packaging

The (near) past¶

At one point in time, there was no such thing as pip or PyPI. To share packages, people would structure their code in a particular way using the distutils library (with a setup.py file), and manually (by-hand) distribute the result to their colleagues.

Then, the Python Package Index (PyPI) was created, and distutils learned how to upload to PyPI. Packages stored on PyPI could be manually downloaded and installed.

setuptools was subsequently invented, which created a new binary distribution format (eggs), an automatic installation tool (easy_install), and other features. With setuptools, you could install packages to your computer from PyPI.

Skipping over some details, the wheel format and pip were invented and setuptools was the way to author a Python package, and pip was the preferred way to install it from PyPI.

The (old) future¶

Now we’re closer to the present. In 2015 a PEP was accepted that set Python on a new trajectory. Instead of a “single” tool that was used to build every Python package, now anyone could create a “build backend” that takes a Python project and builds a Source distribution and Binary distribution. This was PEP 517.

This made it possible to get rid of the imperative setup.py and replace it with something that was nicer to read and write (for both humans and machines!). But, every build-backend had its own configuration format — this was not necessarily an improvement! Another PEP, PEP 621 introduced a way to author “core project metadata” like the project description in a standardised file pyproject.toml. There have been a few more PEPs since.

This introduces a lot of new terms:

Which can help us to understand the status quo: