How Editable Installs Work - Packaging in Python

Install your Package!¶

Installing a package takes time and manual effort, but it’s an important part of using someone elses software; Python packages are not just Python code that we can add to the PYTHONPATH, they’re meant to be installed to be used. But, it’s not convenient to keep re-installing it when you make changes; seeing the result of the edits to your code goes from a simple file-save to running a shell command. How can you see your changes in real-time?

Install your package editably¶

Editable wheel: A regular wheel file containing special file that enable editing of the package contents (using PEP 660).

The solution, as ever, can be found in pip. pip can build and install an editable wheel for your package, using -e:

pip install -e .

import os
import pathlib
import tempfile
path = os.getcwd()
tmp_path = pathlib.Path(tempfile.mkdtemp())
link_src = pathlib.Path(path).parent / "arrow-to-knee"
package_path = (tmp_path / link_src.name)
package_path.symlink_to(link_src)
os.chdir(package_path)

os.environ['HATCH_INTERACTIVE'] = '0'

We can build this wheel by hand:

!hatch build -t wheel:editable .

Creating environment: hatch-build

Checking dependencies

Syncing dependencies

Inspecting build dependencies
──────────────────────────────────── wheel ─────────────────────────────────────

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/runner/.local/share/hatch/env/virtual/arrow-to-knee/h6Q3tDJc/hatch-build/lib/python3.12/site-packages/hatchling/__main__.py", line 6, in <module>
    sys.exit(hatchling())
             ^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/arrow-to-knee/h6Q3tDJc/hatch-build/lib/python3.12/site-packages/hatchling/cli/__init__.py", line 26, in hatchling
    command(**kwargs)
  File "/home/runner/.local/share/hatch/env/virtual/arrow-to-knee/h6Q3tDJc/hatch-build/lib/python3.12/site-packages/hatchling/cli/build/__init__.py", line 82, in build_impl
    for artifact in builder.build(
                    ^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/arrow-to-knee/h6Q3tDJc/hatch-build/lib/python3.12/site-packages/hatchling/builders/plugin/interface.py", line 157, in build
    artifact = version_api[version](directory, **build_data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/arrow-to-knee/h6Q3tDJc/hatch-build/lib/python3.12/site-packages/hatchling/builders/wheel.py", line 547, in build_editable
    return self.build_editable_detection(directory, **build_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/arrow-to-knee/h6Q3tDJc/hatch-build/lib/python3.12/site-packages/hatchling/builders/wheel.py", line 550, in build_editable_detection
    from editables import EditableProject
ModuleNotFoundError: No module named 'editables'

It contains very little!

!zipinfo ./arrow_to_knee*.whl

zipinfo:  cannot find or open ./arrow_to_knee*.whl, ./arrow_to_knee*.whl.zip or ./arrow_to_knee*.whl.ZIP.

No zipfiles found.

What do we see here that’s not familiar to us thus far? It’s the /_arrow_to_knee.pth file. More on that in a moment.

Python is a detective¶

To go any further, we are going to need to understand a little bit about how Python import statements actually work. We’re not going to dig into the line-by-line account of what Python is doing under the hood, but we are going to appreciate how extensible and powerful this part of the interpreter is.

At startup, Python runs some code that prepares sys.path. This includes finding .pth files in known directories, and processing them according to some rules:

.pth files can contain paths that are added to sys.path.
They can also contain import statements, which are exec by the interpreter.

The latter feature is somewhat incredible — one can use .pth files (slightly in poor form) to write code that runs at Python startup!

Editable wheel packages are designed around this .pth mechanism — they are designed to ship a .pth file that either exposes the source directory, or runs some code that handles this slightly more explicitly.