Creating a Python Package, the Cruel Way

This is how you can create a package of Python code so that you can use the module in various projects.

Let’s assume this is the code your module has:

STUFF_VERSION = "1.1-dev"

def get_stuff():
    return "This is stuff. Use it wisely."

Of course the module can be much more complicated with various functions and classes and whatnot, this is just a short example.

Let’s then create the directory structure for the package. First, start with a working directory dedicated to this package, like stuff. There, create a directory that actually contains the code of your module. Here, that directory is also called stuff. And, in that directory, create file __init__.py that contains the code above.

So we have these:

stuff
\------ stuff
           \------ __init__.py

Again, __init__.py contains the code with variable STUFF_VERSION and function get_stuff().

Now, in the working directory, create setup.py with this content:

from setuptools import setup, find_packages

setup(
        name="stuff",
        version="1.1-dev",
        description="The Stuff package by Me",
        packages=find_packages(),
    )

Thus, we now have:

stuff
|------ setup.py
\------ stuff
           \------ __init__.py

That’s all the files we need in this example.

Now, I can install the package for me to use in all my Python projects with pip. In the working directory stuff (the upper one) I can run:

markku@server:~/stuff$ pip3 install .
Processing /home/markku/stuff
Building wheels for collected packages: stuff
  Running setup.py bdist_wheel for stuff ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-bmviu4af/wheels/4a/34/43/8df98ce9cd0b73a3eb668ac0136e2d4bdca9f63065a2830fa9
Successfully built stuff
Installing collected packages: stuff
Successfully installed stuff-1.1.dev0
markku@server:~/stuff$ 

I’m using a Debian testing install for this run. Let’s then test with another code in ~/test/test.py:

import stuff

print("Stuff version {} is saying: '{}'".format(stuff.STUFF_VERSION, stuff.get_stuff()))

Testing it:

markku@server:~/test$ python3 test.py
Stuff version 1.1-dev is saying: 'This is stuff. Use it wisely.'
markku@server:~/test$

So, it’s working, but what happened? Pip installed the code there:

markku@server:~/test$ cat ~/.local/lib/python3.7/site-packages/stuff/__init__.py
STUFF_VERSION = "1.1-dev"

def get_stuff():
     return "This is stuff. Use it wisely."
markku@server:~/test$

That’s in a user-specific directory structure that Python searches when importing modules in test.py. Note that I didn’t use sudo to run the pip3 install . command, so it was not going anywhere system-wide. (Exercise: run sudo pip3 install . in the stuff directory, and use for example find /usr -name stuff to find the location of the package.)

This is simple way to reuse a module if the code is in a git repository and you can clone the repo on the server you need it. Just clone the repo and run pip3 install . and the module can be used in other projects on that same server.

More sophisticated way involves using source distributions or built distributions. See https://packaging.python.org/tutorials/packaging-projects/ for more information, and test with python3 setup.py sdist or similar commands. If using git, prepare to edit your .gitignore to filter out some of the files that these operations generate in the directory structure.

Update: Chris Wilcox (@chriswilcox47) had a nice presentation in PyCon 2019: Shipping your first Python package and automating future publishing