Creating your personal Python environment using virtualenv

Often you are working on a computer where you do not have the user privileges to install custom Python modules (or software in general). Or you do not want to mess with the operating systems Python environment. Then, sometimes you just want (or have to) install python modules with a specific version, e.g. to get the exact behaviour as on a given server. This is where virtualenv comes in. Virtualenv is a tool to create virtual Python environments, that are exactly like a complete Python installation and are therefore fully customizable but do not need root privileges.

In this tutorial, I assume you already have a computer where you have Python up and running, and that you know how to run it in your shell. This is e.g. the case for all Ubuntu based systems.

Now let’s make use of virtualenv. I’m going to assume that you will install it below your $HOME directory. You can actually install it in any directory, so no need to stick to $HOME. But for now, lets create the virtual environment in $HOME/pyenv:

virtualenv $HOME/pyenv

It’s just that easy! No additional software needs to be downloaded or installed as long as Python and virtualenv are already available on your system.

After you created the virtual Python environment, you need to activate it. Here is how it is done in the bash:

source $HOME/pyenv/bin/activate

I personally added the above line to my .bashrc, so the virtualenv gets activated every time I start a shell. Take note, that the virtualenv is not a constantly running program. It does not use any CPU or memory, as it is only setting environment variables, so that the correct Python installation with the correct modules is called, when you run Python.

To deactivate the virtual environment, just type deactivate. I personally leave my custom Python environment always active. If it interferes with e.g. system modules (I run it on Ubuntu), I could simply deactivate it this way. I never ever had to deactiave it to date. So I just keep it. Another reason to deactivate it would be to switch to another virtualenv, e.g. to switch to a specific configuration that is mimicing the Python environment of a specific computing node. Or you could create one virtualenv running Python 2.7, and another using Python 3, which is very convenient for making the transition between the two.

But now that we have our virtual Python environment up and running, we can use pip to install all kind of modules (and specific versions) we want in this environment, without needing root privileges. Here is what packages I usually install:

pip install --upgrade pip
pip install requests[security]
pip install --upgrade ipython
pip install --upgrade jupyter
pip install --upgrade numpy
pip install --upgrade scipy
pip install --upgrade matplotlib
pip install --upgrade astropy
pip install --upgrade sklearn
pip install --upgrade photutils
pip install --upgrade montage-wrapper
pip install --upgrade pandas
pip install --upgrade emcee
pip install --upgrade pymc
pip install --upgrade seaborn
pip install --upgrade glueviz
pip install --upgrade aplpy
pip install --upgrade fits_tools
pip install --upgrade healpy
pip install --upgrade uncertainties
pip install --upgrade pysqlite2
pip install --upgrade pillow

There are a lot of astronomy-specific modules in there, so you might want to select only those modules you really need.

Now what to do if Python is available on your system, but you do not have root privileges (which often is the case at PCs at work), and virtualenv is not installed (check by just typing virtualenv in a shell to see if the command is available)?  – Well, you can either ask an Admin to install virtualenv (should be no big deal), or you can install it under your user direcory using Pip, the program to install packaages from the Python Package Index (PyPI).

For this purpose we create a directory called python_custom under $HOME, install virtualenv there, and make it available through the $PYTHONPATH:

# create Python directory structure for custom virtualenv installation
cd $HOME
mkdir python_custom
# install virtualenv
pip install virtualenv --root=$HOME/python_custom/
export PYTHONPATH=$PYTHONPATH:$HOME/python_custom/usr/lib/python2.7/dist-packages/

Now if you had to install virtualenv yourself, call $HOME/python_custom/usr/bin/virtualenv in the step where you create the environment instead of simply calling virtualenv.

I hope this helps!

Edit: if you do not have Python installed on your computer, I recommend using an Anaconda installation. This is a great option, esp. as you can also use virtualenv with Anaconda (thanks Benjamin for pointing this out) for the fine-tuning and the seamless switch between different Python environments.

To use pysqlite2 with load_modules enabled, install pysqlite:

> pip install –global-option=build_ext –global-option=’-I$USER/anaconda3/include/’ pysqlite