TODO: Creating, activating, using conda environments….
| Code Block |
|---|
module load lang # gateway module
module load Anaconda3 # actual module |
| Code Block |
|---|
conda create --name myenv |
...
Recommendations
Use environments (e.g. conda)
Use the parallel file system (i.e.
/scratch/<group>/<username>/)
Note that HOME is not a good choice for storing Python packages since
it is not designed for parallel use,
it will very likely lead to quota issues (HOME is limited to 20 GB), and
it causes unnecessary backups of reproducible data.
Self-Learning Mini-Tutorials
Conda
| Code Block |
|---|
which python # load module ml lang ml Miniforge3 which python conda create --name myenv python=3.9.5 conda activate myenv # won't work if shell integration hasn't been set up already # shell integration conda init . $HOME/.bashrc # update shell # or `source $(conda info --base)/etc/profile.d/conda. |
...
sh` for immediate + non-permanent conda activate myenv # works now which python conda list conda install numpy # so far so good, but where is our environment (and the packages etc.) located? conda env list # it shouldn't live in $HOME but on the parallel FS # how to resolve this? # 1) provide `--prefix <path>` to `conda create` (and then full path to `conda activate`) # 2) conda config --add envs_dirs /scratch/<group>/<username>/.conda/envs # conda config --add pkgs_dirs /scratch/<group>/<username>/.conda/pkgs # let's recreate it properly... conda deactivate myenv # conda env remove --name myenv .... but let's just rm -rf $HOME/.conda conda config --add envs_dirs /scratch/pc2-mitarbeiter/bauerc/.conda/envs conda config --add pkgs_dirs /scratch/pc2-mitarbeiter/bauerc/.conda/pkgs cat $HOME/.condarc # (potentially) redo the above.... # then `which python` |
Pip (global)
| Code Block |
|---|
which python
pip # command not found
# load module
ml lang
ml Python/3.9.5
which python
which pip
# note that the default path for global packages etc. is $HOME/.local/lib/python<ver>/site-packages
# to change it we can put the following into our .bashrc
# export PYTHONUSERBASE=/scratch/<group>/<username>/.local
export PYTHONUSERBASE=/scratch/pc2-mitarbeiter/bauerc/.local
# you might also want to add
export PATH=/scratch/pc2-mitarbeiter/bauerc/.local/bin:$PATH
pip install numpy
ls /scratch/pc2-mitarbeiter/bauerc/lib/python3.9/site-packages |
However, if you want/need to use it, we generally recommend to use `pip` within a conda environment!
Poetry
| Code Block |
|---|
export POETRY_CACHE_DIR=/scratch/pc2-mitarbeiter/bauerc/.pypoetry # or `poetry config cache-dir /scratch/pc2-mitarbeiter/bauerc/.pypoetry` (config file is here: `$HOME/.config/pypoetry/config.toml`) poetry new myproject cd myproject poetry show # list installed packages poetry add numpy poetry show # how to run something in the environment or use it interactively? which python # still points to the general one.... WHY?! python import numpy # fails.... WHY?! poetry run which python poetry shell which python python import numpy # works within the poetry shell exit # leave the shell # where is the actual virtualenv: `poetry env list --full-path` |
Virtualenv (this is not `python -m venv`!)
| Code Block |
|---|
#virtualenv --python="/usr/bin/python2.6" "/path/to/new/virtualenv/" virtualenv venv source venv/bin/activate # activate local virtual environment which python pip list pip install |
...
numpy
python
import numpy # works
deactivate # deactivates local virtual environment
python
import numpy # fails (as expected) |