Introduction
Anaconda is a completely free Python distribution (including for commercial use and redistribution). It includes over 195 of the most popular Python packages for science, math, engineering, data analysis.
Usage
The usage is: module load anaconda3/2019.07.py3 OR module load anaconda/4.00py2.7 OR: module load anaconda/4.00py3.5 You need to run this after loading the modules: source /sw/anaconda/py27-ver400/etc/profile.d/conda.sh #if using module load anaconda/4.00py2.7 OR source /sw/anaconda/py3-ver400/etc/profile.d/conda.sh #if using “module load anaconda/4.00py3.5” Manual Usage: You will need to activate the environment. For example, to activate biopython source activate biopython discarding /sw/anaconda/2.0.1/bin from PATH prepending /sw/anaconda/2.0.1/envs/biopython/bin to PATH To deactivate this environment, use: source deactivate Module Usage: ============= For example,You can use this to load biopython using python 2.7 version (anaconda) : module load anaconda/py27biopython1.66 For python 3.5 (anaconda env), use this: module load anaconda/py35biopython1.66 Anaconda environment need not be loaded separately. However, it can be loaded as follows if needed for other reasons: module load anaconda/4.00py2.7 module load anaconda/4.00py3.5
Useful Commands
To get a list of environment in anaconda: conda info -e To create an environment: conda create -n biopython biopython De-Activate an environment source deactivate Deleting an environment conda remove -n biopython
conda create -h
usage: conda create [-h] [--yes] [--dry-run] [-f] [--file FILE] [--unknown]
[--no-deps] [-m] [--use-index-cache] [--use-local]
[--no-pin] [-c CHANNEL] [--override-channels]
[-n NAME | -p PATH] [-q] [--alt-hint] [--clone ENV]
[--no-default-packages]
[package_spec [package_spec ...]]
Create a new conda environment from a list of specified packages. To use the created environment, use 'source activate envname' look in that directory first. This command requires either the -n NAME or -p PREFIX option.
positional arguments:
package_spec package versions to install into conda environment
optional arguments:
-h, --help show this help message and exit
--yes do not ask for confirmation
--dry-run only display what would have been done
-f, --force force install (even when package already installed),
implies --no-deps
--file FILE read package versions from FILE
--unknown use index metadata from the local package cache (which
are from unknown channels)
--no-deps do not install dependencies
-m, --mkdir create prefix directory if necessary
--use-index-cache use cache of channel index files
--use-local use locally built packages
--no-pin don't use pinned packages
-c CHANNEL, --channel CHANNEL
additional channel to search for packages. These are
URLs searched in the order they are given (including
file:// for local directories). Then, the defaults or
channels from .condarc are searched (unless
--override-channels is given). You can use 'defaults'
to get the default packages for conda, and 'system' to
get the system packages, which also takes .condarc
into account. You can also use any name and the
.condarc channel_alias value will be prepended. The
default channel_alias is http://conda.binstar.org/
--override-channels Do not search default or .condarc channels. Requires
--channel.
-n NAME, --name NAME name of environment (in /sw/anaconda/2.0.1/envs)
-p PATH, --prefix PATH
full path to environment prefix (default:
/sw/anaconda/2.0.1)
-q, --quiet do not display progress bar
--alt-hint Use an alternate algorithm to generate an
unsatisfiable hint
--clone ENV path to (or name of) existing local environment
--no-default-packages
ignore create_default_packages in condarc file
examples:
conda create -n myenv sqlite
Installation
bash Anaconda-2.0.1-Linux-x86_64.sh Anaconda will now be installed into this location: /root/anaconda - Press ENTER to confirm the location - Press CTRL-C to abort the installation - Or specify an different location below [/root/anaconda] >>> /sw/anaconda/2.0.1 PREFIX=/sw/anaconda/2.0.1 installing: python-2.7.7-0 ... installing: conda-3.5.5-py27_0 ... installing: conda-build-1.3.5-py27_0 ... installing: _license-1.1-py27_0 ... installing: argcomplete-0.6.7-py27_0 ... installing: astropy-0.3.2-np18py27_0 ... installing: atom-0.3.7-py27_0 ... installing: beautiful-soup-4.3.1-py27_0 ... installing: binstar-0.5.3-py27_0 ... installing: bitarray-0.8.1-py27_0 ... installing: blaze-0.5.0-np18py27_1 ... installing: blz-0.6.2-np18py27_0 ... installing: bokeh-0.4.4-np18py27_1 ... installing: boto-2.28.0-py27_0 ... installing: cairo-1.12.2-2 ... installing: casuarius-1.1-py27_0 ... installing: cdecimal-2.3-py27_0 ... installing: chaco-4.4.1-np18py27_0 ... installing: colorama-0.2.7-py27_0 ... installing: configobj-5.0.5-py27_0 ... installing: cubes-0.10.2-py27_4 ... installing: curl-7.30.0-0 ... installing: cython-0.20.1-py27_0 ... installing: datashape-0.2.0-np18py27_1 ... installing: dateutil-2.1-py27_2 ... installing: docutils-0.11-py27_0 ... installing: dynd-python-0.6.2-np18py27_0 ... installing: enable-4.3.0-np18py27_2 ... installing: enaml-0.9.1-py27_1 ... installing: flask-0.10.1-py27_1 ... installing: freetype-2.4.10-0 ... installing: future-0.12.1-py27_0 ... installing: gevent-1.0.1-py27_0 ... installing: gevent-websocket-0.9.3-py27_0 ... installing: greenlet-0.4.2-py27_0 ... installing: grin-1.2.1-py27_1 ... installing: h5py-2.3.0-np18py27_0 ... installing: hdf5-1.8.9-1 ... installing: ipython-2.1.0-py27_2 ... installing: ipython-notebook-2.1.0-py27_0 ... installing: ipython-qtconsole-2.1.0-py27_0 ... installing: itsdangerous-0.24-py27_0 ... installing: jdcal-1.0-py27_0 ... installing: jinja2-2.7.2-py27_0 ... installing: jpeg-8d-0 ... installing: kiwisolver-0.1.2-py27_0 ... installing: launcher-0.1.5-py27_0 ... installing: lcms-1.19-0 ... installing: libdynd-0.6.2-0 ... installing: libpng-1.5.13-1 ... installing: libsodium-0.4.5-0 ... installing: libtiff-4.0.2-0 ... installing: libxml2-2.9.0-0 ... installing: libxslt-1.1.28-0 ... installing: llvm-3.3-0 ... installing: llvmpy-0.12.6-py27_0 ... installing: lxml-3.3.5-py27_0 ... installing: markupsafe-0.18-py27_0 ... installing: matplotlib-1.3.1-np18py27_1 ... installing: mock-1.0.1-py27_0 ... installing: mpi4py-1.3-py27_0 ... installing: mpich2-1.4.1p1-0 ... installing: multipledispatch-0.4.3-py27_0 ... installing: networkx-1.8.1-py27_0 ... installing: nltk-2.0.4-np18py27_0 ... installing: nose-1.3.3-py27_0 ... installing: numba-0.13.2-np18py27_0 ... installing: numexpr-2.3.1-np18py27_0 ... installing: numpy-1.8.1-py27_0 ... installing: openpyxl-1.8.5-py27_0 ... installing: openssl-1.0.1h-0 ... installing: pandas-0.14.0-np18py27_0 ... installing: patsy-0.2.1-np18py27_0 ... installing: pep8-1.5.6-py27_0 ... installing: pil-1.1.7-py27_1 ... installing: pip-1.5.6-py27_0 ... installing: pixman-0.26.2-0 ... installing: ply-3.4-py27_0 ... installing: psutil-2.1.1-py27_0 ... installing: py-1.4.20-py27_0 ... installing: py2cairo-1.10.0-py27_1 ... installing: pycosat-0.6.1-py27_0 ... installing: pycparser-2.10-py27_0 ... installing: pycrypto-2.6.1-py27_0 ... installing: pycurl-7.19.3.1-py27_2 ... installing: pyface-4.4.0-py27_0 ... installing: pyflakes-0.8.1-py27_0 ... installing: pygments-1.6-py27_0 ... installing: pyparsing-2.0.1-py27_0 ... installing: pyqt-4.10.4-py27_0 ... installing: pytables-3.1.1-np18py27_0 ... installing: pytest-2.5.2-py27_0 ... installing: pytz-2014.3-py27_0 ... installing: pyyaml-3.11-py27_0 ... installing: pyzmq-14.3.0-py27_0 ... installing: qt-4.8.5-0 ... installing: readline-6.2-2 ... installing: redis-2.6.9-0 ... installing: redis-py-2.9.1-py27_0 ... installing: requests-2.3.0-py27_0 ... installing: rope-0.9.4-py27_1 ... installing: runipy-0.1.0-py27_0 ... installing: scikit-image-0.10.0-np18py27_0 ... installing: scikit-learn-0.14.1-np18py27_1 ... installing: scipy-0.14.0-np18py27_0 ... installing: setuptools-3.6-py27_0 ... installing: sip-4.15.5-py27_0 ... installing: six-1.6.1-py27_0 ... installing: sphinx-1.2.2-py27_0 ... installing: spyder-2.3.0rc1-py27_0 ... installing: spyder-app-2.3.0rc1-py27_0 ... installing: sqlalchemy-0.9.4-py27_0 ... installing: sqlite-3.8.4.1-0 ... installing: ssl_match_hostname-3.4.0.2-py27_0 ... installing: statsmodels-0.5.0-np18py27_0 ... installing: sympy-0.7.5-py27_0 ... installing: system-5.8-1 ... installing: theano-0.6.0-np18py27_0 ... installing: tk-8.5.15-0 ... installing: tornado-3.2.1-py27_0 ... installing: traits-4.4.0-py27_0 ... installing: traitsui-4.4.0-py27_0 ... installing: ujson-1.33-py27_0 ... installing: util-linux-2.21-0 ... installing: werkzeug-0.9.6-py27_0 ... installing: xlrd-0.9.3-py27_0 ... installing: xlsxwriter-0.5.5-py27_0 ... installing: xlwt-0.7.5-py27_0 ... installing: yaml-0.1.4-0 ... installing: zeromq-4.0.4-0 ... installing: zlib-1.2.7-0 ... installing: anaconda-2.0.1-np18py27_0 ... installing: _cache-0.0-x0 ... Python 2.7.7 :: Continuum Analytics, Inc. creating default environment... installation finished. Do you wish the installer to prepend the Anaconda install location to PATH in your /root/.bashrc ? [yes|no] [no] >>> No Thank you for installing Anaconda!
The binaries are in /sw/anaconda/2.0.1/bin. You can use the module file to load the environment (recommended):
module load anaconda/2.0.1
OR prepend (not advised) to your ~/.bashrc file:
PATH=/sw/anaconda/2.0.1/bin:$PATH
Extra Notes
Dry Run
To see what packages will be downloaded and/or used in an environment before creating it, you can use the --dry-run option.
conda create --dry-run -n foo2 python
e.g.:
conda create --dry-run -n biopython biopython
Fetching package metadata: ..
Solving package specifications: .
Package plan for installation in environment /sw/anaconda/2.0.1/envs/biopython1.64:
The following packages will be downloaded:
package | build
---------------------------|-----------------
biopython-1.64 | np18py27_0 2.1 MB
python-2.7.8 | 0 11.3 MB
------------------------------------------------------------
Total: 13.5 MB
The following packages will be linked:
package | build
---------------------------|-----------------
biopython-1.64 | np18py27_0 hard-link
numpy-1.8.1 | py27_0 hard-link
openssl-1.0.1h | 0 hard-link
python-2.7.8 | 0 hard-link
readline-6.2 | 2 hard-link
sqlite-3.8.4.1 | 0 hard-link
system-5.8 | 1 hard-link
tk-8.5.15 | 0 hard-link
zlib-1.2.7 | 0 hard-link
Dry run: exiting
Create an environment
conda create -n biopython biopython
Fetching package metadata: ..
Solving package specifications: .
Package plan for installation in environment /sw/anaconda/2.0.1/envs/biopython:
The following packages will be downloaded:
package | build
---------------------------|-----------------
biopython-1.64 | np18py27_0 2.1 MB
python-2.7.8 | 0 11.3 MB
------------------------------------------------------------
Total: 13.5 MB
The following packages will be linked:
package | build
---------------------------|-----------------
biopython-1.64 | np18py27_0 hard-link
numpy-1.8.1 | py27_0 hard-link
openssl-1.0.1h | 0 hard-link
python-2.7.8 | 0 hard-link
readline-6.2 | 2 hard-link
sqlite-3.8.4.1 | 0 hard-link
system-5.8 | 1 hard-link
tk-8.5.15 | 0 hard-link
zlib-1.2.7 | 0 hard-link
Proceed ([y]/n)? y
Fetching packages ...
biopython-1.64 100% |##########################| Time: 0:00:15 144.95 kB/s
python-2.7.8-0 100% |##########################| Time: 0:00:14 821.86 kB/s
Extracting packages ...
[ COMPLETE ] |#######################| 100%
Linking packages ...
[ COMPLETE ] |#######################| 100%
#
# To activate this environment, use:
# $ source activate biopython
#
# To deactivate this environment, use:
# $ source deactivate
#
Installed Environment
Useful commands
Useful commands: conda info --envs conda list conda activate
Sample PBS file
#!/bin/bash #PBS -N condatest #PBS -m abe #PBS -M YourEMAIL@griffith.edu.au #PBS -l select=1:ncpus=1:mem=4gb,walltime=10:00:00 module load anaconda/4.00py2.7 source /sw/anaconda/py27-ver400/etc/profile.d/conda.sh ##use /sw/anaconda/py3-ver400/etc/profile.d/conda.sh if using “module load anaconda/4.00py3.5” echo "Starting job: " cd $PBS_O_WORKDIR python <pythonFile> >>>An example of the python file>>>>>>>>> cat pythonFile.py import tensorflow as tf >>>>>>>>>>>>>>
Activate an environment
source /sw/anaconda/py27-ver400/etc/profile.d/conda.sh OR source /sw/anaconda/py3-ver400/etc/profile.d/conda.sh if using “module load module load anaconda/4.00py3.5” source activate biopython e.g: source activate biopython discarding /sw/anaconda/2.0.1/bin from PATH prepending /sw/anaconda/2.0.1/envs/biopython/bin to PATH
De-Activate an environment
source deactivate
Deleting an environment
conda remove -n biopython
n060 anaconda installation
module load anaconda/5.3.1py3 conda info --envs # conda environments: # base * /sw/centos7/anaconda/5.3.1/py3 accelerate /sw/centos7/anaconda/5.3.1/py3/envs/accelerate biopython /sw/centos7/anaconda/5.3.1/py3/envs/biopython deeplearning /sw/centos7/anaconda/5.3.1/py3/envs/deeplearning keras /sw/centos7/anaconda/5.3.1/py3/envs/keras tensorflow-gpu /sw/centos7/anaconda/5.3.1/py3/envs/tensorflow-gpu e.g module load anaconda/5.3.1py3 source activate keras pip install soundfile
Installing packages without root access
Instructions from: https://medium.com/@danielwzou/installing-spacy-and-other-packages-without-sudo-or-root-privileges-using-anaconda-59571ec0ecad
First, navigate to your home directory.
module load anaconda/5.3.1py3
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
sh Miniconda3-latest-Linux-x86_64.sh
Now, we have a local installation of the Conda package manager at ~/miniconda3
Let’s add conda to our bash profile so that we can call it from the command line
vim .bash_profile
i
Add this line to the file:
export PATH=$HOME/miniconda3/bin:$PATH
esc :wq
Now, let’s reload our .bash_profile
source .bash_profile
And check that our installation was completed successfully
which conda
Which should return ~/miniconda3/bin/conda
Now that Conda is installed, we can install Spacy or other Python packages
conda install spacy
python -m spacy download en
Now, let’s make sure that Spacy is installed correctly
python
import spacy
nlp = spacy.load('en')
doc = nlp(u'Hello World!')
print([w.lemma_ for w in doc])
Now that you have Conda installed, you can use it to install many packages locally.
Anaconda on the new cluster
module load anaconda3/2019.07py3
This should have packages like the tensorflow env. You can find with this:
conda info --envs
>>>>>
conda info --envs
# conda environments:
#
/sw/anaconda2/2019.07/envs/bio
/sw/anaconda2/2019.07/envs/bioinfoPy27
base * /sw/anaconda3/2019.07
3point6 /sw/anaconda3/2019.07/envs/3point6
R /sw/anaconda3/2019.07/envs/R
bioinformatics /sw/anaconda3/2019.07/envs/bioinformatics
>>>>>>>
This will give the packages in the base:
conda list
source activate 3point6
Next run: " conda list"
This will list the packages installed in that environment.
It shows among many others:
pytorch 1.3.1 cuda100py36h53c1284_0
tensorboard 2.0.0 pyhb230dea_0
tensorflow 2.0.0 mkl_py36hef7ec59_0
tensorflow-base 2.0.0 mkl_py36h9204916_0
tensorflow-estimator 2.0.0 pyh2649769_0
termcolor 1.1.0 py_2 conda-forge
theano 1.0.4 py36hf484d3e_1000 conda-forge
Remember this cluster has no GPU (currently)
Another sample pbs file
#!/bin/bash #PBS -N condatest #PBS -m abe #PBS -M YourEMAIL@griffith.edu.au #PBS -l select=1:ncpus=1:mem=4gb,walltime=10:00:00 cd $PBS_O_WORKDIR module load anaconda3/2019.07py3 source activate bioinformatics trinity <trinityinput>
Sample python code to write out a plot
vi plot.py
>>>>>>>>>>>>>>>>>
import matplotlib.pyplot as plt
import numpy as np
np.random.seed(19680801)
data = np.random.randn(2, 100)
fig, axs = plt.subplots(2, 2, figsize=(5, 5))
axs[0, 0].hist(data[0])
axs[1, 0].scatter(data[0], data[1])
axs[0, 1].plot(data[0], data[1])
axs[1, 1].hist2d(data[0], data[1])
plt.show()
plt.savefig('test')
>>>>>>>>>>>>>>>>>>
python plot.py would generate a plot named test.png
Reference: https://matplotlib.org/3.4.3/tutorials/introductory/sample_plots.html
Sample codes are here:
https://matplotlib.org/3.4.3/_downloads/55a714c8a6e4dde456b09326202fe492/sample_plots.py
https://matplotlib.org/3.4.3/_downloads/003118d8eb1e7df55671bcdf18bfac3c/sample_plots.ipynb
Reference
1. http://davebehnke.com/using-python-anaconda-distribution.html
2. https://store.continuum.io/cshop/anaconda/
3. https://medium.com/@danielwzou/installing-spacy-and-other-packages-without-sudo-or-root-privileges-using-anaconda-59571ec0ecad
4. http://continuum.io/downloads#all