Managing Your Own Software

Managing Your Own Software

In addition to the software provided through the module system, you may install and manage your own software on SeaWulf. This guide helps you choose the right approach for your needs and understand where to install software safely.

Choosing the Right Installation Method

SeaWulf provides several ways to manage software. Choose the method that best fits your needs:

1. Use System Modules (Recommended First)

When to use: The software you need is already installed

module avail software-name
module load software-name/version

See the Using Modules documentation for details.

2. Python/Data Science: Conda Environments

When to use: Python packages, data science tools, or packages with complex dependencies

Best for: numpy, pandas, scipy, matplotlib, tensorflow, pytorch, jupyter

module load anaconda/3
conda create --name myproject python=3.10 numpy pandas
conda activate myproject

See the Conda Environments documentation for comprehensive guidance.

3. Python-Only Projects: Virtual Environments (venv)

When to use: Lightweight Python projects without compiled dependencies

module load python/3.11.2
python -m venv ~/myproject-env
source ~/myproject-env/bin/activate
pip install package-name

See the Python Package Management guide for details.

4. Install from Source

When to use: Software not available through modules or package managers, or when you need custom compilation options

Best for: Custom tools, research software, specific versions with optimizations

./configure --prefix=$HOME/software/myapp-1.0
make -j 4
make install

See the Installing from Source documentation for detailed instructions.

5. Request System-Wide Installation

When to use: Multiple users need the software, or installation requires system-level access

Submit a request at iacs.supportsystem.com

Storage Locations: Where to Install

Understanding where to place your software is crucial for stability and performance:

Home Directory: /gpfs/home/<Your_NetID>

  • Quota: 20 GB
  • Backed up: Yes
  • Persistent: Yes, permanent storage
  • Best for: Personal tools, conda environments, small applications
  • Example: $HOME/software/myapp-1.0

Project Space: /gpfs/projects/<Group_Name>

  • Quota: Large (varies by allocation)
  • Backed up: No
  • Persistent: Yes, permanent storage
  • Shared with: All members of your research group
  • Best for: Shared tools, large software installations, collaborative environments
  • Example: /gpfs/projects/<Group_Name>/software/myapp-1.0

Scratch Space: /gpfs/scratch/<Your_NetID>

Warning: DO NOT install software in scratch space. Files are automatically deleted after 30 days based on modification time stamps. Even actively used software will be deleted, breaking your installations and potentially disrupting running jobs.

  • Purge policy: Files deleted after 30 days
  • Best for: Temporary job input/output, intermediate results
  • NOT for: Any software installation, conda environments, or persistent data

Decision Tree: Which Method Should I Use?

Is the software already available as a module?
├─ YES → Use module load software-name
└─ NO → Continue...

Is it a Python package or data science tool?
├─ YES → Is it available through Conda?
│   ├─ YES → Use Conda environment
│   └─ NO → Use pip in a virtual environment
└─ NO → Continue...

Will multiple users need this software?
├─ YES → Request system-wide installation
└─ NO → Continue...

Is source code available and can you compile it?
├─ YES → Install from source
└─ NO → Contact support for assistance

Best Practices for Personal Software

Organization

  • Keep software organized by purpose: $HOME/software/, $HOME/conda-envs/
  • Include version numbers in directory names: myapp-1.0, myapp-2.0
  • Document installations in a README file within each directory

Environment Management

  • Create separate environments for different projects
  • Document environment contents: conda env export > environment.yml
  • Test environments interactively before submitting jobs
  • Never modify the base conda environment

Disk Usage

  • Monitor your quota regularly: myquoata
  • Clean conda package cache periodically: conda clean --all
  • Remove old or unused environments
  • Use project space for large installations

Documentation

  • Keep notes on installation commands and configuration
  • Document required modules and dependencies
  • Save environment files for reproducibility
  • Include version information in your research notes

Sharing Software with Your Group

If multiple group members need the same software:

Install in Project Space

# Conda environment for the group
conda create --prefix /gpfs/projects/GroupName/envs/shared-env python=3.10

# Compiled software
./configure --prefix=/gpfs/projects/GroupName/software/myapp-1.0
make install

Set Appropriate Permissions

# Make readable and executable by group
chmod -R g+rX /gpfs/projects/GroupName/software/myapp-1.0

# Prevent accidental modifications
chmod -R go-w /gpfs/projects/GroupName/software/myapp-1.0

Document for Group Members

Create a README file in the project directory explaining:

  • What software is installed and where
  • How to activate/use it
  • Required modules
  • Who to contact for issues

Troubleshooting Common Issues

Quota exceeded:
Check usage (including inodes), remove unused environments, or move software to project space.

Software works interactively but fails in jobs:
Ensure all module loads and path exports are in your job script. Jobs don't inherit your interactive environment.

Command not found:
Check that the executable directory is in your PATH. Use which command-name to debug.

Library not found errors:
Add library directories to LD_LIBRARY_PATH. Load the same modules at runtime that you used during installation.

Permission denied:
Verify you have write access to your installation directory. Never try to install to system directories like /usr/local.

Conflicts between different software:
Use separate environments (Conda) or modulefiles to isolate incompatible software.

When to Request Help

Contact support at iacs.supportsystem.com if:

  • You need software installed system-wide for multiple users
  • Installation requires system-level permissions or licensed software
  • You're unsure about the best installation method for your needs
  • You need help with complex compilation or dependency issues
  • You want guidance on organizing shared software for your group
  • You need additional quota for legitimate research needs