Using Globus on Seawulf
Globus is a non-profit service for secure, reliable research data management.
Initial Setup
Globus allows you to transfer files using a system of endpoints. In order to copy files from one machine to another, each machine must be set up as a Globus endpoint.
Since Globus transfers are an infrequent occurence on the Seawulf cluster, we chose not to make Seawulf a permanent Globus endpoint. Instead, we ask our users to create a temporary endpoint whenever they need to transfer files.
Globus command line tools can be loaded into your environment by loading the respective anaconda environment like so (note that this currently only works on nodes accessed through milan1/milan2):
module load globus/3.15.0
First, you'll need to login with your globus account and set up your endpoint. This only needs to be done once.
globus login
Open the provided URL in your browser. Select Stony Brook University as your organization and log in with your NetID and password. Provide the requested access, and copy/paste the authorization code into your terminal window. Then run this command to create the endpoint:
globus endpoint create --default-directory /gpfs/scratch/$USER/ --personal seawulf
This will output an Endpoint ID and Setup Key. Take note of the Endpoint ID because you'll need it later. Run the following command to finish setting up your endpoint (replace SETUP_KEY with your setup key):
globusconnectpersonal -setup SETUP_KEY
Using Your Endpoint
You are now ready to start and stop your endpoint whenever you need to transfer files. Use this command to start your endpoint:
globusconnectpersonal -start -restrict-paths rw/gpfs/scratch/$USER,rw/gpfs/home/$USER &
Note the ampersand at the end of the command will make your endpoint run in the background. You can confirm it is working by attempting to list the contents of your scratch directory. Replace $ep1 with your Endpoint ID you received from the "globus endpoint create" command:
globus ls $ep1
To initiate a transfer between two endpoints, use the following command (replacing $ep1 and $ep2 with your Endpoint ID's):
globus transfer $ep1:/gpfs/scratch/$USER/myfolder $ep2:~/otherfolder --recursive --label "CLI single folder"
You can also check the status of your endpoint with this command:
globusconnectpersonal -status
And when you're done transferring files, don't forget to shut down your endpoint with this command:
globusconnectpersonal -stop