Using Allas in CSC’s HPC environment

Before the actual exercise, open a view to the Allas service in your browser using the Roihu web interface.

Go to https://www.roihu.csc.fi and login with your account.
Configure an Allas S3 connection using the Cloud storage configuration tool.
- You need to first authenticate by providing your CSC password.
- If you have several projects available, choose one that you want to use in this exercise.
Once you’ve configured a connection, select s3allas-project_<id> from the Files dropdown menu in the top navigation bar. Replace <id> with the number of the project you chose to use (e.g. 2001234).
During the exercise, you can use this web interface to get another view to the buckets and objects in Allas.

ssh <username>@roihu-cpu.csc.fi    # replace <username> with your CSC username

In Roihu, check your environment with the command:
```
csc-workspaces
```

Move to the /scratch directory of your project

cd /scratch/<project>  # replace <project> with your CSC project, e.g. project_2001234

Create your own subdirectory named with your username:
```
mkdir -p $USER
```
Move to the directory:
```
cd $USER
```

2. Download data with `wget`

Next, download a dataset and uncompress it
- The dataset contains some pythium genomes with related BWA indexes
```
wget https://a3s.fi/course_12.11.2019/pythium.tgz
tar -xzvf pythium.tgz  
tree pythium
```

3. Using Allas

Open a connection to Allas:
```
module load allas
allas-conf
```
If you have several Allas projects available, select the same project as earlier

Upload case 1: `rclone`

Upload the data from Roihu to Allas with rclone:
```
rclone -P copyto pythium s3allas:$USER-genomes-rc/
```
- How long did the data upload take?
- What was the transfer rate?
- How long would it take to transfer 100 GiB assuming the same speed?

Study what you have uploaded to Allas with the commands:

rclone lsd s3allas:
rclone ls s3allas:$USER-genomes-rc/
rclone lsl s3allas:$USER-genomes-rc/
rclone lsf s3allas:$USER-genomes-rc/

Check how this looks like in the Roihu web interface. Open a browser and go to https://www.roihu.csc.fi/
In the Roihu web interface, go to the Files app and select s3allas-project_<id> to list the buckets of your project (replace <id> as needed).
Locate your own $USER-genomes-rc bucket and download one of the uploaded files to your local computer

💡 You can read more about moving files at Docs CSC: Copying files using scp and Moving data with rclone

Upload case 2: `a-put`

Upload the pythium directory from Roihu to Allas using a-commands

Case 1: Store everything as a single object (replace <project number> with your CSC project number, e.g. 2001234):

a-put pythium      
a-list
a-list <project number>-roihu-scratch
a-info <project number>-roihu-scratch/$USER/pythium.tar

Case 2: Each subdirectory (species) as a separate object (replace <project number> with your CSC project number, e.g. 2001234):

a-put pythium/*
a-list <project number>-roihu-scratch 
a-info <project number>-roihu-scratch/$USER/pythium/Pythium_vexans.tar

Case 3: Use a custom bucket name (replace <project number> with your project number, e.g. 2001234):

a-put pythium/* -b <project number>-$USER-genomes-ap
a-list <project number>-$USER-genomes-ap

Can you see the difference between the three a-put commands above?

Study the <project number>-$USER-genomes-ap bucket with commands:

a-list <project number>-$USER-genomes-ap
rclone ls s3allas:<project number>-$USER-genomes-ap 

Why do the two commands above list a different amount of objects?
Try the command (replace <project number> with your project number, e.g. 2001234):
```
a-info <project number>-$USER-genomes-ap/Pythium_vexans.tar
```

This command is actually the same as:

rclone cat s3allas:<project number>-$USER-genomes-ap/Pythium_vexans.tar_ameta

Finally, try the command:

a-flip pythium/Pythium_vexans/Pythium_vexans.amb

Try opening the public link that a-flip produced with your browser

4. Exit

The data in the pythium directory is now stored in many ways in Allas, so we can remove the data from Roihu and log out:
```
rm -r pythium
exit
```

5. Downloading data from Allas to Roihu

ssh <username>@roihu-cpu.csc.fi   # replace <username> with your CSC username
cd /scratch/<project>/$USER   # replace `<project>` with your CSC project, e.g. project_2001234

In Roihu, check you projects with the command:
```
csc-workspaces
```
Set up the Allas connection:
```
module load allas
allas-conf 
```

Then run the commands (we will use the same bucket that was created earlier):

a-list
rclone lsd s3allas:
# replace <project number> with your project number, e.g. 2001234
a-list <project number>-$USER-genomes-ap
rclone ls s3allas:<project number>-$USER-genomes-ap
a-find Pythium_vexans.amb
a-find -a Pythium_vexans.amb

Next, download the data in different ways:

1. Download with `rclone`

Copy everything:

mkdir rclone_dir
cd rclone_dir/
mkdir all
rclone ls s3allas:<project number>-$USER-genomes-ap
rclone copyto -P s3allas:<project number>-$USER-genomes-ap all/
ls all

Copy a set of objects:

mkdir vexans 
rclone copyto s3allas:$USER-genomes-rc/Pythium_vexans vexans/
ls vexans

Copy just one object:

rclone copyto s3allas:$USER-genomes-rc/Pythium_vexans/Pythium_vexans.amb ./vexans.amb
ls

2. Download with `a-get`

Return to your $USER directory under your project’s /scratch on Roihu (The pwd command should print /scratch/<project/$USER):
```
cd ..
pwd
```
Make a new directory:
```
mkdir a_dir
cd a_dir/
```
Create a directory all and move there:
```
mkdir all
cd all
```
List your default scratch bucket (replace <project number> with your project number, e.g. 2001234):
```
a-list <project number>-roihu-scratch
a-list <project number>-roihu-scratch/$USER
```

Look for the file Pythium_vexans.amb in your Roihu scratch bucket:

a-find Pythium_vexans.amb -b <project number>-roihu-scratch    # replace <project number> with your project number, e.g. 2001234

Download the full dataset with command:

a-get <project number>-roihu-scratch/$USER/pythium.tar   # replace <project number> with your project number, e.g. 2001234

Check what you got:
```
ls -l
ls -R
```

Now, download just a single genome dataset:

cd ..
a-get <project number>-roihu-scratch/$USER/pythium/Pythium_vexans.tar   # replace <project number> with your project number, e.g. 2001234
ls -l pythium/
ls -l pythium/Pythium_vexans/

Using Allas in CSC’s HPC environment

1. Login to Roihu

2. Download data with wget

3. Using Allas

Upload case 1: rclone

Upload case 2: a-put