Ctrl+K
Psst, remember the cheatsheet!

Allas exercise

Allas exercise#

Timing

15 min

Goals

Learn how to:

  • Configure connection to Allas and get S3 credentials

  • Sync local folder to Allas (manual back-up)

  • See what data is Allas

  • Use s3cmd.

Prerequisites

Change the default project and username

  • project_200xxxx is example project name, replace with your own CSC project name.

  • cscusername is example username, replace with your username.

  • Open Puhti web interface and log in

  • Open Login node shell from Tools menu.

  • Set up Allas connection

module load allas
allas-conf --mode s3cmd
# It asks to select the project, select the project by number. 
# The configuration takes a moment, please wait.

Get your S3 credentials

allas-conf --mode s3cmd output includes your S3 credentials: access key and secret key, these are needed for creating the connection with other tools.

# Create a new bucket
# s3cmd mb <name_of_your_bucket>

s3cmd mb s3://project_200xxxx-cscusername

# Upload (later syncronize) a folder to Allas
# s3cmd sync <local_folder> s3://<name_of_your_bucket>
s3cmd sync /appl/data/geo/mml/dem10m/2019/W3/W33/ s3://project_200xxxx-cscusername

# List all buckets
s3cmd ls

# List all files in one bucket
# s3cmd ls s3://<name_of_your_bucket>
s3cmd ls s3://project_200xxxx-cscusername

# Read and write directly to Allas with GDAL
# Make GDAL avaialble
module load geoconda

# See metadata of a file from GDAL exercise
gdalinfo /vsis3/project_200xxxx-cscusername/W3333.tif

# Enable writing with GDAL
export CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE=YES

# Make the .tif file to Cloud-Optimized GeoTiff
gdal_translate /vsis3/project_200xxxx-cscusername/W3333.tif /vsis3/project_200xxxx-cscusername/W3333_COG.tif -of COG

# See metadata of the new file
gdalinfo /vsis3/project_200xxxx-cscusername/W3333_COG.tif

# Delete all from Allas
s3cmd del --recursive --force s3://project_200xxxx-cscusername
s3cmd rm s3://project_200xxxx-cscusername

Key points

  • Take a moment to plan your bucket and object naming, so that it would be easy to use later.

  • Allas is a good place to keep a back-up of your data.

  • Many GIS tools can read (and write) directly to Allas.