Allas exercise

Allas exercise#

Timing

15 min

Goals

Learn how to:

  • Configure connection to Allas and get S3 credentials

  • Sync local folder to Allas (manual back-up)

  • See what data is in Allas

  • Use s3cmd.

Prerequisites

Change the default project

  • project_2020458 is an example project name, replace with your own CSC project name.

  • Open Roihu web interface and log in

  • Open Login node shell (Roihu-CPU) from Tools menu.

  • Set up Allas connection

module load allas
allas-conf
# It asks to select the project, select the project with the corresponding number. 
# The configuration takes a moment, please wait.

Get your S3 credentials

allas-conf output includes your S3 credentials: access key and secret key, these are needed for creating the connection with other tools.

# Create a new bucket
# s3cmd mb <name_of_your_bucket>

s3cmd mb s3://project_2020458-$USER

# Upload (later syncronize) a folder to Allas
# s3cmd sync <local_folder> s3://<name_of_your_bucket>
s3cmd sync /dataset/project_2019680/mml/dem10m/2019/W3/W33/ s3://project_2020458-$USER

# List all buckets
s3cmd ls

# List all files in one bucket
# s3cmd ls s3://<name_of_your_bucket>
s3cmd ls s3://project_2020458-$USER

# Read and write directly to Allas with GDAL
# Make GDAL avaialble
module load python-geo

# See metadata of a file from GDAL exercise
gdalinfo /vsis3/project_2020458-$USER/W3333.tif

# Enable writing with GDAL
export CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE=YES

# Make the .tif file to Cloud-Optimized GeoTiff
gdal_translate /vsis3/project_2020458-$USER/W3333.tif /vsis3/project_2020458-$USER/W3333_COG.tif -of COG

# See metadata of the new file
gdalinfo /vsis3/project_2020458-$USER/W3333_COG.tif

# Delete all from Allas
s3cmd del --recursive --force s3://project_2020458-$USER
s3cmd rb s3://project_2020458-$USER

Key points

  • Take a moment to plan your bucket and object naming, so that it would be easy to use later.

  • Allas is a good place to keep a back-up of your data.

  • Many GIS tools can read (and write) directly to Allas.