Rclone setup

Rclone setup and usage

See the rclone installation website for full instructions. However, for a Linux system script-based installation, use the following command.

sudo -v ; curl https://rclone.org/install.sh | sudo bash

Next you will need to configure rclone.

You can view your rclone configurations by using the command rclone config. The first time you do this, you should see something similar to the following output:

No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q>

Select n, and give the remote a name. In this case, massive.

name> massive

It will come up with the following output:

Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
{many numbered options...}

Select the option for SSH/SFTP, which as of writing this document, is number 29. You will see the following output:

Option host.
SSH host to connect to.
E.g. "example.com".
Enter a value.

Input the address for M3 massive.

host> m3.massive.org.au

It will then then ask you to enter your SSH username. Enter this.

[M3 username]

It will then ask you to enter an SSH port number. Press Enter to use the default port.

You then need to enter your M3 massive password.

Leave the next few steps at their default value, and press Enter through them. You do not need to set these either.

It will present you with a summary of the configuration settings. Input y to confirm this looks correct.

This completes the setup of rclone. Input q to exit the configuration menu.

Transferring files

Transferring files in a folder from your Google Drive is simple:

# Navigate to the folder you want you fastq files to be downloaded to
cd path/to/folder

# Transfer shotgun files (example path)
rclone sync -v massive:mf33_scratch/Matt/ShotgunSequencing/anvio/ .

At this stage, the wildcard * in filenames is not allowed directly within the path, but typically you will want everything in the shotgun data folder anyway.

If however you do want to select individual files, or a set of files by wildcard *, you can use the --include flag. For example, if I wanted all files beginning with 137 (samples for patient 137), I could use the command:

# Transfer just files beginning with "137" from the example folder above
rclone sync -v --include "137*" \
massive:mf33_scratch/Matt/ShotgunSequencing/raw_fastq/ .

If you wanted to transfer files back to M3 massive from your local computer, you would just invert the directory order and place googledrive:/ second.