Rclone - Sync File and Directories to Cloud Storage in Linux

Hi all, today we'll learn what is Rclone and how to install it in our Linux Based Operating System.

Rclone is a command line program to sync files and directories to and from Google Drive, Amazon S3, Openstack Swift / Rackspace cloud files / Memset Memstore, Dropbox, Google Cloud Storage, The local filesystem. It is a Go program and comes as a single binary file. Rclone is a Free and Open Source Software under the terms of MIT license which is hosted in its Github Repository.

Some Salient Features of Rclone are:

• MD5SUMs checked at all times for file integrity
• Timestamps preserved on files
• Partial syncs supported on a whole file basis
• Copy mode to just copy new/changed files
• Sync mode to make a directory identical
• Check mode to check all MD5SUMs
• Can sync to and from network, eg. two different Drive accounts

Installation

Rclone is a Go program and comes as a single binary file. We can Download the relevant binary from their official site.

Use RaiDrive to Mount Cloud Drives ...

To view this video please enable JavaScript, and consider upgrading to a web browser that supports HTML5 video

Use RaiDrive to Mount Cloud Drives to Local System as Network Drives

Or install it via Go program. If you don't have Go Program, you can install it as shown below:

1. Installing Go

On a Ubuntu system:

$ sudo apt-get install golang

On Fedora, CentOS, RHEL system:

a) Download and setup Fedora EPEL Repository

# yum install http://ftp.riken.jp/Linux/fedora/epel/6/i386/epel-release-6-8.noarch.rpm

b) Installing golang using yum manager

# yum install golang

2. Installing Rclone with Go

We can install rclone with Go program easily with just a single command firing which will build the binary in $GOPATH/bin .

$ go get github.com/ncw/rclone

Note: If you get any error like package github.com/ncw/rclone: cannot download, $GOPATH not set. For more details see: go help gopath then, it is clear that we haven't defined $GOPATH.

It is the directory where our Go program downloads and builds packages. To fix this we should execute the following command in a shell or terminal and then retry above.

$ export GOPATH="$HOME/gopath/"

Here, the above command will point $GOPATH to /home/username/gopath/ directory.

Configuring

Now, as we have successfully installed our rclone under our /home/username/gopath/ directory. We'll now want to configure our rclone. So, to do that, we'll need to enter into the $GOPATH/bin as the binary rclone is inside that folder.

$ cd $GOPATH/bin

Now, as we are inside the bin directory where an executable file named rclone is situated. We'll want to make rclone binary easily accessible as global binary command. To do so, we'll copy file rclone to /usr/bin/ as root mode.

$ sudo cp rclone /usr/bin/

Now, we'll want to configure it as.

$ rclone config

Now, we'll get option to create a new remote. As we have no remote currently, we'll surely wanna add it.
To add a new remote, we'll need to enter n, and give some name to it, we've named it as gdrive. After that, we'll be listed a set of source that we want to use with our new remote. As we're going for testing Google Drive, we'll wanna choose 6) drive.

Then, we'll be asked some questions on using the client we can just press enter and use the rclone's app by default. Then, we'll be given a link which we should accept inorder to connect rclone with Google Drive. We should browse that link in a web browser, then we'll be given a code which we'll need to paste into our rclone's Command Line Interface.

Now, we'll wanna enter Y to confirm the configuration and we wanna quit rclone config now. Now, we have successfully connected Rclone with Google Drive.

Using Rclone

Now, we'll learn how to use Rclone to transfer/sync files and folders with Google Drive.  Here are some usages that we can perform with Rclone tool.

Copy

To copy a file or folder, we can execute rclone copy source:path dest:path command as follows:

$ rclone copy /home/arun/Documents/linoxide/ gdrive:linoxide

Here, we'll copy folder named linoxide which is inside /home/arun/Documents/ to the remote server's folder named linoxide .

Similarly to copy a single file we can point the source:path to file to copy a file to the destination.

Note: It doesn’t delete files from the destination.

Sync

We can easily synchronize the source to the destination using the command rclone sync source:path dest:path as:

$ rclone sync /home/arun/Documents/linoxide/ gdrive:linoxide

Here, the directory /home/arun/Documents/linoxide/ is synchronized to a remote directory called linoxide.

Note: It deletes any files that exist in source that don’t exist in destination. Since this can cause data loss, test first with the -dry-run flag.

Listing

We can view the list of files, folders of our remote server which we are connected. Here, as we have added and connected with our Google Drive as remote name gdrive. We'll enter the following commands to view them.

$ rclone ls gdrive:

Lists all the objects in the the path with sizes.

$ rclone lsl gdrive:

Lists all the objects in the path with sizes and timestamps.

$ rclone lsd gdrive:

Lists all directories/objects/buckets in the the path.

Creating and Removing Paths

Creating a new directory to the remote server gets easy with rclone mkdir . We can simply enter the following command in order to create a directory inside our remote. Here, we have created a folder named arun inside linoxide which doesn’t already exist before.

$ rclone mkdir gdrive:/linoxide/arun

We can remove an empty directory with rmdir command. Here, we are gonna delete the newly created folder arun which is under linoxide parent directory.

$ rclone rmdir gdrive:/linoxide/arun

Now, if we wanna delete a non-empty directory and its contents then, we'll use purge for that. Here, we've a directory data which contains many files and folders inside.

$ rclone purge gdrive:/linoxide/data

Verifying the paths

Now, to check and match the files in the source and destination we'll use check command. It compares sizes and MD5SUMs and prints a report of files which don’t match. It doesn’t alter the source or destination.

$ rclone check /home/arun/linoxide/linux.tar  gdrive:/linoxide/linux.tar

If we wanna produce an md5sum file for all the objects in the path we can run md5sum command with rclone pointing to the remote directories. This is in the same format as the standard md5sum tool produces.

$ rclone md5sum gdrive:/linoxide/

Conclusion

Hurray we have successfully installed Rclone and learned how to use it. So as we know, this article tutors us about what is Rclone, its features, procedure to install and use it. Rclone is a command line program to sync files and directories to and from Google Drive, Amazon S3, Openstack Swift / Rackspace cloud files / Memset Memstore, Dropbox, Google Cloud Storage, The local filesystem. If you have any questions, comments, feedback please do write on the comment box below and let us know what stuffs needs to be added or improved. Thank You! Enjoy Rclone :-)