rclone¶
rclone is a popular tool for copying, synchronizing, and managing data in Object Storage services compatible with the Amazon S3 API.
You can use it with Cloud4You Object Storage S3 to:
- list buckets,
- upload files,
- download files,
- copy directories,
- synchronize data,
- automate tasks,
- mount a bucket as a directory using FUSE.
Cloud4You Object Storage S3 parameters¶
Use the following values when configuring rclone:
You also need:
Installation¶
Windows¶
After installation, check:
macOS¶
If you use Homebrew:
Check:
Linux¶
After installation, check:
Configuration¶
Run:
Select the option to create a new configuration:
Enter a name, for example:
For the storage type, select:
Then configure the connection.
Provider¶
If rclone displays a list of S3 providers, select:
If your version does not contain an appropriate entry or the configuration requires a generic provider, you can use:
Access Key and Secret Key¶
Enter the credentials received from Cloud4You.
Example:
Do not share the Secret Key
Treat the Secret Key like a password.
Do not publish the rclone configuration file or fragments of it that contain access credentials.
Region¶
Enter:
Endpoint¶
Enter:
ACL¶
If rclone asks about ACL and you do not have special requirements, leave the default setting.
Do not enable public access to data unless there is a deliberate need for it.
Example configuration¶
A configuration fragment may look like this:
[cloud4you]
type = s3
provider = Ceph
access_key_id = <Access Key>
secret_access_key = <Secret Key>
endpoint = https://s3.cloud4you.pl
region = eu-pl-1
Do not publish the real rclone.conf file
The configuration file may contain the Secret Key.
Configuration file location¶
To check where the configuration file is stored:
rclone will display the path to the currently used rclone.conf file.
Testing the connection¶
After saving the configuration, list the available buckets:
Example output:
If the list is displayed without an error, the configuration works correctly.
Listing bucket contents¶
For a more readable view of directories and objects:
Uploading a single file¶
Downloading a single file¶
Copying a directory to S3¶
Example:
The copy command copies missing and changed data but does not delete files that exist only at the destination.
Downloading a directory from S3¶
Synchronizing a directory to S3¶
sync can delete data
rclone sync makes the destination match the source.
Objects that exist at the destination but not at the source may be deleted.
If you only want to copy new or changed files without deleting data, use rclone copy.
Testing synchronization¶
Before running sync for the first time, we recommend using:
--dry-run shows planned operations without executing them.
Use --dry-run first
This is particularly important for large datasets and automated jobs.
Synchronizing from S3 to a computer¶
Remember that sync can also delete data at the destination in this direction.
Creating a bucket¶
rclone can create a bucket:
Afterwards, check:
Deleting an empty bucket¶
The bucket must be empty.
Deleting a single object¶
Destructive operation
Before deleting anything, carefully check the bucket and object name.
Deleting multiple objects¶
Example:
This command may delete many objects.
Before running it, you can use:
Checking storage usage¶
You can calculate the number of objects and their total size:
Example output:
The value may differ from the data reported by the billing system or quota due to factors such as versioning and the way storage usage is calculated.
Checking differences¶
rclone can compare the source and destination:
This is useful after larger migrations or copy operations.
More information during transfers¶
To display progress:
You can also use the shorter form:
Logging¶
Example of writing a log file:
rclone copy \
./dane \
cloud4you:firma-backup/dane \
--log-file /var/log/rclone-cloud4you.log \
--log-level INFO
Check the log before sharing it
Logs may contain bucket names, object names, and other technical information.
Before sending a log to a third party, review its contents.
Mounting a bucket as a directory¶
On systems that support FUSE, you can use rclone mount.
Example:
mkdir -p ~/Cloud4You-S3
rclone mount \
cloud4you:firma-backup \
~/Cloud4You-S3 \
--vfs-cache-mode full
After running the command, the bucket contents will be visible under:
S3 is not a POSIX filesystem
rclone mount creates a layer that emulates a traditional filesystem.
The behavior of such a mount differs from a local disk, NFS, or SMB.
We do not recommend using rclone mount as direct storage for:
- active databases,
- virtual machine disks,
- applications that require file locking,
- workloads that require full POSIX semantics.
Automation¶
rclone can easily be used in scripts and scheduled tasks.
Example:
#!/bin/bash
rclone copy \
/srv/archive \
cloud4you:firma-backup/archive \
--log-file /var/log/rclone-cloud4you.log \
--log-level INFO
When automating rclone, it is especially important to:
- protect
rclone.conf, - monitor exit codes,
- monitor logs,
- test commands with
--dry-run, - use
synccarefully.
Permissions for rclone.conf¶
On Linux systems, it is worth restricting access to the configuration file.
First check its location:
Then set appropriate permissions, for example:
Adjust the path to the location shown by rclone config file.
Most common problems¶
AccessDenied¶
Check:
- Access Key,
- Secret Key,
- bucket name,
- user permissions.
SignatureDoesNotMatch¶
Check:
and:
Also verify the Secret Key and system time.
The bucket is not visible¶
First run:
If the bucket does not appear in the list, check the credentials and permissions.
Connection problem¶
Run the command with more detailed output:
-vv produces detailed logs
Before publishing or sending the log, check that it does not contain information you do not want to share.
Most commonly used commands¶
List buckets:
List objects:
Upload:
Download:
Copy a directory:
Synchronization:
Check size: