Migrate Data From Amazon S3 To AIOZ W3S

Prerequisites

Before you begin the migration process, ensure that you have the following:

1. rclone: Make sure you have rclone installed on your system. If not, you can download it from the official website: rclone.org (opens in a new tab)
2. AIOZ W3S Access Grants: You will need the access grants for your AIOZ W3S account. This typically includes an access key and a secret key. Refer to this page manage-access-grants.md for how to get access grants.

Amazon S3

Prerequisites

1. Amazon S3 Credentials: Ensure you have your Amazon S3 account credentials handy. Refer to this page (opens in a new tab) for how to get the credentials.

Step-by-Step guide

Follow these steps to migrate your data from Amazon S3 to AIOZ W3S:

Step 1: Locate rclone config path

1. Open a terminal window.

2. Run the following command:

   rclone config file

3. It will output some text like:

   Configuration file is stored at: /home/user/.config/rclone/rclone.conf

Step 2: Configure Amazon S3 and AIOZ W3S remotes in rclone

1. Open the rclone config file from the above path.

2. Write config for Amazon S3:

   [s3]
   type = s3
   provider = AWS
   access_key_id = <your-s3-access-key-id>
   secret_access_key = <your-s3-secret-access-key-id>
   region = <your-s3-region>
   endpoint = <your-s3-endpoint>
   acl = private
   server_side_encryption =
   storage_class =

3. Write config for AIOZ W3S:

   [aioz-w3s]
   type = s3
   provider = Other
   access_key_id = <your-w3s-access-key-id>
   secret_access_key = <your-w3s-secret-access-key>
   endpoint = <your-w3s-endpoint>

4. Save the config file.

Step 3: Transfer data

1. Use the following command to sync data from your Amazon S3 bucket to your AIOZ W3S bucket:

   rclone sync --progress s3:your-bucket aioz-w3s:your-bucket

   If you counter corrupted on transfer: sizes differ error, you can add a flag --ignore-size.

2. If you want to ensure that the data on AIOZ W3S is identical to Amazon S3, you can perform a checksum check using the --check flag:

   rclone check --progress s3:your-bucket aioz-w3s:your-bucket

Step 4: Verify and finalize

1. After the transfer is complete, verify that the data has been migrated correctly by comparing files and their content between the two storage locations.
2. Update any applications, scripts, or processes that reference the data to point to the new AIOZ W3S location.
3. Once you're confident that the migration was successful, you can remove the data from Amazon S3 if it's no longer needed.

Congratulations! You have successfully migrated your data from Amazon S3 to AIOZ W3S using rclone.