Metadata

Learn how to setup IonFS CLI metadata repositories.

S3#

IonFS can use Amazon S3, and other compatible S3 storage services as a metadata repository. To setup an S3 IonFS repository, you'll need:

An S3 bucket
AWS, or other API credentials to access the bucket.

Setting up the S3 bucket#

AWS Console#

To create an S3 bucket for use as an IonFS metadata repository, navigate to the S3 console on AWS and select Create Bucket.

IonFS Create Bucket

Choose a name for your S3 bucket. Bear in mind, S3 bucket names have to be globally unique.

IonFS Bucket Name

Select the region you would like the S3 bucket to be created in.

IonFS Bucket Region

Next, configure S3 to block all public access to the bucket. IonFS never requires public access for an S3 bucket metadata repository.

IonFS Bucket No-Public

Ensure the following settings are set:

Disable Bucket Versioning

IonFS Bucket Disable Versioning

Next, enable Default Encryption.

Enabling default encryption provides an extra layer of protection for IonFS metadata stored in S3. The Amazon S3 key is more than sufficient for this use case, but an AWS KMS key can also be used.

Please note, using an AWS KMS key will incur additional costs.

IonFS Bucket Encryption

Ensure Object Lock is disabled.

IonFS Bucket Object Lock

Finally, you can now select Create bucket.

Please now verify that the following settings are visible on your bucket from the S3 console:

The bucket is set in the desired region, and
Access to the buckets and objects are marked as not public.

IonFS Bucket Verify

Terraform#

The following Terraform config can be used to create an S3 bucket on AWS:

resource "aws_s3_bucket" "s3-example-ionfs" {
  bucket = "s3-example-ionfs"
  acl    = "private"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

Setting up the IAM user#

AWS Console#

To access your configured S3 bucket, you will need an AWS IAM account and corresponding access policy.

IAM Policy#

First, we need to create an IAM policy. Navigate to the IAM console on AWS and click the Policies heading under Access Management. Click Create Policy

IonFS IAM Policies

Select the JSON tab on the Create Policy screen.

IonFS IAM JSON

Delete the default content and paste the following into the JSON tab.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3Ionfs",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::change-me/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucket",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::change-me"
            ]
        }
    ]
}

You can either replace change-me with your bucket name, or open the S3 console, select your desired bucket and click Copy ARN. Paste this value into each of the Resource blocks.

Ensure that the ARN specified in the first Resource block also has the trailing /*, so the policy applies to objects stored within the S3 bucket.

IonFS IAM ARN

Your policy should now look like the following.

IonFS IAM Policy

You are now ready to click Review policy.

IonFS IAM Policy Review

You can now provide a name for your policy (required) and description (optional).

IonFS IAM Policy Name

Once ready, click Create policy.

IonFS IAM Policy Create

Your policy will now be visible in the IAM Policies list.

IonFS IAM Policy List

Terraform#

The following Terraform config can also be used to setup a suitably secured IAM policy to attach to the IAM user:

resource "aws_iam_policy" "s3-ionfs" {
  name        = "s3-ionfs"
  path        = "/"
  description = "s3 policy for ionfs"

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3Ionfs",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "${aws_s3_bucket.s3-example-ionfs.arn}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucket",
                "s3:ListBucket"
            ],
            "Resource": [
                "${aws_s3_bucket.s3-example-ionfs.arn}"
            ]
        }
    ]
}
EOF
}

IAM User#

You can now create an IAM user that can use the policy. Click the Users heading under Access Management. Click Add User.

IonFS IAM User

Pick a name for your IAM user, and select Programmatic access only.

IonFS IAM User Name

Click Next: Permissions to proceed to the policy selection.

IonFS IAM User Name

To attached the created IonFS policy to your user, click the Attach existing policies directly tab, and select the IonFS policy. The list of policies can be filtered by name to make it easy to find.

IonFS IAM User Policy

You can continue to the next stage by clicking Next: Tags.

IonFS IAM User Next

You can now verify the user before creation, and once ready click Create user.

IonFS IAM User Create

Your user will now be created and ready to use. You can now either click Download .csv to obtain a copy of your user's IAM credentials, or copy them directly from the table.

Please note, your IAM user's Secret access key is only available on this page. If you fail to download the csv file or take a copy of the credentials, you will need to generate a new set.

IonFS IAM User Credentials

You are now ready to configure IonFS with your S3 bucket!

MongoDB#

Coming soon!