Skip to main content

Get started with the IonFS CLI and S6

Josh Fraser

Josh Fraser

Engineer @ Ionburst Cloud

Overview#

The IonFS Command Line Interface provides a set of tools to manage data stored by Ionburst Cloud S6 as if it were a remote filesystem. While the IonFS CLI stores files within Ionburst Cloud S6, the metadata is stored in a customer-owned metadata repository.

Anyone that has been granted access to this repository, and the appropriate Ionburst Cloud credentials, can interact with the stored data.

For this tutorial, we will be using Amazon S3 as the ionfs metadata repository.

Shared Responsibility Model Breakdown#

Customer Responsibility#

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by ionfs.
  • You, the customer, are responsible for the security of ionfs metadata repositories and the metadata stored in them.

Ionburst Cloud Responsibility#

  • We are responsible for the security of all data stored in Ionburst Cloud S6 using ionfs.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

Getting Started#

In this tutorial we will cover:

  1. Configuring ionfs for S6.
  2. Working with ionfs metadata repositories.
  3. Listing IBC classifications with ionfs.
  4. Working with ionfs directories.
  5. Managing files with ionfs.

Configuration Overview#

Before we get started, we'll cover a summary of how each configuration file should look once they've been setup. If you would like, you can skip the overview.

To use ionfs with S6, we need to configure three things:

  • The Ionburst credentials file.
  • The metadata repository settings - in this case, the AWS credentials file.
  • The ionfs settings file.
note

If you have already configured ionfs for S6, you can jump ahead.

Ionburst Credentials#

ionfs requires a profile configured in the Ionburst credentials file, a well-known directory in the user's home folder.

  • On MacOS/Linux, this file is located at ~/.ionburst/credentials, and
  • on Windows, it is located at %USERPROFILE%\.ionburst\credentials.

An example of the final credentials file should look similar to this. In this example, the credentials profile we will create is called ionfs-example, and the profile is configured to use the Ionburst Cloud eu-west-1 region.

[ionfs-example]
ionburst_id=IBGCFRFEUJ2TDEXAMPLE
ionburst_key=bG9va211bWknbWFiYXNlNjRlbmNvZGVkc3RyaW5n
ionburst_uri=https://api.eu-west-1.ionburst.cloud

Metadata settings#

As this example is using Amazon S3 for the ionfs metadata, we have to create an AWS credentials file. This file is stored within a well-known directory in the user's home folder.

  • On MacOS/Linux, this file is located at ~/.aws/credentials, and
  • on Windows, it is located at %USERPROFILE%\.aws\credentials.

The credentials file is INI formatted, and allows multiple sets of credentials, or profiles, to be used:

[example_profile]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

IonFS CLI settings#

ionfs uses its own configuration file located within a well-known directory in the user's home folder.

  • On MacOS/Linux, this file is located at ~/.ionfs/appsettings.json, and
  • on Windows, it is located at %USERPROFILE%\.ionfs\appsettings.json.

An example of the ionfs configuration file should look similar to the below. In this example, ionfs is configured to use a metadata repository "s3-example-ionfs", the Ionburst credentials profile "ionfs-example", and the AWS credentials profile "example_profile".

{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information"
}
},
"IonFS": {
"MaxSize": "50000000",
"Verbose": "false",
"DefaultClassification": "Restricted",
"Repositories": [
{
"Name": "s3-example-ionfs",
"Usage": "Data",
"Class": "Ionburst.Apps.IonFS.Repo.S3.MetadataS3",
"Assembly": "Ionburst.Apps.IonFS.Repo.S3",
"DataStore": "s3-example-ionfs"
}
],
"DefaultRepository": "s3-example-ionfs",
},
"Ionburst": {
"Profile": "ionfs-example",
"IonburstUri": "https://api.eu-west-1.ionburst.cloud/",
"TraceCredentialsFile": "OFF"
},
"AWS": {
"Profile": "example_profile",
"Region": "eu-west-1"
}
}

1. Configuring IonFS CLI#

1.1 - Creating the credentials file#

1.1.1 - Open a new terminal.

1.1.2 - Next, we need to create the .ionburst directory:

mkdir ~/.ionburst

1.1.3 - The credentials file can then be created within this directory:

touch ~/.ionburst/credentials

1.1.4 - You can now verify this with the following:

ls ~/.ionburst/credentials

1.1.5 - On verifying, the output should look similar to this:

[hello@ionfs-example ~]$ ls ~/.ionburst/credentials
/home/hello/.ionburst/credentials

1.2 Add a profile to the credentials file#

1.2.1 - To create a new profile in the Ionburst credentials file, the following can be used:

echo "[ionfs-example]" >> ~/.ionburst/credentials
echo "ionburst_id=IBGCFRFEUJ2TDEXAMPLE" >> ~/.ionburst/credentials
echo "ionburst_key=bG9va211bWknbWFiYXNlNjRlbmNvZGVkc3RyaW5n" >> ~/.ionburst/credentials
echo "ionburst_uri=https://api.eu-west-1.ionburst.cloud/" >> ~/.ionburst/credentials

1.2.2 - This will add a profile ionfs-example to the credentials file. Please ensure the >> operator is used to avoid overwriting any existing contents in the credentials file.

The file contents can be verified with the following:

cat ~/.ionburst/credentials

1.2.3 - An example output should look similar to:

[hello@ionfs-example ~]$ echo "[ionfs-example]" >> ~/.ionburst/credentials
[hello@ionfs-example ~]$ echo "ionburst_id=IBGCFRFEUJ2TDEXAMPLE" >> ~/.ionburst/credentials
[hello@ionfs-example ~]$ echo "ionburst_key=bG9va211bWknbWFiYXNlNjRlbmNvZGVkc3RyaW5n" >> ~/.ionburst/credentials
[hello@ionfs-example ~]$ echo "ionburst_uri=https://api.eu-west-1.ionburst.cloud/" >> ~/.ionburst/credentials
[hello@ionfs-example ~]$ cat ~/.ionburst/credentials
echo "[ionfs-example]" >> ~/.ionburst/credentials
echo "ionburst_id=IBGCFRFEUJ2TDEXAMPLE" >> ~/.ionburst/credentials
echo "ionburst_key=bG9va211bWknbWFiYXNlNjRlbmNvZGVkc3RyaW5n" >> ~/.ionburst/credentials
echo "ionburst_uri=https://api.eu-west-1.ionburst.cloud/" >> ~/.ionburst/credentials

1.3 - Creating the AWS credentials file#

1.3.1 - Next, we need to create the .aws directory:

mkdir ~/.aws

1.3.2 - The credentials file can then be created within this directory:

touch ~/.aws/credentials

1.3.3 - You can now verify this with the following:

ls ~/.aws/credentials

1.3.4 - On verifying, the output should look similar to this:

[hello@ionfs-example ~]$ ls ~/.aws/credentials
/home/hello/.aws/credentials

1.4 - Adding a profile to the AWS credentials file#

1.4.1 - To create a new profile in the AWS credentials file, the following can be used:

echo "[example-profile]" >> ~/.aws/credentials
echo "aws_access_key_id=AKIAIOSFODNN7EXAMPLE" >> ~/.aws/credentials
echo "aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" >> ~/.aws/credentials

1.4.2 - This will add a profile example-profile to the credentials file. Please ensure the >> operator is used to avoid overwriting any existing contents in the credentials file.

The file contents can be verified with the following:

cat ~/.aws/credentials

1.4.3 - An example output should look similar to:

[hello@ionfs-example ~]$ echo "[example-profile]" >> ~/.aws/credentials
[hello@ionfs-example ~]$ echo "aws_access_key_id=AKIAIOSFODNN7EXAMPLE" >> ~/.aws/credentials
[hello@ionfs-example ~]$ echo "aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" >> ~/.aws/credentials
[hello@ionfs-example ~]$ cat ~/.ionburst/credentials
[example-profile]
aws_access_key_id=IBGCFRFEUJ2TDEXAMPLE
aws_secret_access_key=bG9va211bWknbWFiYXNlNjRlbmNvZGVkc3RyaW5n

1.5 Creating the ionfs settings file#

1.5.1 - Next, we need to create the .ionfs directory:

mkdir ~/.ionfs

1.5.2 - The settings file can then be created within this directory:

touch ~/.ionfs/appsettings.json

1.5.3 - You can now verify this with the following:

ls ~/.ionfs/appsettings.json

1.6 Configure the ionfs settings file#

1.6.1 - Open the ionfs settings file in the text editor of your choice.

1.6.2 - Paste the example file above into the file.

1.6.3 - Adjust the following values:

  • Within the Repositories section
    • Change the "Name" value to the desired name for your metadata repo (optional)
    • Ensure the "Usage" value is set to Data
    • Ensure the "Class" value is set to Ionburst.Apps.IonFS.Repo.S3.MetadataS3
    • Ensure the "Assembly" value is set to Ionburst.Apps.IonFS.Repo.S3
    • Change the "DataStore" value to the name of the S3 bucket created to store ionfs metadata
  • Ensure the value for "DefaultRepository" matches the value in "Name"
  • Within the Ionburst section
    • Change the "Profile" value to the profile created in your Ionburst credentials file
    • Change the "IonburstUri" value to your desired Ionburst Cloud API endpoint
    • Ensure "TraceCredentialsFile" is set to "OFF"
  • Within the AWS section
    • Change the "Profile" value to the profile created in your AWS credentials file
    • Change the "Region" value to the region your S3 bucket was created in

Basic Usage#

Now that ionfs has been configured, we can now start using it to store and manage data in Ionburst Cloud S6.

ionfs allows us to do the following:

  • List configured metadata repositories.
  • List available IBC classifications.
  • Create, list and delete ionfs directories.
  • Upload, download and delete data from IBC.

2. Metadata Repositories#

ionfs makes use of metadata repositories, or repos, to track data that has been secured by Ionburst Cloud S6. Metadata repos are specified in the configuration file stored under ~/.ionfs/appsettings.json.

To list the configured repos, the following ionfs command can be used:

ionfs repos

An example output would look like:

[hello@ionfs-example ~]$ ionfs repos
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Repositories (*default):
* [d] ion://s3-example-ionfs/ (Ionburst.Apps.IonFS.Repo.S3.MetadataS3)

3. Classifications#

Data can be secured by Ionburst Cloud according to available security policies. ionfs can be used to view the policies currently available to an Ionburst Cloud party.

To list available policies, the following can be used:

ionfs policy

An example output would look like:

[hello@ionfs-example ~]$ ionfs policy
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Classifications:
2:Restricted

4. Directories#

Data secured by Ionburst Cloud S6 through ionfs can partition its repo using a typical directory structure.

List directories#

To list available directories within a repo, the following can be used:

ionfs list

An example output would look like:

[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/
d example/

By default, this will list the contents of the repo's root directory. To list a specific directory, the following can be used:

ionfs list ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/example/
Remote directory is empty

Create a directory#

To create a new directory within a repo, the following can be used:

ionfs mkdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs mkdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/
d example/
d new-directory/

Delete a directory#

To remove a directory within a repo, the following can be used:

ionfs rmdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs rmdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/
d example/

5. Files#

Finally, and most importantly we can now look at uploading (Put), downloading (Get) and deleting data from IBC S6 using ionfs. In these examples, we'll use a file called my-file.txt.

First, we need to create my-file.txt:

echo "We may guard your data, but we'll never take its freedom" > my-file.txt

Put#

To upload a file to Ionburst Cloud with ionfs, the following can be used:

ionfs put my-file.txt ion://

An example output would look like:

[hello@ionfs-example ~]$ ionfs put my-file.txt ion://
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/
d example/
my-file.txt 23/4/2021 13:49:51

To upload data to a specific directory within your repo, use the following:

ionfs put my-file.txt ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs put my-file.txt ion://example
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/example/
example/my-file.txt 23/4/2021 13:50:23

Get#

To retrieve a file with ionfs, use the following:

ionfs get ion://example/my-file.txt

An example output would look like:

[hello@ionfs-example ~]$ rm my-file.txt
[hello@ionfs-example ~]$ ionfs get ion://example/my-file.txt
[hello@ionfs-example ~]$ ls
my-file.txt
[hello@ionfs-example ~]$ cat my-file.txt
We may guard your data, but we'll never take its freedom

By default, this will download the file from Ionburst Cloud S6 to the current directory, with the name used in ionfs. To download to a specific local directory, or to download to a different name, use the following:

ionfs get -n my-file-2.txt ion://example/my-file.txt

An example output would look like:

[hello@ionfs-example ~]$ ionfs get -n my-file-2.txt ion://example/my-file.txt
[hello@ionfs-example ~]$ ls
my-file.txt my-file-2.txt
[hello@ionfs-example ~]$ cat my-file-2.txt
We may guard your data, but we'll never take its freedom

Delete#

To delete a file from the ionfs repo and from Ionburst Cloud S6, the following can be used:

ionfs del ion://example/my-file.txt

An example output would look like:

[hello@ionfs-example ~]$ ionfs del ion://example/my-file.txt
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/example/
Remote directory is empty

Conclusion#

You should now be able to perform basic file operations on Ionburst Cloud S6 using the ionfs tool. If you're interested in learning more about the IonFS CLI, please see the Ionburst Cloud docs.