IonFS CLI: A Client and Metadata Store

Jamie Hunter

Jamie Hunter

Engineer @ Ionburst Cloud

Background

Ionburst Cloud offers a revolutionary way to store data securely and privately in the Cloud, beyond the reach of hackers and unwanted surveillance. Data is transformed and persisted as redundant fragments across collections of storage nodes called Cloudlets™.

Resources needed:

Overview

Ionburst Cloud does not allow the data it holds to be browsed. Its privacy by design default means the ability to request a list of objects stored does not exist. If this functionality is required, it is down to the client applications to track the metadata in an appropriate manner to meet its requirements. IonFS has been developed to illustrate how a client application typically integrates with Ionburst Cloud.

IonFS provides a set of tools to manage data stored by Ionburst Cloud as if it were a remote filesystem. Whilst IonFS stores primary data items within Ionburst Cloud, the metadata is stored in an S3 bucket; anyone with access to this bucket, and the appropriate Ionburst Cloud credentials, can interact with the data stored within.

S3 was selected due to its availability and prevalence in the market, although any platform capable of storing data can be used. S3 is an example of what is called a metadata repository, or “repo” for short.

IonFS has one simple focus – to enable files to be stored by Ionburst Cloud, and to retain the ability to interact as if they had been stored on a filesystem. Traditional filesystem metaphors allow file and folders to be created, deleted, uploaded, download, moved, copied, and renamed.

Furthermore, even though Ionburst Cloud stores the data in a completely secure and fully redundant manner, it is possible to add further encryption and decryption for data from within IonFS.

IonFS Diagram

[Only objects up to size cap currently 50MB – on Ionburst Cloud products can be uploaded, so additional manipulation and tracking is required. IonFS implements this chunking functionality, with a configurable threshold.]

ionfs --help
IonFS:
Securing your data on Ionburst Cloud.
Usage:
IonFS [options] [command]
Options:
-v, --version
-?, -h, --help Show help and usage information
Commands:
list show the contents of a remote path, prefix remote paths with ion://
get download a file, prefix remote paths with ion://
put upload a file, prefix remote paths with ion://
del delete an object, prefix remote paths with ion://
move move a file to/from a remote file system, prefix remote paths with ion://
copy copy a file to/from a remote file system, prefix remote paths with ion://
mkdir create a folder, prefix remote paths with ion://
rmdir remove a folder, prefix remote paths with ion://
policy list the current Ionburst Cloud Classification Policies
repos list the currently registered Repositories

Dependencies

IonFS is built using .NET Core 3.1, allowing cross-platform builds for Windows, MacOS and Linux. Using the tools available through the AWS S3 .NET SDK, it is possible programmatically to build a file and directory structure with metadata wrapped around the fundamental building blocks of Ionburst Cloud; PUT, GET, and DEL. The Ionburst .NET SDK is used exclusively to interact with the Ionburst Cloud API.

Configuration

The primary configuration for IonFS is managed within appsettings.json, located in the .ionfs folder located in your home directory.

The IonFS section contains the main configuration items:

"IonFS": {

MaxSize controls the chunking of data items being uploaded, for extra details to be logged to screen Verbose can be set to true. Note, some commands allow this to be overridden on the command line using -v or –version. DefaultClassification is the default Ionburst Cloud policy applied to data being uploaded, this can be explicitly set on the PUT command.

"MaxSize": "65536",
"Verbose": "false",
"DefaultClassification": "Restricted",

The Repositories section allows multiple metadata repositories to be accessed.

"Repositories": [
{
"Name": "first-S3",
"Class": "Ionburst.Apps.IonFS.MetadataS3",
"DataStore": "ionfs-metadata-first"
},
{
"Name": "second-S3",
"Class": "Ionburst.Apps.IonFS.MetadataS3",
"DataStore": "ionfs-metadata-second"
}
],

If a repository is not explicitly included in the remote path, the DefaultRepository will be used.

"DefaultRepository": "first-S3",
},

The Ionburst Cloud section is required to access the Ionburst SDK.

"Ionburst": {
"Profile": "example_profile",
"IonburstUri": "https://api.example.ionburst.io/",
"TraceCredentialsFile": "OFF"
},

The AWS section is required to access the AWS SDK.

"AWS": {
"Profile": "example",
"Region": "eu-west-1"
}

Security

Credential files are required to access both Ionburst Cloud and AWS. These files can be found in your home directory within .ionburst and .aws folders respectively. Explicit access must be granted, and suitable access credentials provided for any resources being used via IonFS. It is not recommended to share access credentials for Ionburst Cloud or AWS.

For Ionburst Cloud, the ionburst_id and ionburst_key are generated from the Ionburst Cloud User Portal:

[example_profile]
ionburst_id=
ionburst_key=

AWS credentials, or access keys, are created in “My security credentials” in the AWS portal under Access keys for CLI, SDK, & API access.

[example]
aws_access_key_id=
aws_secret_access_key=

IonFSObject

IonFS passes around instances of IonFSObject for guidance on how to interact with objects being operated on. IonFSObject can represent a file on the local file system, or a file or directory on remote file system, and manages the repository on which the item exists or is to be stored.

The primary properties of IonFSObject are:

public string Repository { get; set; }
public string Name { get; set; }
public string Path { get; set; }
public DateTime LastModified { get; set; }
public Boolean IsFolder { get; set; }
public Boolean IsRoot { get; set; }
public Boolean IsRemote { get; set; }
public string FullName { get; }
public string FullFSName { get; }

Repository is the name given to a registered metadata repo. Name and Path are the filename and full path of the object, and FullName and FullFSName provide a single value representing the object fully qualified name.

IonFSMetadata

The Metadata stored within the files on S3 store just enough information to provide a window into Ionburst Cloud. IonFSMetadata has the following structure:

public List Id { get; }
public string Name { get; set; }
public long ChunkCount { get; set; }
public long MaxSize { get; set; }
public long Size { get; set; }
public String Hash { get; set; }
public String IV { get; set; }

Id is an ordered list of 1 to n GUIDs, each representing an individual object stored within Ionburst Cloud. Ionburst Cloud has a maximum size limit for any data uploaded in a single operation (this limit is available from the Ionburst Cloud client via GetUploadSizeLimit()). Any file being uploaded to Ionburst Cloud must be split into chunks less than or equal to this maximum size.

IonFS has its own limit (MaxSize) which can be any value up to this hard limit, any data object above this size will be split into multiple chunks; setting this value smaller than the maximum size can help parallelise Ionburst Cloud operations for smaller files, and allows a degree of optimisation.

A base 64 encoded SHA256 Hash for the original object is stored, along with the Nonce (IV) used if the object has been pre-encrypted using AES256.

Usage

ionfs meta ion://pictures/sm6.jpg
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Metadata for ion://first-S3/pictures/sm6.jpg
{
"Id": [
"3af99988-6db3-44ca-8133-250df1bd6bbb",
"cf450adf-5e91-411f-bfcd-70e9c05210bc",
"7ad0cdd2-bdea-499b-b7d2-3710e0a1a2ff"
],
"Name": "sm6.jpg",
"ChunkCount": 3,
"MaxSize": 65536,
"Size": 168496,
"Hash": "/SsHARgLJPIZeGUJMJX6Cx5PVgjXRBzMwZOUOth5L3w=",
"IV": "mtORrA4rPjhUACF4cr1kOA=="
}

IIonFSMetadata

IonFS stores its metadata using the constructs of the metadata storage provider. A specific metadata handler is required for each repo which must implement the interface IIonFSMetadata. The S3 metadata handler is contained in a class called MetadataS3.

public Task MakeDir(IonFSObject folder);
public Task Move(IonFSObject source, IonFSObject target);
public Task> List(IonFSObject folder, bool recursive);
public Task IsEmpty(IonFSObject folder);
public Task<bool> Exists(IonFSObject fso);
public Task PutMetadata(IonFSMetadata metadata, IonFSObject folder);
public Task GetMetadata(IonFSObject file);
public Task DelMetadata(IonFSObject fsObject);

A metadata item represents an object being stored within Ionburst Cloud. The directory structure is maintained only within the meta repository.

Repositories

IonFS:

public List Repositories { get; }

When more than one repository has been registered with IonFS it is possible to copy and move metadata between repositories.

Usage

ionfs repos
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Repositories (*default):
* ion://first-S3/ (Ionburst.Apps.IonFS.MetadataS3)
ion://second-S3/ (Ionburst.Apps.IonFS.MetadataS3)

Classifications

IonFS:

public Task> GetClassifications()

Usage

ionfs policy
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Classifications:
0: Restricted

PUT (Upload)

IonFS:

public Task> PutAsync(IonFSObject source, IonFSObject target)

Metadata:

public Task PutMetadata(IonFSMetadata metadata, IonFSObject folder)

PutAsync has 5 main tasks:

  • Manages the metadata for the file being uploaded
  • Performs any pre-encryption of the source file being uploaded
  • Splits the source file into chucks should it be over the IonFS MaxSize
  • Uploads each individual chunk (or burst) of data to Ionburst
ion.PutObjectRequest putObjectRequest = new ion.PutObjectRequest()
{
PolicyClassification = Classification,
Particle = burst.id.ToString()
};
putObjectRequest.DataStream = new MemoryStream(burst.data);
ion.PutObjectResult putResult = GetIonburst().PutAsync(putObjectRequest).Result;
  • Store the metadata. The metadata for an Ionburst Cloud object is stored in the metadata repository. The location in which is to be stored is described by an instance of:
IonFSObject.mh.PutMetadata(metadata, target);
  • In the case of S3:
s3.PutObjectRequest s3PutRequest = new s3.PutObjectRequest
{
BucketName = s3.GetBucket(),
Key = folder.FullName,
ContentBody = JsonConvert.SerializeObject(metadata)
};
s3.PutObjectResponse response = await s3.S3.PutObjectAsync(s3PutRequest);

Usage

ionfs put --help
IonFS put [options]  
ionfs put sm1.jpg ion://first-S3/pictures/ --name sm5.jpg

GET (Download)

IonFS:

public Task> GetAsync(IonFSObject file, IonFSObject to)

Metadata:

public Task GetMetadata(IonFSObject file)

GetAsync

  • Gets the metadata from the metadata repository
IonFSMetadata metadata = mh.GetMetadata(file);
  • In the case of S3
s3.GetObjectRequest getRequest = new s3.GetObjectRequest
{
BucketName = s3.GetBucket(),
Key = file.FullName
};
using s3.GetObjectResponse response = await s3.S3.GetObjectAsync(getRequest);
  • Downloads chunks from Ionburst Cloud
ion.GetObjectRequest getObjectRequest = new ion.GetObjectRequest
{
Particle = id.ToString()
};
ion.GetObjectResult getObjectResult = await GetIonburst().GetAsync(getObjectRequest);
  • Combines the data from chunks downloaded
  • Decrypts any data pre-encrypted as part of the upload

GetAsync returns a list chunks and the response code returned by Ionburst Cloud.

Usage

ionfs get --help
IonFS get [options]  
ionfs get ion://first-S3/pictures/sm5.jpg

DEL (Delete)

IonFS:

public Task> DelAsync(IonFSObject file)

Metadata:

public Task DelMetadata(IonFSObject fSObject)

DelAsync

  • Gets the metadata
IonFSMetadata metadata = await mh.GetMetadata(file);
  • Deletes each chunk from Ionburst Cloud
ion.DeleteObjectRequest delRequest = new ion.DeleteObjectRequest
{
Particle = id.ToString(),
TimeoutSpecified = true,
RequestTimeout = new TimeSpan(0, 2, 0)
};
  • Checks each chunk has been deleted
  • Deletes the file metadata
mh.DelMetadata(file);
  • In the case of S3
s3.DeleteObjectRequest s3DelRequest = new s3.DeleteObjectRequest
{
BucketName = s3.GetBucket(),
Key = fSObject.FullName
};
s3.DeleteObjectResponse response = await s3.S3.DeleteObjectAsync(s3DelRequest);

Usage

ionfs del --help
IonFS del [options]
ionfs del ion://first-S3/pictures/sm5.jpg

Crypto

The crypto functionality used by IonFS is encapsulated in the class IonFSCrypto. Currently only providing simple symmetric encryption using AES with a 256–bit key, the key can be provided, or generated using a passphrase.

Encryption can be requested by including the path to a 256–bit key using –key, or by suppling a passphrase using –passphrase. The keygen command can be used to create a 256bit key from a passphrase. 

Note: the crypto functionality currently implemented in IonFS should not be considered production ready, and currently only serves as an example of future functionality.

Put:

ionfs put sm1.jpg ion://first-S3/pictures/ --name sm6.jpg --passphrase "this is a test"

Get:

ionfs get ion://first-S3/pictures/sm6.jpg --passphrase "this is a test"

KeyGen:

ionfs keygen "this is a test"
256:54B0C58C7CE9F2A8B551351102EE09385DB473ED03AF8A7C0FE47316964ACD29

Future development will add asymmetric keys and other crypto/security options, and the option to integrate with key management systems such as AWS Key Management Service (KMS).

List

IonFS:

public Task> ListAsync(IonFSObject folder, bool recursive = false)

Metadata:

public Task> List(IonFSObject folder, bool recursive = false)

Usage

ionfs list ion://first-S3/pictures/
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://first-S3/pictures/
pictures/sm3.jpg 15/08/2020 18:39:15
pictures/sm4.jpg 16/08/2020 10:41:40
pictures/sm5.jpg 16/08/2020 11:19:01
pictures/sm6.jpg 16/08/2020 11:25:39

Version

### IonFS:

public List IonburstVersion { get; }

Version provides some important details relating to the current version of both the Ionburst Cloud API and the Ionburst SDK, and whether the service is currently online. In the event of any connectivity issues, this should be first point of call.

Usage

ionfs --version
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
We may guard your data, but we'll never take its freedom
Usage: IonFS --help
Ionburst is Online
API version: 1.0.10069.4082
SDK version: 1.1.3.0
Max Upload: 50000000 bytes
Max Size: 65536 bytes

Supported Commands

Remote names take the general form:

ion://

Repository (repo) names, are defined in the configuration file appsettings.json. If the first component of a path after ion:// is not in this list, the first item is assumed to be a folder, and the default repository is selected. Any path ending with a “/’ is treated as a folder.

ionfs listList the root folder; default fs prefix, default repo
ionfs list ion://List the root folder; default repo
ionfs list –recursive ion://List the root folder, recursively; default repo
ionfs put li.jpg ion://Upload the file li.jpg into the root folder; default repo
ionfs put –name li-1.jpg li.jpg ion://Upload the file li.jpg into the root folder, with a new name of li-4.jpg; default repo
ionfs put –name li-2.jpg –key mykey li.jpg ion://Encrypt li.jpg using AES256 with the symmetric key mykey; default repo
ionfs put –name li-3.jpg –passphrase “my secret” li.jpg ion://Encrypt il.jpg using AES256 with the symmetric key generated from the passphrase “my secret”; default repo
ionfs put –classification Secret –name li-4.jpg li.jpg ion://Upload the file li.jpg into the root folder, use a classification of Secret; default repo
ionfs get ion://li.jpgDownload the file li.jpg into the current folder; default repo
ionfs get –key mykey ion://li.jpgDecrypt li.jpg using the symmetric key mykey; default repo
ionfs get –passphrase “my secret” ion://li.jpgDecrypt li.jpg using the symmetric key generated from the passphrase “my secret”; default repo
ionfs get –name li2.jpg ion://li.jpgDownload the file li.jpg into the current folder with the new name li2.jpg; default repo
ionfs del ion://li.jpgDelete the file li.jpg; default repo
ionfs del –recursive ion://folder/Recursively delete all files under a folder; default repo
ionfs del ion://*.jpgWIP! Delete files matching the regular expression *.jpg; default repo
ionfs copy file.txt ion://PUT; default repo
ionfs copy ion://file.txt file.txtGET; default repo
ionfs copy *.txt ion://WIP! Copy all files matching the regular expression *.txt to the root folder; default repo
ionfs move file.txt ion://PUT + DEL (although DEL is currently disabled during testing); default repo
ionfs move ion://file.txt file.txtGET + DEL; default repo
ionfs move *.txt ion://WIP! Move all the files matching the regular expression *.txt to the root folder. Source files will be removed; default repo
ionfs move ion://file1.jpg ion://file2.jpgMove the metadata from one file to the other file; default repo
ionfs mkdir ion://folder/Create a folder; default repo
ionfs rmdir ion://folder/Remove a folder; default repo
ionfs meta ion://li.jpgQuery the metadata for the file li.jpg; default repo
ionfs sync data/ ion://data/Planned! Sync local and remote folders; default repo
ionfs policyShow the available Ionburst policies
ionfs –versionShows Ionburst API and SDK version, current MaxSize and MaxUploadSize, a Ionburst API status
ionfs reposShow what repos have been configured to store metadata; default repo is indicated with an *
ionfs put li.jpg ion:///Upload the file li.jpg into the root folder of the repo named <repo_name>
ionfs get ion:///li.jpgDownload the file li.jpg into the current folder of the repo named <repo_name>
ionfs del ion:///li.jpgDelete the file li.jpg from the repo named <repo_name>
ionfs keygen “my secret”Generate a 256bit key using the passphrase “my secret”
ionfs rm-id cf450adf-5e91-411f-bfcd-70e9c05210bcRemove the Id from Ionburst externally identified by the GUID cf450adf-5e91-411f-bfcd-70e9c05210bc
ionfs rm-meta ion://li.jpgRemove the metadata for the object li.jpg from the default repo, use with care, data will remain in Ionburst
ionfs add-meta metadata.json ion://Create an item in the default metadata repo from the data in metadata.json (see ionfs meta)