Ionburst Cloud offers a revolutionary way to store data securely and privately in the Cloud, beyond the reach of hackers and unwanted surveillance. Data is transformed and persisted as redundant fragments across collections of storage nodes called Cloudlets™.
- An active Ionburst Cloud subscription. Get started for free.
- The GitLab repository for the IonFS CLI project is available here.
Ionburst Cloud does not allow the data it holds to be browsed. Its privacy by design default means the ability to request a list of objects stored does not exist. If this functionality is required, it is down to the client applications to track the metadata in an appropriate manner to meet its requirements. IonFS has been developed to illustrate how a client application typically integrates with Ionburst Cloud.
IonFS provides a set of tools to manage data stored by Ionburst Cloud as if it were a remote filesystem. Whilst IonFS stores primary data items within Ionburst Cloud, the metadata is stored in an S3 bucket; anyone with access to this bucket, and the appropriate Ionburst Cloud credentials, can interact with the data stored within.
S3 was selected due to its availability and prevalence in the market, although any platform capable of storing data can be used. S3 is an example of what is called a metadata repository, or “repo” for short.
IonFS has one simple focus – to enable files to be stored by Ionburst Cloud, and to retain the ability to interact as if they had been stored on a filesystem. Traditional filesystem metaphors allow file and folders to be created, deleted, uploaded, download, moved, copied, and renamed.
Furthermore, even though Ionburst Cloud stores the data in a completely secure and fully redundant manner, it is possible to add further encryption and decryption for data from within IonFS.
[Only objects up to size cap currently 50MB – on Ionburst Cloud products can be uploaded, so additional manipulation and tracking is required. IonFS implements this chunking functionality, with a configurable threshold.]
IonFS is built using .NET Core 3.1, allowing cross-platform builds for Windows, MacOS and Linux. Using the tools available through the AWS S3 .NET SDK, it is possible programmatically to build a file and directory structure with metadata wrapped around the fundamental building blocks of Ionburst Cloud; PUT, GET, and DEL. The Ionburst .NET SDK is used exclusively to interact with the Ionburst Cloud API.
The primary configuration for IonFS is managed within
appsettings.json, located in the
.ionfs folder located in your home directory.
The IonFS section contains the main configuration items:
MaxSize controls the chunking of data items being uploaded, for extra details to be logged to screen
Verbose can be set to true. Note, some commands allow this to be overridden on the command line using
DefaultClassification is the default Ionburst Cloud policy applied to data being uploaded, this can be explicitly set on the PUT command.
The Repositories section allows multiple metadata repositories to be accessed.
If a repository is not explicitly included in the remote path, the
DefaultRepository will be used.
The Ionburst Cloud section is required to access the Ionburst SDK.
The AWS section is required to access the AWS SDK.
Credential files are required to access both Ionburst Cloud and AWS. These files can be found in your home directory within
.aws folders respectively. Explicit access must be granted, and suitable access credentials provided for any resources being used via IonFS. It is not recommended to share access credentials for Ionburst Cloud or AWS.
For Ionburst Cloud, the ionburst_id and ionburst_key are generated from the Ionburst Cloud User Portal:
AWS credentials, or access keys, are created in “My security credentials” in the AWS portal under Access keys for CLI, SDK, & API access.
IonFS passes around instances of
IonFSObject for guidance on how to interact with objects being operated on.
IonFSObject can represent a file on the local file system, or a file or directory on remote file system, and manages the repository on which the item exists or is to be stored.
The primary properties of
Repository is the name given to a registered metadata repo.
Path are the filename and full path of the object, and
FullFSName provide a single value representing the object fully qualified name.
The Metadata stored within the files on S3 store just enough information to provide a window into Ionburst Cloud. IonFSMetadata has the following structure:
Id is an ordered list of 1 to n GUIDs, each representing an individual object stored within Ionburst Cloud. Ionburst Cloud has a maximum size limit for any data uploaded in a single operation (this limit is available from the Ionburst Cloud client via
GetUploadSizeLimit()). Any file being uploaded to Ionburst Cloud must be split into chunks less than or equal to this maximum size.
IonFS has its own limit (
MaxSize) which can be any value up to this hard limit, any data object above this size will be split into multiple chunks; setting this value smaller than the maximum size can help parallelise Ionburst Cloud operations for smaller files, and allows a degree of optimisation.
A base 64 encoded SHA256 Hash for the original object is stored, along with the Nonce (IV) used if the object has been pre-encrypted using AES256.
IonFS stores its metadata using the constructs of the metadata storage provider. A specific metadata handler is required for each repo which must implement the interface
IIonFSMetadata. The S3 metadata handler is contained in a class called
A metadata item represents an object being stored within Ionburst Cloud. The directory structure is maintained only within the meta repository.
When more than one repository has been registered with IonFS it is possible to copy and move metadata between repositories.
PutAsync has 5 main tasks:
- Manages the metadata for the file being uploaded
- Performs any pre-encryption of the source file being uploaded
- Splits the source file into chucks should it be over the IonFS MaxSize
- Uploads each individual chunk (or burst) of data to Ionburst
- Store the metadata. The metadata for an Ionburst Cloud object is stored in the metadata repository. The location in which is to be stored is described by an instance of:
- In the case of S3:
- Gets the metadata from the metadata repository
- In the case of S3
- Downloads chunks from Ionburst Cloud
- Combines the data from chunks downloaded
- Decrypts any data pre-encrypted as part of the upload
GetAsync returns a list chunks and the response code returned by Ionburst Cloud.
- Gets the metadata
- Deletes each chunk from Ionburst Cloud
- Checks each chunk has been deleted
- Deletes the file metadata
- In the case of S3
The crypto functionality used by IonFS is encapsulated in the class
IonFSCrypto. Currently only providing simple symmetric encryption using AES with a 256–bit key, the key can be provided, or generated using a passphrase.
Encryption can be requested by including the path to a 256–bit key using
–key, or by suppling a passphrase using
keygen command can be used to create a 256bit key from a passphrase.
Note: the crypto functionality currently implemented in IonFS should not be considered production ready, and currently only serves as an example of future functionality.
Future development will add asymmetric keys and other crypto/security options, and the option to integrate with key management systems such as AWS Key Management Service (KMS).
Version provides some important details relating to the current version of both the Ionburst Cloud API and the Ionburst SDK, and whether the service is currently online. In the event of any connectivity issues, this should be first point of call.
Remote names take the general form:
Repository (repo) names, are defined in the configuration file
appsettings.json. If the first component of a path after
ion:// is not in this list, the first item is assumed to be a folder, and the default repository is selected. Any path ending with a “/’ is treated as a folder.
|ionfs list||List the root folder; default fs prefix, default repo|
|ionfs list ion://||List the root folder; default repo|
|ionfs list –recursive ion://||List the root folder, recursively; default repo|
|ionfs put li.jpg ion://||Upload the file li.jpg into the root folder; default repo|
|ionfs put –name li-1.jpg li.jpg ion://||Upload the file li.jpg into the root folder, with a new name of li-4.jpg; default repo|
|ionfs put –name li-2.jpg –key mykey li.jpg ion://||Encrypt li.jpg using AES256 with the symmetric key mykey; default repo|
|ionfs put –name li-3.jpg –passphrase “my secret” li.jpg ion://||Encrypt il.jpg using AES256 with the symmetric key generated from the passphrase “my secret”; default repo|
|ionfs put –classification Secret –name li-4.jpg li.jpg ion://||Upload the file li.jpg into the root folder, use a classification of Secret; default repo|
|ionfs get ion://li.jpg||Download the file li.jpg into the current folder; default repo|
|ionfs get –key mykey ion://li.jpg||Decrypt li.jpg using the symmetric key mykey; default repo|
|ionfs get –passphrase “my secret” ion://li.jpg||Decrypt li.jpg using the symmetric key generated from the passphrase “my secret”; default repo|
|ionfs get –name li2.jpg ion://li.jpg||Download the file li.jpg into the current folder with the new name li2.jpg; default repo|
|ionfs del ion://li.jpg||Delete the file li.jpg; default repo|
|ionfs del –recursive ion://folder/||Recursively delete all files under a folder; default repo|
|ionfs del ion://*.jpg||WIP! Delete files matching the regular expression *.jpg; default repo|
|ionfs copy file.txt ion://||PUT; default repo|
|ionfs copy ion://file.txt file.txt||GET; default repo|
|ionfs copy *.txt ion://||WIP! Copy all files matching the regular expression *.txt to the root folder; default repo|
|ionfs move file.txt ion://||PUT + DEL (although DEL is currently disabled during testing); default repo|
|ionfs move ion://file.txt file.txt||GET + DEL; default repo|
|ionfs move *.txt ion://||WIP! Move all the files matching the regular expression *.txt to the root folder. Source files will be removed; default repo|
|ionfs move ion://file1.jpg ion://file2.jpg||Move the metadata from one file to the other file; default repo|
|ionfs mkdir ion://folder/||Create a folder; default repo|
|ionfs rmdir ion://folder/||Remove a folder; default repo|
|ionfs meta ion://li.jpg||Query the metadata for the file li.jpg; default repo|
|ionfs sync data/ ion://data/||Planned! Sync local and remote folders; default repo|
|ionfs policy||Show the available Ionburst policies|
|ionfs –version||Shows Ionburst API and SDK version, current MaxSize and MaxUploadSize, a Ionburst API status|
|ionfs repos||Show what repos have been configured to store metadata; default repo is indicated with an *|
|ionfs put li.jpg ion:///||Upload the file li.jpg into the root folder of the repo named <repo_name>|
|ionfs get ion:///li.jpg||Download the file li.jpg into the current folder of the repo named <repo_name>|
|ionfs del ion:///li.jpg||Delete the file li.jpg from the repo named <repo_name>|
|ionfs keygen “my secret”||Generate a 256bit key using the passphrase “my secret”|
|ionfs rm-id cf450adf-5e91-411f-bfcd-70e9c05210bc||Remove the Id from Ionburst externally identified by the GUID cf450adf-5e91-411f-bfcd-70e9c05210bc|
|ionfs rm-meta ion://li.jpg||Remove the metadata for the object li.jpg from the default repo, use with care, data will remain in Ionburst|
|ionfs add-meta metadata.json ion://||Create an item in the default metadata repo from the data in metadata.json (see ionfs meta)|