Build a Secure File Sharing Application

Josh Fraser

Josh Fraser

Engineer @ Ionburst Cloud

Background

Ionburst Cloud offers a revolutionary way to store data securely and privately in the Cloud, beyond the reach of hackers and unwanted surveillance. Data is transformed and persisted as redundant fragments across collections of storage nodes called Cloudlets™.

Resources needed:

Overview

In this tutorial, we will integrate Ionburst Cloud as the back-end storage for the open-source file-sharing application, Firefox Send.

Firefox Send is a free, end-to-end encrypted file-sharing application developed by Mozilla, that allows users to easily and safely share files over the Web. The Send back-end is written in Node.js, allowing us to integrate with the Ionburst Node.js SDK.

Note: The public Firefox Send instance was temporarily taken down on the 7th July, due to reports it was being abused by malicious actors sharing malware. This appears to have also had an impact on the stability of the code base on Github. For this reason, we have based our Ionburst Cloud integration on an older, stable commit that we found to work – available here.

IonFS Diagram

Digging Into the Send Source Code

From a cursory review of the source code and running the application locally, it looked like the focus of our integration would be the server directory, which houses the code for Send’s back-end services.

Of most interest was the storage sub-directory, which contains the functionality for integrating Send with the following:

  • Local filesystem storage;
  • Google Cloud Storage;
  • Amazon S3;

A review of these files outlined common pieces of functionality expected from storage integrations:

  • length – returns the object size from the configured storage method;
  • getStream – retrieves the object from the configured storage method;
  • set – uploads or writes the object to the configured storage method;
  • del – removes the object from the configured storage method;

Of these, Ionburst Cloud provides functionality for three out of four, as it does not expose the ability to query object size. Further digging suggested that the length function was being used to set the Content-Length header on file download to the user, so wasn’t a functional requirement for the storage integration.

Exploring the storage sub-directory also confirmed how Send handles object metadata. In development, Send uses a local, in-memory store to track each object but is designed to use Redis in production. To gain a better understanding of the Send application, all Ionburst Cloud integration work was carried out using Redis as the Send metadata store.

The final checks required were to see how the Send back-end handled storage configuration. The base back-end configuration is handled in the config.js file found in the server directory, which defines which storage method is selected by the index.js file found in the storage sub-directory.

Integrating Ionburst Cloud – Configuration

To begin integrating Ionburst Cloud with Send, we first have to add new configuration options to the Send project to tell it to use Ionburst Cloud as the new storage method, along with the initial Ionburst SDK configuration.

The Ionburst SDK was added to the project using npm:

npm install ionburst-sdk-javascript

A local Redis instance is deployed to track Send metadata using Docker:

docker run -ti -p 6379:6379 redis:latest

A config.json file is added to the root of the Send project for the Ionburst Cloud config file.

{
"Ionburst": {
"Profile": "example",
"IonBurstUri": "https://api.eu-west-1.ionburst.io/",
"TraceCredentialsFile": "ON"
}
}

A new configuration item is added to the Send config.js file for Ionburst Cloud. Note: this configuration entry is only used to select Ionburst Cloud as the chosen back-end storage, and does not perform any other configuration. An entry is also added to ensure Redis was used in development:

const conf = convict({
ionburst: {
format: String,
default: 'hiya'
},
… (omitted)
redis_host: {
format: String,
default: '127.0.0.1',
env: 'REDIS_HOST'
},
… (omitted)
});

A configuration option is added to the storage index.js file, to ensure Ionburst Cloud was selected as the storage method:

class DB {
constructor(config) {
let Storage = null;
if (config.ionburst) {
Storage = require('./ionburst');
} else if (config.s3_bucket) {
Storage = require('./s3');
} else if (config.gcs_bucket) {
Storage = require('./gcs');
} else {
Storage = require('./fs');
}
this.log = mozlog('send.storage');
this.dir = config.file_dir;
this.storage = new Storage(config, this.log);
this.redis = createRedisClient(config);
this.redis.on('error', err => {
this.log.error('Redis:', err);
});
}
… (omitted)
}

Finally, an ionburst.js file is created within the storage sub-directory, and a constructor created for applicable configuration:

class IonburstStorage {
constructor(config, log) {
this.log = log;
}

Integrating Ionburst Cloud – File Operations

Ionburst Cloud PUT

From the storage index.js file, we can see how Send kicks off a file upload to its configured storage:

async set(id, file, meta, expireSeconds = config.default_expire_seconds) {
const prefix = getPrefix(expireSeconds);
const filePath = `${prefix}-${id}`;
await this.storage.set(filePath, file);
this.redis.hset(id, 'prefix', prefix);
if (meta) {
this.redis.hmset(id, meta);
}
this.redis.expire(id, expireSeconds);
}

From this, we can see that Send generates an identifier for each file stored, before passing it and the file to the configured storage method. As Ionburst Cloud doesn’t care how a given object is identified, we can simply pass this identifier to Ionburst Cloud too.

To upload the data to Ionburst Cloud, the following function was created in ionburst.js:

set(id, file) {
return new Promise((resolve, reject) => {
const filepath = path.join(this.dir, id);
console.log(filepath);
const fstream = fs.createWriteStream(filepath);
file.pipe(fstream);
file.on('error', err => {
fstream.destroy(err);
});
fstream.on('error', err => {
fs.unlinkSync(filepath);
reject(err);
});
fstream.on('finish', async function () {
var upload_data = fs.readFileSync(filepath);
console.log(upload_data);
let data = await ionburst.putAsync({
id: id,
data: upload_data,
})
console.log(data);
fs.unlink(filepath, function (error) {
if (error) {
throw error;
}
});
resolve();
});
});
}

We encountered some issues passing the file object directly to the Ionburst SDK. Instead we leveraged the existing filesystem functionality to write the file to a temporary directory, creating a read stream for the Ionburst SDK, then removed the temporary file after successful upload.

This temporary file/directory leveraged functionality used by Send’s file-system storage, and it was simply a matter of pulling the temporary directory configuration into the Ionburst Cloud storage constructor:

class IonburstStorage {
constructor(config, log) {
this.log = log;
this.dir = config.file_dir;
mkdirp.sync(this.dir);
}

Ionburst Cloud GET

Similar to the upload function, the main download functionality can be found in the storage index.js file:

async get(id) {
const filePath = await this.getPrefixedId(id);
console.log(filePath);
return this.storage.getStream(filePath);
}

To keep things simple, we replicate the same temporary file functionality for the file download from Ionburst Cloud:

async getStream(id) {
let data = await ionburst.getAsync(id);
var getPath = path.join(this.dir, id)
fs.writeFileSync(getPath, data);
var returnData = fs.createReadStream(getPath);
fs.unlink(getPath, function (error) {
if (error) {
throw error;
}
});
return returnData;
}

We first grab the file from Ionburst Cloud, write it to the temporary directory, and create and return a read stream.

Ionburst Cloud DELETE

Send requires delete functionality from the configured storage method to remove uploaded files once they have reached their download limit, or expiry time.

The Ionburst Cloud delete function was simple to implement:

del(id) {
return ionburst.delete(id, function (err, data) {
if (err) {
throw err;
}
console.log(data);
});
}

Integrating Ionburst Cloud – Final fixes

With the basic upload, download and delete functionality for Send’s integration into Ionburst Cloud, it was a simple case of tidying things up.

Ionburst Cloud Configuration issues

The Ionburst SDK for Node.js uses two methods to obtain its configuration, a config.json file to be kept in the root of the project and credentials file located in the user’s home directory, or by environment variable.

As we don’t typically develop in Node.js, Send is the first project we’ve encountered that uses Webpack. While beginning the Ionburst Cloud integration, we found that Webpack interferes with the methods used by the Ionburst SDK to find the root directory of the project, namely: require.main.filename and process.mainModule.filename.

There are two workarounds for this issue. Ionburst Cloud credentials can instead be specified by environment variable, and the Ionburst Cloud API URI specified on import:

const Ionburst = require('ionburst-sdk')
var ionburst = Ionburst("https://api.example.ionburst.io/");

Or, the Ionburst SDK settings method can be modified as such:

_getRoot() {
//return path.dirname(require.main.filename || process.mainModule.filename);
return process.env.PWD;
}

While this method works, we haven’t yet pushed this change to the live SDK, so we can evaluate the impact of the change. This issue is currently being tracked here on Gitlab.

Any feedback from the Node.js community is much appreciated.

Caveats

Content-Length

As Ionburst Cloud has no method of returning a stored object’s size, and the Send metadata doesn’t track it either; a different method needs to be found to attach the object size to the final download request.

Currently, the Content-Length is set server-side in filelist.js but removing the original method doesn’t appear to have an effect on functionality.

File Size

The publicly hosted version of Send has a file size limit of 1GB, or 2.5GB for registered users. Ionburst Cloud currently supports a maximum object size of 50MB, with larger objects requiring client-side processing before upload.

As a simple proof-of-concept, we’ve kept this limit in place – allowing larger objects to be uploaded to Ionburst Cloud by Firefox Send would require additional development beyond the scope of this tutorial.

Conclusion

Integrating Firefox Send with Ionburst Cloud was a quick and simple process. It provided an opportunity to try out our Node.js SDK in practice. There were some intricacies to be handled, but it allowed us to demonstrate how Ionburst Cloud can be integrated to secure the data of existing applications.