File Upload

The Editor Node.JS libraries provide an Upload class that can be used with the upload and uploadMany field types to provide end users with the ability to upload files to a server. The goal of the Upload class is to make the server-side aspect of file uploading as simple as possible, while retaining the same flexibility that you would expect from the Editor libraries.

The documentation here refers to the Node.JS aspect of the upload field type. Please refer to the client-side documentation for details on how to configure the Javascript aspect of the file upload options in Editor.

Overview

The Upload class provides the following features:

• Save files to local file system or database
• Store information about each file in a database table - what information is stored is configurable
• File saving override options for custom actions (for example creating thumbnails)
• Easy validation of file extensions
• Custom validation for more complex options

In a typical situation it is expected that you will store the file on the server's file system (not the database), although that is possible if you require) and meta information about the file on the database so it can be accessed through the file() method.

Consider for example the following two tables:

Table: staff                      Table: image
+----+---------+-------+-------+  +----+-----------------+----------+
| id | name    | title | image |  | id | fileName        | fileSize |
+----+---------+-------+-------+  +----+-----------------+----------+
| 1  | Allan   | CEO   | 1     |  | 1  | Allan.png       | 4532     |
| 2  | Charlie | CTO   | 4     |  | 4  | Charlie-CTO.png | 9408     |
| 3  | Fred    | CFO   | 9     |  | 9  | Photo.png       | 2054     |
+----+---------+-------+-------+  +----+-----------------+----------+

Here we have a staff table where each row has a reference to the image table. The Editor Node.JS initialisation to read the data from the staff table might look like the following:

router.all('/api/upload', async function(req, res) {
    let editor = new Editor(db, 'staff').fields(
        new Field('name'),
        new Field('title'),
        new Field('image')
    );

    await editor.process(req.body);
    res.json(editor.data());
});

This gives us the information required on the client-side to show staff information, but not yet to upload files and display information about files.

The documentation below will build upon this basic example to add the ability to upload a new file in the Editor interface. Note that although this example focuses on images, any file type can be used - what files you use will be dependent upon your own requirements. Additionally the database structure shown above is not fixed - for example you can use a link table if you require by adding an additional left join.

The Upload class

The new Upload instance is attached to the field that should have its value edited when a new image is selected - in the example discussed here that is the image column. This is done using the Field.upload() method:

new Field( "image" )
    .upload( new Upload( ... ) )
    .setFormatter( Format.ifEmpty(null) );

The Upload class provides the ability to make use of chaining, so it is easy to construct a full CRUD application, including conditional uploads in a few lines of code.

Note that we also use a set formatter, the Format.ifEmpty formatter in this case, which will cause Editor to write a null value to the database when no image value is submitted to the client-side. This is optional, and will depend upon the exact schema used in your database, but this is typically desirable.

One-to-many support

The uploadMany field type provides the ability to assign multiple files to a single field in an Editor form - i.e. one-to-many. Editor's one-to-many support is provided through the Mjoin class (see the Mjoin documentation).

The following documentation will focus primarily on using the upload type with fields in the main Editor instance, but the Upload class can be attached to fields defined in an Mjoin instance in exactly the same was as for fields in the main Editor instance to provide the data required for the uploadMany field type.

File location

The Upload constructor accepts a single, optional, parameter - a string with the full system path where the uploaded file should be stored. Note that it is important you use the full path rather than a relative path to reduce complexities of include files! Use the __dirname global variable to get the path to the directory where your file resides if needed. A relative location can be added to that location (e.g. /.. to go back up a directory).

The path given can have three special strings in it that the Upload class will replace so the file can be easily referenced from a database and to remove the chance of a file name collision:

• {id} - The primary key value of the row that refers to this file
• {extn} - The file extension (including the dot)
• {name} - The full file name (including the extension).

These parameters are simply inserted into the string where you would like them to appear:

new Field( "image" )
    .upload( new Upload( __dirname + '/../public/uploads/{id}.{extn}' )
    .setFormatter( Format.ifEmpty(null) );

If you wish to perform more complex actions for storing files, such using a directory structure to limit the number of files in each directory to something safe for your operating system if you are working with large numbers of files, please see the Custom upload actions section below.

If no value is given in the constructor for the file path the Upload class will not attempt to write the uploaded file to the file system unless Upload.action() is used (which operates in exactly the same way as the constructor). This can be useful if you are storing the file's binary data in the database.

Database information

For efficiency you will typically wish to store information about each file in a database table. In combination with the file() method, discussed above, this makes it very easy to get information about each file such as the file's path to load it in a web-browser, the file type and even if the file exists!

The Upload class provides a db() method that is used for this purpose. It takes up to four parameters:

1. string - The database table name where the file information should be stored
2. string - The database primary key column name. The Upload class requires that the database table have a single primary key so each row can be uniquely identified.
3. object - A list of the columns to be written to on upload. The objects key's are the database column names and the values can be one of:
   • A Upload.DbType enum - describes information that should be obtains from the file - see the table below for a full list of the options available
   • A value (e.g a string, number, etc) which is written directly to the database. Note that as a string, the value can contain the same substitution values as the path described above (e.g. {name}, {extn} and {id}) which will be replaced based on the information from the file.
   • An function which is executed and the return value written to the database. The function is passed two parameters: 1. The database connection, 2. The file upload object for the file being uploaded.
4. function (optional) - A formatting function for each row read from the database. A single parameter is passed in, the object for the file from the database. It expects no return value - any manipulation of the object is propagated back.

The constants that can be used for the values are:

Constant	Meaning
Upload.DbType.Content	File content
Upload.DbType.ContentType	Content type
Upload.DbType.Extn	File extension (with the dot)
Upload.DbType.FileName	File name (with extension)
Upload.DbType.FileSize	File size (bytes)
Upload.DbType.MimeType	MIME type
Upload.DbType.ReadOnly	Editor will not write a value to this field, just read its value which might come from a default, trigger or be updated from some other location.
Upload.DbType.SystemPath	Full system path to the file

The db() method is used for both writing information to the database, and also automatically populating the data for the file() method. The parameters defined here are also available on the client-side (with the exception of the Content option).

Continuing the illustrative example we now have:

new Field( 'image' )
    .upload( new Upload( __dirname + '/../public/uploads/{id}.{extn}' )
        .db('image', 'id', {
            fileName: Upload.DbType.FileName,
            fileSize: Upload.DbType.FileSize
        })
    )
    .setFormatter( Format.ifEmpty(null) );

Validation

It is important that the files uploaded are validated to ensure that they are of an expected type (you don't want a Word document uploaded where your software expects an image for example). The Upload class provides a validator() method that operates in the same manner as the Field.validator() method and accepts two different forms:

• Validation by the provided functions
• Custom validation function

Provided validation functions

The Validate class provides two built in validation methods that can be used to validate file uploads. The function signatures for the file validations differ from the field validators in that they do not accept a Validate.Options parameter - instead they take parameters which are defined by the validator, including the error message to be returned.

The built in validators are:

• fileExtensions( valid: string[], errorMessage: string ) - Check that the uploaded file's extension is one of those provided in the valid list. This check is case-insensitive, so png in the list would accept .PNG, .pnG, etc, as valid extensions). Also, it is worth noting that you should not rely upon extension validation only for secure validation as it is trivial to modify a file name extension. It is however a useful and simple sanity check.
• fileSize( size: number, errorMessage: string ) - Check that the uploaded file is equal to or below a given size (in bytes).

As an example, to check that only image files are uploaded you might use:

new Upload( ... )
    .validator(
        Validate.fileExtensions(
            ['png', 'jpg', 'gif'],
            'Only image files can be uploaded (png, jpg and gif)'
        )
    )

Custom validation

A custom validation method can perform any check you wish, for example actually going into the file's binary content to ensure that it is of the type expected, limiting file size, etc. This is done through use of a function that is passed to the Upload.validator() method.

The function, when executed, will be given a single method, an object that contains the details of the uploaded file. The return value should be true if validation passes and a string containing an error message if it fails.

Consider the following example

new Upload( ... )
    .validator( file => {
        if ( file.size >= 500000 ) {
            return "Files must be smaller than 500K";
        }

        return true;
    } )

Multiple validation methods can be used for a single Upload instance by simply calling Upload.validator() multiple times. The validators are executed in sequence when a file is uploaded.

The structure of the object passed in as the file parameter is:

{
    field: string;
    file: string;     // full tmp path where the file was uploaded by the server to
    filename: string; // name + extn
    encoding: string;
    mimetype: string;
    size: number;
    extn: string;
    name: string;
}

Custom upload actions

There may be occasions when you find that the simple file move options provided by the Upload class constructor aren't enough for your application. This might be because you want to create thumbnails from an uploaded image so they are immediately available for use, or to use a complex directory structure for the uploaded files. Whatever the reason the Upload class provides the option of specifying a closure function that will be executed in place of the detail action.

As with the simple file path the closure method is specified using the constructor or the Upload.action() method. When executed it will receive two parameters:

1. An information object about the file, containing the information shown above.
2. The primary key value for the file if Upload.db() was used to store information in the database (otherwise this value will be null).

The return value is the value given to the client-side to identify the file. Typically this will be the primary key value, but you could also use a file path or any other information.

Consider the following example:

new Field( 'image' )
    .upload( new Upload( async (fileInfo, id) => {
            await rename( fileInfo.file, '/images/'+id );
        } )
        .db('image', 'id', {
            fileName: Upload.DbType.FileName,
            fileSize: Upload.DbType.FileSize
        })
    )
    .setFormatter( Format.ifEmpty(null) );

In this case we are simply storing the file using rename() (a promisifyed version of rename), but much more complex actions can be taken if required.

Deletion of orphaned files

Providing users the ability to upload files to a server also requires the ability to remove files - although users only have this ability indirectly (by deleting the data that has a reference to the file). The Upload class has a dbClean() method available that gives you this ability.

The method takes either one or two arguments:

1. A table and field name string (in table.field format) to tell Editor where to look for files that are in use. This parameter is optional for a standard field, but is required when used inside an Mjoin field.
2. A callback function that is executed whenever Editor performs a database clean (which is on create, edit and remove - not upload). It is passed a single argument; a list of information from the database about rows which are no longer referenced by the external table (note this only checks the host Editor table, not any other database tables - so if you have multiple tables using the same files table, you will need to execute your own queries). The information provided in the given list of dictionaries is defined by the options specified for the Upload.db() method.

The callback function can return true to indicate that the rows should be removed from the database. false or null will result in no action being performed by the libraries.

The callback function can therefore be used to delete files from the file system, or potentially to execute your own queries on the database to ensure there are no unwanted orphaned records.

let unlink = promisify(fs.unlink);

new Field('image')
    .setFormatter(Format.ifEmpty(null))
    .upload(
        new Upload(__dirname + '/../public/uploads/{id}.{extn}')
            .db('files', 'id', {
                filename: Upload.Db.FileName,
                filesize: Upload.Db.FileSize,
                web_path: '/uploads/{id}.{extn}',
                system_path: Upload.Db.SystemPath
            })
            .dbClean(async function(data) {
                for (let i = 0, ien = data.length; i < ien; i++) {
                    await unlink(data[i].system_path);
                }
                return true;
            })

Events

Events can also be used to handle the deletion of files that are no longer referenced. The preRemove event for example can be used to remove files immediately prior to a row being deleted.

Node.JS API documentation

The Node.JS API developer documentation for the Editor Node.JS classes is available for detailed and technical discussion about the methods and classes discussed above.