stream CSV to JSON via FTP

In this post I will be discussing how to stream CSV to JSON via FTP , which will enable you to integrate your application with any number of legacy applications to extract data.

Problem domain

It is a common requirement for applications to be able to download and parse a Comma Separated Value (CSV) file from a FTP location. Usually this is to enable some kind of data interaction between systems.

CSV is a common, relatively simple file format that is widely supported by consumer, business, and scientific applications. Among its most common uses is moving tabular data between programs that natively operate on incompatible (often proprietary and/or undocumented) formats. This works because so many programs support some variation of CSV at least as an alternative import/export format.
Wikipedia

Prerequisites

We'll be making use of two Node modules to implement this functionality:

Install modules

The two modules we're going to use are available through the Node Package Manager (npm) so installing them is pretty easy using the command line

sudo npm install ftp

sudo npm install fast-csv

The Module

The following module is my straw man proposal , to establish whether I could achieve this functionality quite simply in node.js. The aim of the application is quite simply access an FTP folder, stream a CSV file on the server, and transform that file into JSON objects so eventually I could import them into a database.

For the purpose of this module, we'll only be printing the JSON objects out the console for review.

// Required modules
var Client = require('ftp');
var csv = require('fast-csv');
 
//ftp credentials, evetually we will extract this from a //DB of some sort
var hostName = "ftp.someserver.com";
var userName = "SomeUserName";
var password = "SomePassword";
var fileName = 'SomeFileName.csv';
 
// Here lies the meat and potatoes of the application
var c = new Client();
c.on('ready', function() {
    c.get(fileName, function(err, stream) {
        if (err) throw err;
        stream.once('close', function() { c.end(); });
        csv.fromStream(stream,  {headers: true})
            .on("data", function(data){
            console.log(data);
        })
            .on("end", function(){
                console.log("done");
            });
    });
});
c.connect({host:hostName,
    user: userName,
    password:password
});

Conclusion

Despite the fact that Douglas Crockford, author of Javascript : The Good Parts, would be breaking out into cold sweats as far as JavaScript coding conventions and practices go, we pretty simple solution to the problem. We will be able to use this library now to further expand our solution.

This is another fine example of, just how extensible node.js is , and how by making use of some pre-existing libraries we are able to quickly and efficiently knock up solutions to common problems.

About
Latest Posts

Gary Woodfine

Technical Director at threenine.co.uk

Gary is Technical Director at threenine.co.uk, an independent software vendor specialising in IoT, Field Service and associated managed services,enabling customers to be efficient, productive, secure and scale-able.