stream CSV to JSON via FTP





In this post I will be discussing how to stream CSV to JSON via FTP , which will enable you to integrate your application with any number of legacy applications to extract data.

Problem domain

It is a common requirement for applications to be able to download and parse a Comma Separated Value (CSV) file from a FTP location.  Usually this is to enable some kind of data interaction between systems.

CSV is a common, relatively simple file format that is widely supported by consumer, business, and scientific applications. Among its most common uses is moving tabular data between programs that natively operate on incompatible (often proprietary and/or undocumented) formats. This works because so many programs support some variation of CSV at least as an alternative import/export format.

Prerequisites

We’ll be making use of two Node modules to implement this functionality:




Install modules

The two modules we’re going to use are available through the Node Package Manager (npm) so installing them is pretty easy using the command line

The Module

The following module is my straw man proposal ,  to establish whether I could achieve this functionality quite simply in node.js. The aim of the application is quite simply access an FTP folder, stream a CSV file on the server, and transform that file into JSON objects so eventually I could import them into a database.

For the purpose of this module, we’ll only be printing the JSON objects out the console for review.




Conclusion

Despite the fact that Douglas Crockford, author of Javascript : The Good Parts, would be breaking out into cold sweats as far as JavaScript coding conventions and practices go, we pretty simple solution to the problem.   We will be able to use this library now to further expand our solution.

This is another fine example of, just how extensible node.js is , and how by making use of some pre-existing libraries we are able to quickly and efficiently knock up solutions to common problems.

 

Gary Woodfine

Freelance Full Stack Developer at threenine.co.uk
Helps businesses by improving their technical proficiencies and eliminating waste from the software development pipelines.

A unique background as business owner, marketing, software development and business development ensures that he can offer the optimum business consultancy services across a wide spectrum of business challenges.
π