stream CSV to JSON via FTP





In this post I will be discussing how to stream CSV to JSON via FTP , which will enable you to integrate your application with any number of legacy applications to extract data.

Problem domain

It is a common requirement for applications to be able to download and parse a Comma Separated Value (CSV) file from a FTP location.  Usually this is to enable some kind of data interaction between systems.

CSV is a common, relatively simple file format that is widely supported by consumer, business, and scientific applications. Among its most common uses is moving tabular data between programs that natively operate on incompatible (often proprietary and/or undocumented) formats. This works because so many programs support some variation of CSV at least as an alternative import/export format.

Prerequisites

We’ll be making use of two Node modules to implement this functionality:




Install modules

The two modules we’re going to use are available through the Node Package Manager (npm) so installing them is pretty easy using the command line

The Module

The following module is my straw man proposal ,  to establish whether I could achieve this functionality quite simply in node.js. The aim of the application is quite simply access an FTP folder, stream a CSV file on the server, and transform that file into JSON objects so eventually I could import them into a database.

For the purpose of this module, we’ll only be printing the JSON objects out the console for review.




Conclusion

Despite the fact that Douglas Crockford, author of Javascript : The Good Parts, would be breaking out into cold sweats as far as JavaScript coding conventions and practices go, we pretty simple solution to the problem.   We will be able to use this library now to further expand our solution.

This is another fine example of, just how extensible node.js is , and how by making use of some pre-existing libraries we are able to quickly and efficiently knock up solutions to common problems.

 

Gary Woodfine

Freelance Full Stack Developer at threenine.co.uk
Helps businesses by improving their technical proficiencies and eliminating waste from the software development pipelines.

A unique background as business owner, marketing, software development and business development ensures that he can offer the optimum business consultancy services across a wide spectrum of business challenges.

Latest posts by Gary Woodfine (see all)

Affiliate Disclaimer

Disclosure: Please note that some of the links included in the blog posts are affiliate links, which means I will earn a commission if you decide to make a purchase.

I only include affiliate links to products, services and companies that I have personal experience and have actually used. I only recommend them because they are helpful and useful, not because of the small commissions I make if you decide to buy something.

Please do not spend any money on these products unless you feel they are relevant, suitable and will help you achieve your goals.

Buying anyone of these products and the commisions earned will not make me an overnite multi millionaire, but they will help to cover the hosting costs and compensate for the countless hours and effort I put in to create meaningful and valuable free content I provide to help others succeed.

You've also undoubtedly noticed that I also use various advertising networks - unless of of course you're using an Ad blocker, this is also an attempt to reduce my monthly deficit I experience in providing free relevant, valuable and on occassion insightful content for the benefit of others.

I only really make some money if people click on the ads, unless of course you consider 1c per 1000 impressions real money!

Bear in mind just how many impressions I need to make before I can cover my £20 hosting costs a month!

If you are using an adblocker and guilt stricken you can of course donate using any of the buttons below, but I won't be holding my breath.

Buy me a coffeeBuy me a coffee

π