FTP Over Proxy using VFS and WSO2 ESB

This article shows how to transfer files from local to vsftpd server, through WSO2 ESB and load balanced multiple proxies. The architecture is given below:

Figure 1: Process flow architecture

Prerequisite

·        jdk1.7.0_79

·        WSO2 ESB (Tested in wso2esb-4.9.0-ALPHA4 / wso2esb-4.9.0-BETA-SNAPSHOT)

·        BIND9

·        Squid

·        vsftpd

·        Ubuntu

Setting up the environment

Java

Download the  jdk1.7.0_79 from   http://www.oracle.com/technetwork/java/javase/downloads/jre7-downloads-1880261.html

unzip and setup the JAVA_HOME and PATH using the below commands:

export JAVA_HOME=LOCATION_OF_EXTRACTED_JAVA
export PATH=$JAVA_HOME/bin:$PATH

WSO2 ESB

Download it from http://wso2.com/products/enterprise-service-bus/ and extract to the local directory.

For wso2esb-4.9.0-BETA-SNAPSHOT  : http://svn.wso2.org/repos/wso2/people/malaka/ESB/beta/

BIND9

Berkeley Internet Name Daemon (BIND) is the reference implementation of the Domain Name System (DNS) protocols and it's a popular software for translating domain names into IP addresses and usually found on Linux servers. Use the below steps to install bind9 in Ubuntu.

apt-get install bind9 dnsutils

To make bind as a DNS load balancer, configure the bind as below.

1) sudo vi /etc/resolv.conf : change the ip to the local machines IP where the bind was installed.

2) sudo vi /etc/bind/named.conf.local: create a zone as below:

zone "testproxy.com" {
      type master;
      file "/etc/bind/db.testproxydns";
};

3) create a file with the below to do the load balancing.

sudo vi /etc/bind/db.testproxydns

add the below:

Squid

Squid is a full-featured web proxy cache server application which provides proxy and cache services for Hyper Text Transport Protocol (HTTP), File Transfer Protocol (FTP), and other popular network protocols. Use the below steps to install squid in ubuntu.

apt-get update
apt-get install squid

Install another squid proxy instance in another ubuntu instance for the use of multiple proxies.

/etc/squid3/squid.conf

vsftpd

FTP is used to transfer files from one host to another over TCP network. vsftpd which is lightweight and less Vulnerability.

apt-get update
apt-get install vsftpd

Create FTP user to transfer the files. Use the below steps.

sudo groupadd ftp-users
sudo useradd --home FTP_HOME_DIRECTORY --group ftp-users ftpadmin
sudo passwd ftpadmin 
sudo chown -R ftpadmin FTP_HOME_DIRECTORY
sudo chmod 755 FTP_HOME_DIRECTORY

make pam_service_name = ftp in /etc/vsftpd.conf

uncomment the write_enable=YES in /etc/vsftpd.conf

Integration

Create a proxy service [1] in WSO2 ESB to read the content from the curl post and create the file and send it to the ftp server through http proxy.

curl post sample:

curl -X POST -H 'Content-Type: application/json' -d " {\"key\":\"wso2-`date +%Y%m%d%H%M%S_%N`\", \"payload\": \"`cat test3.txt`\"} " -v http://ESB_HOST:PORT/services/FileTransferProxyService

   

proxy service sample:

Note: Here due to auto formatting in the editor the <inSequence> is converted to <insequence>. Use <inSequence> when you try out the sample. 

<proxy name="FileTransferProxyService" startonload="true" trace="disable" transports="https http" xmlns="http://ws.apache.org/ns/synapse">
   <description>
   <target>
      <endpoint>
<address uri="vfs:ftp://ftppass:ftppass@ftphost:21/home/ubuntu/proxy_ftp/ftp_home/?proxyServer=PROXY_HOST&amp;proxyPort=PROXYPORT&amp;tryCount=3"/>
</endpoint>
      <insequence>
         <log level="full"/>
         <property expression="json-eval($.key)" name="file_name"/>
         <property name="OUT_ONLY" value="true"/>
         <property name="FORCE_SC_ACCEPTED" scope="axis2" value="true"/>
         <property name="HTTP_SC" scope="axis2" value="200"/>
         <property expression="fn:concat(get-property('file_name'), '.xml')" name="transport.vfs.ReplyFileName" scope="transport"/>
         <property name="OUT_ONLY" value="true"/>
      </insequence>
   </target>
</description>
</proxy>

Useful commands for testing

1) Simultaneous curl posts:

curl -X POST -H 'Content-Type: application/json' -d " {\"key\":\"wso2-`date +%Y%m%d%H%M%S_%N`\", \"payload\": \"`cat test3.txt`\"} " -v http://ESB_HOST:PORT/services/FileTransferProxyService & curl -X POST -H 'Content-Type: application/json' -d " {\"key\":\"wso2-`date +%Y%m%d%H%M%S_%N`\", \"payload\": \"`cat test3.txt`\"} " -v http://ESB_HOST:PORT/services/FileTransferProxyService & curl -X POST -H 'Content-Type: application/json' -d " {\"key\":\"wso2-`date +%Y%m%d%H%M%S_%N`\", \"payload\": \"`cat test3.txt`\"} " -v http://ESB_HOST:PORT/services/FileTransferProxyService

2) Reading the logs at real time

squid proxy logs: tail -f /var/log/squid3/access.log

vsftpd logs: tail -f /var/log/vsftpd.log

Encountered issues and Workarounds

1) 425 Bad IP connecting

425 issue will be produced when the IP of the control connection and the IP of the data connection received by FTP server during a FTP transfer were different. And it's said this can happen when we use a load balancer [3].

This can be overcome by making the pasv_promiscous=YES in the /etc/vsftpd.conf.

2) Additional .lock files created in ftp location

When posting above 500 simultaneous requests, in the ftp location found some additional files extension with .lock. This is due to the created lock files were not released, after the file tranfer completed.

In vfs, there is a feature to create .lock files to avoid duplicate files to be written to ftp server while transferring files. By using transport.vfs.Locking=disable, this .lock file creation can be disabled.

<address uri="vfs:ftp://ftppass:ftppass@ftphost:21/home/ubuntu/proxy_ftp/ftp_home/?proxyServer=PROXY_HOST&amp;proxyPort=PROXYPORT&amp;tryCount=3&amp;transport.vfs.Locking=disable"/>

References:

[1] https://docs.wso2.com/display/ESB470/Adding+a+Proxy+Service

[2] https://docs.wso2.com/display/ESB481/VFS+Transport

[3] http://www.jscape.com/blog/bid/80512/Active-v-s-Passive-FTP-Simplified