Ajanthan Eliyathamby's Tech Thoughts: 2015

Friday, December 25, 2015

Sending POST Requests to WSO2 ESB Proxy Service without Body

This article shows how to send a Request without body to a ESB proxy service. When we send a post request to ESB without body and any content aware mediators in the way then the Send Timeout error will be thrown in the ESB carbon log. This can be reproduced using a log mediator in the synapse configuration. To overcome this issue need to set the below property in synapse configuration.

<property name="FORCE_POST_PUT_NOBODY" scope="axis2" type="BOOLEAN" value="true"/>

Thursday, December 24, 2015

Differentiating the Admin/API/Proxy/Main Sequence service calls in WSO2 ESB at axis2 level

This article shows how to differentiate the Admin/API/ProxyMain sequence service calls using a axis2 handler. Here only giving a sample class to handle this requirement. To get more information on how to create module and handler refer [1], [2].

package com.custom.axis.loggers;

import org.apache.axis2.AxisFault;
import org.apache.axis2.context.MessageContext;
import org.apache.axis2.description.AxisService;
import org.apache.axis2.description.Parameter;
import org.apache.axis2.handlers.AbstractHandler;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.synapse.SynapseConstants;

/*
 * This class differentiate's the Proxy Service / API / Admin Service Calls.
 *
 */

public class CustomLoggingHandler extends AbstractHandler {

    private static final Log log = LogFactory
            .getLog(CustomLoggingHandler.class);

    public InvocationResponse invoke(MessageContext msgContext)
            throws AxisFault {

        log.debug("----------------------CustomLoggingHandler Invoked----------------------");

        AxisService axisService = msgContext.getAxisService();

        if (isProxyService(msgContext, axisService)) { // For Proxy Services checking the ServiceType Parameter
            if (log.isDebugEnabled()) {
                log.debug("Is a proxy service.");
            }
        } else { // For APIs/Main Sequence and Admin Services
            if (axisService.getAxisServiceGroup().getServiceGroupName()
                    .equals(SynapseConstants.SYNAPSE_SERVICE_NAME)) { //For APIs and main sequence
                if (log.isDebugEnabled()) {
                    log.debug("Is a API.");
                }
            } else { // For Admin Services
                if (log.isDebugEnabled()) {
                    log.debug("Is a Admin Service.");
                }
            }
        }

        return InvocationResponse.CONTINUE;
    }

    // Method to check whether request is for Proxy Service.
    public static boolean isProxyService(MessageContext messageContext,
            AxisService axisService) {

        if (axisService != null) {
            Parameter val = axisService.getParameter("serviceType");
            if (val != null
                    && val.getValue().toString().equalsIgnoreCase("Proxy")) {
                if (log.isDebugEnabled()) {
                    log.debug("Parameter Value Service Type: "
                            + val.getValue().toString());
                }
                return true;
            }
        }
        return false;
    }

[1] http://ajanthane.blogspot.com/2015/12/creating-axis2-module-for-custom.html
[2] http://ajanthane.blogspot.com/2015/10/creating-custom-axis2-handler.html

Creating Axis2 Module for Custom Logging and Integrating to WSO2 ESB

This article is about creating Axis2 module and Integrating it with WSO2 ESB.

1) Create a Maven Project using eclipse. And add the below to the dependencies in pom.xml.Below is the pom.xml created for this sample.


    4.0.0

    CustomAxis2ModuleProject
    CustomAxis2ModuleProject
    0.0.1-SNAPSHOT
    jar

    CustomAxis2ModuleProject
    http://maven.apache.org

    
        UTF-8
    

    
        
            junit
            junit
            3.8.1
            test
        
        
            org.apache.axis2.wso2
            axis2
            1.6.1.wso2v14
        
        
            org.apache.synapse
            synapse-core
            2.1.3-wso2v11
        
        
            commons-logging
            commons-logging
            1.1.1
        
    
    
        
            
                true
                daily
                ignore
            
            wso2-nexus
            http://maven.wso2.org/nexus/content/groups/wso2-public/
        
    
    
        
            
                true
                daily
                ignore
            
            wso2-nexus
            http://maven.wso2.org/nexus/content/groups/wso2-public/

2) Create the Axis2 Module Implementation.This can be done by creating a CustomLoggingModule class by implementing the Module class like below.

package com.custom.axis.loggers;

import org.apache.axis2.AxisFault;
import org.apache.axis2.context.ConfigurationContext;
import org.apache.axis2.description.AxisDescription;
import org.apache.axis2.description.AxisModule;
import org.apache.axis2.modules.Module;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.neethi.Assertion;
import org.apache.neethi.Policy;

public class CustomLoggingModule implements Module {

    private static final Log log = LogFactory.getLog(CustomLoggingModule.class);

    public void applyPolicy(Policy arg0, AxisDescription arg1) throws AxisFault {
        // TODO Auto-generated method stub

    }

    public boolean canSupportAssertion(Assertion arg0) {
        // TODO Auto-generated method stub
        return false;
    }

    public void engageNotify(AxisDescription arg0) throws AxisFault {
        // TODO Auto-generated method stub

    }

    public void init(ConfigurationContext arg0, AxisModule arg1)
            throws AxisFault {
        if (log.isDebugEnabled()) {
            log.debug("Custom Logging Module Init Method Invoked");
        }
    }

    public void shutdown(ConfigurationContext arg0) throws AxisFault {
        // TODO Auto-generated method stub

    }

}

3) Create the Handlers. This can be done by creating a CustomLoggingHandler class by extending AbstractHandler class.

package com.custom.axis.loggers;

import org.apache.axis2.AxisFault;
import org.apache.axis2.context.MessageContext;
import org.apache.axis2.handlers.AbstractHandler;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

/*
 * Custom Axis2 Handler for custom logging.
 *
 */

public class CustomLoggingHandler extends AbstractHandler {

    private static final Log log = LogFactory
            .getLog(CustomLoggingHandler.class);

    public InvocationResponse invoke(MessageContext msgContext)
            throws AxisFault {

        // Add your custom logging logic here

        return InvocationResponse.CONTINUE;
    }

}

4) Create a module.xml in src/main/resources/META-INF. The sample module.xml is below. Here only invoking the handler in the InFlow.

Note: Here due to auto formatting in the editor the <inFlow> is converted to <inflow>. Use <inFlow> when you try out the sample.

5) The maven project structure will look like below.

6) Build the project using Maven build.

7) Copy the jar created to ESB_HOME/repository/deployment/server/axis2modules/ and rename with .mar extension.

8) Modify the axis2.xml to add the phase to the InFlow.

9) To assign the module globally to all the service requests add the module to axis2.xml as below.

10) Now the custom module is ready to use.

References

[1] http://wso2.com/library/777/

[2] https://axis.apache.org/axis2/java/core/docs/modules.html

[3] https://axis.apache.org/axis2/java/core/docs/Axis2ArchitectureGuide.html#requirements

Sunday, September 6, 2015

FTP Over Proxy using VFS and WSO2 ESB

This article shows how to transfer files from local to vsftpd server, through WSO2 ESB and load balanced multiple proxies. The architecture is given below:

Figure 1: Process flow architecture

Prerequisite

· jdk1.7.0_79

· WSO2 ESB (Tested in wso2esb-4.9.0-ALPHA4 / wso2esb-4.9.0-BETA-SNAPSHOT)

· BIND9

· Squid

· vsftpd

· Ubuntu

Setting up the environment

Java

Download the jdk1.7.0_79 from http://www.oracle.com/technetwork/java/javase/downloads/jre7-downloads-1880261.html

unzip and setup the JAVA_HOME and PATH using the below commands:

export JAVA_HOME=LOCATION_OF_EXTRACTED_JAVA
export PATH=$JAVA_HOME/bin:$PATH

WSO2 ESB

Download it from http://wso2.com/products/enterprise-service-bus/ and extract to the local directory.

For wso2esb-4.9.0-BETA-SNAPSHOT : http://svn.wso2.org/repos/wso2/people/malaka/ESB/beta/

BIND9

Berkeley Internet Name Daemon (BIND) is the reference implementation of the Domain Name System (DNS) protocols and it's a popular software for translating domain names into IP addresses and usually found on Linux servers. Use the below steps to install bind9 in Ubuntu.

apt-get install bind9 dnsutils

To make bind as a DNS load balancer, configure the bind as below.

1) sudo vi /etc/resolv.conf : change the ip to the local machines IP where the bind was installed.

2) sudo vi /etc/bind/named.conf.local: create a zone as below:

zone "testproxy.com" {
      type master;
      file "/etc/bind/db.testproxydns";
};

3) create a file with the below to do the load balancing.

sudo vi /etc/bind/db.testproxydns

add the below:

Squid

Squid is a full-featured web proxy cache server application which provides proxy and cache services for Hyper Text Transport Protocol (HTTP), File Transfer Protocol (FTP), and other popular network protocols. Use the below steps to install squid in ubuntu.

apt-get update
apt-get install squid

Install another squid proxy instance in another ubuntu instance for the use of multiple proxies.

/etc/squid3/squid.conf

vsftpd

FTP is used to transfer files from one host to another over TCP network. vsftpd which is lightweight and less Vulnerability.

apt-get update
apt-get install vsftpd

Create FTP user to transfer the files. Use the below steps.

sudo groupadd ftp-users
sudo useradd --home FTP_HOME_DIRECTORY --group ftp-users ftpadmin
sudo passwd ftpadmin 
sudo chown -R ftpadmin FTP_HOME_DIRECTORY
sudo chmod 755 FTP_HOME_DIRECTORY

make pam_service_name = ftp in /etc/vsftpd.conf

uncomment the write_enable=YES in /etc/vsftpd.conf

Integration

Create a proxy service [1] in WSO2 ESB to read the content from the curl post and create the file and send it to the ftp server through http proxy.

curl post sample:

curl -X POST -H 'Content-Type: application/json' -d " {\"key\":\"wso2-`date +%Y%m%d%H%M%S_%N`\", \"payload\": \"`cat test3.txt`\"} " -v http://ESB_HOST:PORT/services/FileTransferProxyService

proxy service sample:

Note: Here due to auto formatting in the editor the <inSequence> is converted to <insequence>. Use <inSequence> when you try out the sample.

<proxy name="FileTransferProxyService" startonload="true" trace="disable" transports="https http" xmlns="http://ws.apache.org/ns/synapse">
   <description>
   <target>
      <endpoint>
<address uri="vfs:ftp://ftppass:ftppass@ftphost:21/home/ubuntu/proxy_ftp/ftp_home/?proxyServer=PROXY_HOST&amp;proxyPort=PROXYPORT&amp;tryCount=3"/>
</endpoint>
      <insequence>
         <log level="full"/>
         <property expression="json-eval($.key)" name="file_name"/>
         <property name="OUT_ONLY" value="true"/>
         <property name="FORCE_SC_ACCEPTED" scope="axis2" value="true"/>
         <property name="HTTP_SC" scope="axis2" value="200"/>
         <property expression="fn:concat(get-property('file_name'), '.xml')" name="transport.vfs.ReplyFileName" scope="transport"/>
         <property name="OUT_ONLY" value="true"/>
      </insequence>
   </target>
</description>
</proxy>

Useful commands for testing

1) Simultaneous curl posts:

curl -X POST -H 'Content-Type: application/json' -d " {\"key\":\"wso2-`date +%Y%m%d%H%M%S_%N`\", \"payload\": \"`cat test3.txt`\"} " -v http://ESB_HOST:PORT/services/FileTransferProxyService & curl -X POST -H 'Content-Type: application/json' -d " {\"key\":\"wso2-`date +%Y%m%d%H%M%S_%N`\", \"payload\": \"`cat test3.txt`\"} " -v http://ESB_HOST:PORT/services/FileTransferProxyService & curl -X POST -H 'Content-Type: application/json' -d " {\"key\":\"wso2-`date +%Y%m%d%H%M%S_%N`\", \"payload\": \"`cat test3.txt`\"} " -v http://ESB_HOST:PORT/services/FileTransferProxyService

2) Reading the logs at real time

squid proxy logs: tail -f /var/log/squid3/access.log

vsftpd logs: tail -f /var/log/vsftpd.log

Encountered issues and Workarounds

1) 425 Bad IP connecting

425 issue will be produced when the IP of the control connection and the IP of the data connection received by FTP server during a FTP transfer were different. And it's said this can happen when we use a load balancer [3].

This can be overcome by making the pasv_promiscous=YES in the /etc/vsftpd.conf.

2) Additional .lock files created in ftp location

When posting above 500 simultaneous requests, in the ftp location found some additional files extension with .lock. This is due to the created lock files were not released, after the file tranfer completed.

In vfs, there is a feature to create .lock files to avoid duplicate files to be written to ftp server while transferring files. By using transport.vfs.Locking=disable, this .lock file creation can be disabled.

References:

[1] https://docs.wso2.com/display/ESB470/Adding+a+Proxy+Service

[2] https://docs.wso2.com/display/ESB481/VFS+Transport

[3] http://www.jscape.com/blog/bid/80512/Active-v-s-Passive-FTP-Simplified