Explanation on Talend Functional Architecture

This article explains how a Talend Functional Architecture and Project creation to Execution works. Here I'm using a flow diagram as it will be easy to understand the flow through it. Also note that here I'm considering the components which are mainly involve when we design a project through Talend Studio.

Note: The Talend 5.6.1 considered when analyzing the flow and here I'm specifically going through the flow where Artifacts loaded to Nexus and then retrieving it to execute. There are another two ways to load and execute the projects:
1) Export and Upload the Project to Job Conductor
2) Execute directly from Talend Studio.

Here in the below diagram:
Blue Font:  Explains the flow when creating users and providing authorizations.
Green Font: Explains the flow when creating and designing the project.

References

[1] https://help.talend.com/reader/kj4pY1SkKiM4v91iTLuI3Q/NIYCD0QApSsXDzTkhMcXXg
[2] https://help.talend.com/reader/kj4pY1SkKiM4v91iTLuI3Q/RAQIxfznH6FF4A8AvlxK4Q

Useful LDAP Search Queries

This article provides some mostly used LDAP queries, which were commonly used in Integration projects.

H - ldap URI eg: ldap://192.168.1.1/
D - Distinguished Name of the Ad User
b - Search base
s sub -  {base|one|sub|children}
x - Use simple authentication instead of SASL
w - Password

 1) Retrieving Particular User Information based on CN  
   
 ldapsearch -LLL -H <LDAP-URI> -D '<LDAP-AD-USER>' -b '<LDAP-SEARCH-BASE>' -s sub -x "(&(objectClass=User)(CN=Test User \282001\29))" "cn objectGUID" -w <PASSWORD>  
   
 Note: Here \28 is for ( and \29 is for ), otherwise there will be a exception  
   
 2) Retrieving Particular User Information based on DN  
   
 ldapsearch -LLL -H <LDAP-URI> -D '<LDAP-AD-USER>' -b '<LDAP-SEARCH-BASE>' -s sub -x "(&(objectClass=User)(distinguishedname=<DN-USER>))" "cn" "objectGUID" -w <PASSWORD>  
   
 3) Retrieving Only the Active Users  
   
 Need to use this filter (!(userAccountControl:1.2.840.113556.1.4.803:=2)) to retrieve only the active users.  
   
 ldapsearch -LLL -H <LDAP-URI> -D '<LDAP-AD-USER>' -b '<LDAP-SEARCH-BASE>' -s sub -x "(&(objectClass=User)(distinguishedname=<DN-USER>)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))" "cn" "objectGUID" -w <PASSWORD>  
   
 4) Retrieve Groups of a particular User  
   
 ldapsearch -LLL -H <LDAP-URI> -D '<LDAP-AD-USER>' -b '<LDAP-SEARCH-BASE>' -s sub -x "(&(objectClass=Group)(member=<DN-USER>))" "cn" "objectGUID" -w <PASSWORD>  
   
 Note: Here for member=<Need to provide distinguishedName>  
   
 5) Retrieve Users of a particular Group  
   
 ldapsearch -LLL -H <LDAP-URI> -D '<LDAP-AD-USER>' -b '<LDAP-SEARCH-BASE>' -s sub -x "(&(objectClass=User)(memberOf=<DN-GROUP>))" "cn" "objectGUID" -w <PASSWORD>  
   
 6) Retrieving Users of a Group with pagination  
   
 ldapsearch -LLL -H <LDAP-URI> -D '<LDAP-AD-USER>' -b '<LDAP-SEARCH-BASE>' -s sub -x "(&(objectClass=Group)(CN=LoopTest2Group))" "cn" "objectGUID" "member;range=0-1" -w <PASSWORD>  
   

Retrieving subset of xml using xslt mediator in WSO2 ESB 5.0.0

This article explain a solution to a scenario, where our back end service is not supporting a pagination and responds with a large xml and our client expects an API which need to support pagination.

The flow diagram below summarizes the task we are going to achieve.

The API I have created for this sample as below:

<api xmlns="http://ws.apache.org/ns/synapse" name="PaginationAPI" context="/listStudents">
   <resource methods="GET" uri-template="/getPaginatedStudentsList?start={start}&amp;count={count}" protocol="http">
      <inSequence>
         <log level="custom">
            <property name="STATUS" value="PaginationAPI Invoked..."/>
            <property name="start" expression="$ctx:query.param.start"/>
            <property name="count" expression="$ctx:query.param.count"/>
         </log>
         <script language="js">var count = parseInt(mc.getProperty('query.param.count'));                var start = parseInt(mc.getProperty('query.param.start'));        var startIndex = parseInt(start);       var endIndex = parseInt(start) + parseInt(count) + 1;         mc.setProperty('startIndex', startIndex);        mc.setProperty('endIndex', endIndex);</script>
         <call>
            <endpoint>
               <http method="GET" uri-template="http://www.mocky.io/v2/5be818f7300000750058c312"/>
            </endpoint>
         </call>
         <log level="full" description="before xslt transformation"/>
         <xslt key="gov:/pagination-test.xslt">
            <property name="startIndex" expression="$ctx:startIndex"/>
            <property name="endIndex" expression="$ctx:endIndex"/>
         </xslt>
         <respond/>
      </inSequence>
      <outSequence/>
      <faultSequence/>
   </resource>
</api>
                        

The back end service has been created in mocky.io and the sample data response from it will be as below:

<soapenv:Envelope
    xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
 <soapenv:Header/>
 <soapenv:Body>
  <students>
   <school>St.Michael's College, Batticaloa</school>
   <address>Batticaloa, Srilanka</address>
   <student>
    <recno>1</recno>
    <name>NTest1</name>
    <surname>STest1</surname>
    <address>ATest1</address>
    <age>27</age>
   </student>
   <student>
    <recno>2</recno>
    <name>NTest1</name>
    <surname>STest1</surname>
    <address>ATest1
   </address>
    <age>27</age>
   </student>
   <student>
    <recno>3</recno>
    <name>NTest1</name>
    <surname>STest1</surname>
    <address>ATest1</address>
    <age>27</age>
   </student>
   <student>
    <recno>4</recno>
    <name>NTest1</name>
    <surname>STest1</surname>
    <address>ATest1</address>
    <age>27</age>
   </student>
   <student>
    <recno>5</recno>
    <name>NTest1</name>
    <surname>STest1</surname>
    <address>ATest1</address>
    <age>27</age>
   </student>
   <student>
    <recno>6</recno>
    <name>NTest1</name>
    <surname>STest1</surname>
    <address>ATest1</address>
    <age>27</age>
   </student>
   <student>
    <recno>7</recno>
    <name>NTest1</name>
    <surname>STest1</surname>
    <address>ATest1</address>
    <age>27</age>
   </student>
   <student>
    <recno>8</recno>
    <name>NTest1</name>
    <surname>STest1</surname>
    <address>ATest1</address>
    <age>27</age>
   </student>
   <student>
    <recno>9</recno>
    <name>NTest1</name>
    <surname>STest1</surname>
    <address>ATest1</address>
    <age>27</age>
   </student>
   <student>
    <recno>10</recno>
    <name>NTest1</name>
    <surname>STest1</surname>
    <address>ATest1</address>
    <age>27</age>
   </student>
   <student>
    <recno>11</recno>
    <name>NTest1</name>
    <surname>STest1</surname>
    <address>ATest1</address>
    <age>27</age>
   </student>
   <student>
    <recno>12</recno>
    <name>NTest1</name>
    <surname>STest1</surname>
    <address>ATest1</address>
    <age>27</age>
   </student>
  </students>
 </soapenv:Body>
</soapenv:Envelope>

Then the XSLT will be as below:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet
 xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
 <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>
 <xsl:param name="startIndex"></xsl:param>
 <xsl:param name="endIndex"></xsl:param>
 <xsl:template match="/">
  <students>
   <xsl:apply-templates select="//students/student[position() > $startIndex and position() &lt; $endIndex]"/>
  </students>
 </xsl:template>
 <xsl:template match="//students/student[position() > $startIndex and position() &lt; $endIndex]">
  <xsl:copy-of select="."/>
 </xsl:template>
</xsl:stylesheet>

Below are the test results:
1) When calling the API with URL: http://172.17.0.1:8280/listStudents/getPaginatedStudentsList?start=0&count=20

<students>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>1</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>2</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1
   </address>
        <age>27</age>
    </student>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>3</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>4</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>5</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>6</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>7</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>8</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>9</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>10</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>11</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>12</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
</students>

2) When calling the API with URL: http://172.17.0.1:8280/listStudents/getPaginatedStudentsList?start=5&count=3

<students>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>6</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>7</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
    <student xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        <recno>8</recno>
        <name>NTest1</name>
        <surname>STest1</surname>
        <address>ATest1</address>
        <age>27</age>
    </student>
</students>

That's it... and hope this helps someone implementing a pagination scenario.