Monday, May 15, 2017

Monitoring / Analyzing WSO2 ESB 4.9.0 Performance with Java Flight Recorder ( JFR )

Sometimes there is a need that, we need to check which mediator or code segment causing the performance problem in ESB mediation. This is needed to find the root cause behind the delayed response. In this kind of situation, JFR is a tool, which can help us to analyze, which part of mediation logic causing the delay or taking time to execute.

This article explains how we can enable and monitor WSO2 ESB 4.9.0 with JFR.

What is JFR?

Java Flight Recorder ( JFR ) is a tool provided by Java from the JDK release 7u40 or above. This is a  profiler to determine which parts of your code are causing large amounts of memory allocation or causing excess CPU to be consumed. Refer [1] for more information on JFR.
[1] https://www.javacodegeeks.com/2015/01/java-flight-recorder-jfr.html

Configuring WSO2 ESB 4.9.0 to use JFR


Consider here I'm using the below Java version:
 ajanthan@ajanthan-ThinkPad-T440p:~$ java -version  
 java version "1.7.0_80"  
 Java(TM) SE Runtime Environment (build 1.7.0_80-b15)  
 Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)  
 ajanthan@ajanthan-ThinkPad-T440p:~$ echo $JAVA_HOME  
 /usr/lib/jvm/java-7-oracle  
 ajanthan@ajanthan-ThinkPad-T440p:~$   

1) Copy the below to wso2server.sh file
 -XX:+UnlockCommercialFeatures \  
 -XX:+FlightRecorder \  
 -XX:FlightRecorderOptions=defaultrecording=true,disk=true,repository=./tmpjfr,dumponexit=true,dumponexitpath=./ \  

Here, defaultrecording=true : Will start to capture from the start of the application.
disk=true : Continuosly write to disk
repository=./tmpjfr : A temporary folder will be created once started and write to a file inside
dumponexitpath=./ : Once the server / application stopped, then the temporary file will be moved to the root folder of that directory.

Check http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/java.html for more information.

2) Updated wso2server.sh file part will be as below:
 while [ "$status" = "$START_EXIT_STATUS" ]  
 do  
   $JAVACMD \  
   -Xbootclasspath/a:"$CARBON_XBOOTCLASSPATH" \  
   -Xms256m -Xmx1024m -XX:MaxPermSize=256m \  
   -XX:+HeapDumpOnOutOfMemoryError \  
   -XX:HeapDumpPath="$CARBON_HOME/repository/logs/heap-dump.hprof" \  
   $JAVA_OPTS \  
   -Dcom.sun.management.jmxremote \  
   -classpath "$CARBON_CLASSPATH" \  
   -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \  
   -Djava.io.tmpdir="$CARBON_HOME/tmp" \  
   -Dcatalina.base="$CARBON_HOME/lib/tomcat" \  
   -Dwso2.server.standalone=true \  
   -Dcarbon.registry.root=/ \  
   -Djava.command="$JAVACMD" \  
   -Dcarbon.home="$CARBON_HOME" \  
   -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \  
   -Dcarbon.config.dir.path="$CARBON_HOME/repository/conf" \  
   -Djava.util.logging.config.file="$CARBON_HOME/repository/conf/etc/logging-bridge.properties" \  
   -Dcomponents.repo="$CARBON_HOME/repository/components/plugins" \  
   -Dconf.location="$CARBON_HOME/repository/conf"\  
   -Dcom.atomikos.icatch.file="$CARBON_HOME/lib/transactions.properties" \  
   -Dcom.atomikos.icatch.hide_init_file_path=true \  
   -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false \  
   -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true \  
   -Dcom.sun.jndi.ldap.connect.pool.authentication=simple \  
   -Dcom.sun.jndi.ldap.connect.pool.timeout=3000 \  
   -Dorg.terracotta.quartz.skipUpdateCheck=true \  
   -Djava.security.egd=file:/dev/./urandom \  
   -Dfile.encoding=UTF8 \  
   -XX:+UnlockCommercialFeatures \  
   -XX:+FlightRecorder \  
   -XX:FlightRecorderOptions=defaultrecording=true,disk=true,repository=./tmpjfr,dumponexit=true,dumponexitpath=./ \  
   -Djava.net.preferIPv4Stack=true \  
   -Dcom.ibm.cacheLocalHost=true \  
   -DworkerNode=false \  
   org.wso2.carbon.bootstrap.Bootstrap $*  
   status=$?  
 done  

3) Start the WSO2 ESB server.

4) Execute the below command ( After moving to the ESB_HOME directoty ) to check whether the JFR is running properly.
 jcmd `cat wso2carbon.pid` JFR.check  

Result:
 ajanthan@ajanthan-ThinkPad-T440p:~/wso2/blog/jfr/wso2esb-4.9.0$ jcmd `cat wso2carbon.pid` JFR.check  
 22685:  
 Recording: recording=0 name="HotSpot default" maxage=15m (running)  
 ajanthan@ajanthan-ThinkPad-T440p:~/wso2/blog/jfr/wso2esb-4.9.0$   

5) Now if we want to take dumps frequently can execute the below command:
 jcmd `cat wso2carbon.pid` JFR.dump recording=0 filename="recording-test.jfr"  

6) Here, for example I'm going to capture a dump during a load test, to check which component / mediator taking more execution time. For that I'm using the
below api.
 <api xmlns="http://ws.apache.org/ns/synapse" name="TestJFR" context="/jfr">  
   <resource methods="POST" uri-template="/jfrcall">  
    <inSequence>  
      <log level="custom">  
       <property name="STATUS" value="--------------------TestJFR API Invocation : START---------------------"/>  
      </log>  
      <script language="js">var log = mc.getServiceLog();          log.info("-----------Executing Script Mediator-------------------- ");</script>  
      <payloadFactory media-type="json">  
       <format>  {"test":"Out from Payload factory Mediator"}  </format>  
       <args/>  
      </payloadFactory>  
      <log level="custom">  
       <property name="STATUS" value="--------------------TestJFR API Invocation : END---------------------"/>  
      </log>  
      <respond/>  
    </inSequence>  
   </resource>  
 </api>  

7) Configured Jmeter as below:




7) While running the load test use the below command to take the dump.
 ajanthan@ajanthan-ThinkPad-T440p:~/wso2/blog/jfr/wso2esb-4.9.0$ jcmd `cat wso2carbon.pid` JFR.dump recording=0 filename="recording-test-2.jfr"  
 22685:  
 Dumped recording 0, 21.4 MB written to:  
 /home/ajanthan/wso2/blog/jfr/wso2esb-4.9.0/recording-test-2.jfr  
 ajanthan@ajanthan-ThinkPad-T440p:~/wso2/blog/jfr/wso2esb-4.9.0$  

8) Now we have taken the dump and need to analyze it. To do that we can use "jmc" command to open the Java Mission Control screen.

Through file -> openfile option we can open the .jfr file. Once we loaded we can do the analysis based on the screens available through JMC console.

Below are some of the screen's captured during this testing for reference.




You can refer http://blog.takipi.com/oracle-java-mission-control-the-ultimate-guide/#jfrinitial to get more on analysis screens.

9) That's all, we have successfully configured WSO2 ESB 4.9.0 with JFR for analysis.

References


[1] http://isuru-perera.blogspot.com/2015/02/java-flight-recorder-continuous-recordings.html

No comments:

Post a Comment