Thing which seemed very Thingish inside you is quite different when it gets out into the open and has other people looking at it

Wednesday, January 29, 2014

Real time log event analysis with WSO2 BAM(CEP features)

One of the major problems when it comes to managing/monitoring distributed systems is not being able to detect when your system is giving issues all of a sudden. Lets say you have like 100 servers and you need to detect when a fatal error appears in the system and you want to act right away. This is nearly impossible if you do not have proper monitoring tools.
WSO2 BAM with WSO2 CEP features has the perfect mechanism to monitor real time logs and alert it to relevant parties when an expectation/ suspicious error occurs.



In this sample I will be using WSO2 Appserver and WSO2 BAM in order to demonstrate the monitoring and alerting capabilities of WSO2 BAM 2.4.0.

  • WSO2 Appserver will be used to send log events to BAM.
  • WSO2 BAM will do all the monitoring and alerting when logs are captured.
Send Logs to BAM


To send logs all you need to is go to WSO2_AS_HOME/repository/conf/log4j.properties and add LOGEVENT to the root logger. And start Appserver. (please make sure your BAM server is up and running before starting the Appserver)

To test if the logs are successfully sent to BAM, you can log in to cassandra explorer and see if there is a new column family created under EVENT_KS keyspace

ie log.AS.0.2013.1.12

Configuring WSO2 BAM for REAL TIME analytics

In this demo, I will be sending an email if an ERROR log occurs in WSO2 Appserver.  Since we will be using mail transport in BAM to send email alerts to recipients, we have to enable mail transport in BAM. To do that go to repository->conf->axis2->axis2-client.xml and add email configurations.


<transportSender name="mailto" class="org.apache.axis2.transport.mail.MailTransportSender">
       <parameter name="mail.smtp.from">wso2esb.mail@gmail.com</parameter>
       <parameter name="mail.smtp.user">wso2esb.mail</parameter>
       <parameter name="mail.smtp.password">wso2mail</parameter>
       <parameter name="mail.smtp.host">smtp.gmail.com</parameter>
       <parameter name="mail.smtp.port">587</parameter>
       <parameter name="mail.smtp.starttls.enable">true</parameter>
       <parameter name="mail.smtp.auth">true</parameter>
   </transportSender>

Here you can give your own email configurations. 

Restart/Start the BAM server.

Assuming the logs are getting published to BAM, lets see how we can capture these log events for real time analytics.

Step 1 - Creating Event Adapters

In order to do real time analytics we need to create an execution plan. For that we need two input adapters.


  1. Input adapter - to capture log events coming to BAM (in  this case it will be a wso2event).
  2. Output adapter - to send emails
To create adapters you need to login to BAM management console. Under configurations, there will be "Event Processor Configs" and add input event adapters and output event adapters as shown below.

Input Event adapter


Output Event Adapter


Step 1 - Creating Stream Definitions

Now that we have created event adapters, we need to create a stream definition to capture the LogEvent in order to do complex event processing. In the appserver log event, these are the attributes we have.

  • Meta Data
    • clientType {String}
  • Payload Data
    • tenantID  {String}
    • serverName {String}
    • appName  {String}
    • logTime  {Long}
    • priority {String}
    • message {String}
    • logger {String}
    • ip {String}
    • instance {String}
    • stacktrace {String}
To create the stream definition, go to main tab and under create stream definition you can create the log event stream as shown below.


Step 1 - Creating the Execution Plan

In the execution plan we will be specifying the input stream and writing a CEP Query (SQL Like Query)  for the event stream.

Go to Create Execution Plan, and give suitable name for the execution plan. Select the needed stream and give an alias. Click on import after selecting the stream. In our CEP Query we will analyze events and if an event has an error, we will send it to an output stream.

CEP QUERY

from LogEvents[priority == "ERROR"]
select message,stacktrace,serverName
insert into ExceptionStream


After creating the Query, we need to add the exported stream (which is the stream that we are sending Error logs) according to our CEP querry exported stream name should be ExceptionStream give the value of the exported stream name and select "Create Stream Definition" to create the ExceptionStream. This stream will be auto generated by looking at the CEP as shown below.


Once we create the Exception stream select the Exception stream as exported stream and create a new formatter. This formatter is used to specify the email body and email related information like the subject, to address ect.  Give the output mapping type as text so we can give the content of the email message body inline. 


Email Body

Error Occurred in {{serverName}} – {{message}}
{{stacktrace}} 

In this body we are taking message, stacktrace and server name from the OutputStream (ExceptionStream) and adding a readable message for the email message body.

Add the event formatter and save the execution plan. Now we have successfully created the event trigger to monitory error logs for wso2 appserver. You can test this by invoking a service with an error.

If you want more in depth information on real time log event analytics you can follow the following screen cast for more details

Wednesday, January 15, 2014

[Screencast] Monitoring system logs with WSO2 Business Activity Monitor

This screencast explain how you can do your enterprise monitoring using BAM and CEP features.



Overview of the screencast

  * Send Log Events to BAM/CEP

  * Create analytics to analyse logs daily( can be scheduled accordingly)

  * Create gadgets with  log statistics (charts/tables)

  * Monitor logs real time with CEP (send alerts to relevant parties when an error log appears.)

video


Thursday, January 9, 2014

Eventing using WSO2 DSS - Send data to a JMS Queue for a given event trigger

If you want to trigger an event depend on an SQL related service call this is a great example on how you can do it using WSO2 Data Service Server.

In this sample I am going to explain how you can handle event triggers in WSO2 Data service server.  In WSO2 DSS, you have two types of event triggers.

1. Input event triggers - which trigger's an event looking at the request.
2. Output event triggers - which trigger's an event looking at the response.

For both of these trigger's you can specify an ex-path expression to trigger the event.

To demonstrate this functionality I  am going to write a small data service which insert Student information. And every time an insertion happen, primary key of the inserted record will be send to a JMS Queue.


Prerequisites

In order to try out this sample you need to have
  -> WSO2 Data Services Server Downloaded
  -> You need to have a database of your choice
  -> Add related drivers to DSS_HOME/repository/component/lib folder
  -> ActiveMQ downloaded and started

Step 1 - Enable JMS senders in WSO2 DSS.

In order to send JMS messages from data services, you need to enable the JMS sender in data services server.
To enable JMSSender -> Go to DSS_HOME/repository/conf/axis2/axis2.xml and un-comment the following line.

<transportSender name="jms" class="org.apache.axis2.transport.jms.JMSSender"/>


and then start the data services instance.

Step 2 - Create data Service.

If you are new on creating data services, please refer the following post on how to create a data service using WSO2 Data Service Server.

I have a small table called Student. Lets create a simple data service for the table student.

describe Student;
+---------+-------------+------+-----+---------+----------------+
| Field   | Type        | Null | Key | Default | Extra          |
+---------+-------------+------+-----+---------+----------------+
| sid     | int(11)     | NO   | PRI | NULL    | auto_increment |
| name    | varchar(90) | YES  |     | NULL    |                |
| address | varchar(75) | YES  |     | NULL    |                |
| country | varchar(75) | YES  |     | NULL    |                |
| phone   | varchar(20) | YES  |     | NULL    |                |
| major   | varchar(50) | YES  |     | NULL    |                |
| gpa     | float       | YES  |     | NULL    |                |
| tutorid | int(11)     | YES  |     | NULL    |                |
+---------+-------------+------+-----+---------+----------------+
8 rows in set (0.00 sec)

Log in to DSS management console -> Create New Data Service. Give the data service name appropriately and click on next.

Click on Add Data Source and give database configuration appropriately and click on next.


Next section we will discuss how we can write a data service Query which can invoke a trigger according to the query result.

In this Query my SQL is simple insert

INSERT INTO Student(name,address,country,phone,major,gpa,tutorid) VALUES(:name,:address,:country,:phone,:major,:gpa,:tutorid)

Click on "Generate Input Mappings", and this will generate the input mappings for the input parameters.

Since we need a response back (ideally it will be the primary key of the newly inserted record) we can click  Return generated key which will auto generate the response.




To add the event trigger, scroll down a little bit. Under Events, click on Manage Events. There, we give the eventing configurations.

Event ID - student_addition_trigger

Xpath - //*[local-name()='ID' and namespace-uri()='http://ws.wso2.org/dataservice']>0

Target Topic - student_insertion_topic

Event Sink URL ( JMS URL according to your activeMQ)
jms:/student_insertion_topic?transport.jms.DestinationType=queue&transport.jms.ContentTypeProperty=Content-Type&java.naming.provider.url=tcp://localhost:61616&java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&transport.jms.ConnectionFactoryType=queue&transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory

Go to Main Configuration and add the Output trigger as shown below.


Now we have done with our Query Section. Click on Save to save the Query and Click next to add the  operation.
Add New Opperation ->
Operation Name* - InsertStudent
Query ID* - insertQ

Save and finish.

Now we can test this functionality by using the try it function provided by Data Service Server

My Request


   <body> <p:InsertStudent xmlns:p="http://ws.wso2.org/dataservice">
      <p:name>Amani</p:name>
      <p:address>Soysa</p:address>
      <p:country>SL</p:country>
      <p:phone>1923123111</p:phone>
      <p:major>SE</p:major>
      <p:gpa>4.2</p:gpa>
      <p:tutorid>12</p:tutorid>
   </p:InsertStudent>
</body>

My Response


   <GeneratedKeys xmlns="http://ws.wso2.org/dataservice"> <Entry>
      <ID>5</ID>
   </Entry>
</GeneratedKeys>

If you go to your active MQ console, You would see number of pending messages as shown below. You can go inside your Queue and explore more!!!!





Saturday, October 12, 2013

How to Receive Emails to WSO2 ESB

In one of my previous blogs, I have explained how we can send emails using WSO2 ESB, in this post I am going to explain how we receive emails to WSO2 ESB, so that you can collect data from in coming emails and do some processing and send them to data storage or any other end point.


Before we start ESB server, you need to enable mail transport listeners in ESB. In order to do that you need to go to WSO2_ESB_HOME/repository/conf/axis2/axis2.xml and un comment the following.
<transportReceiver name="mailto" class="org.apache.axis2.transport.mail.MailTransportListener" />

Then start the server.

Creating the proxy service

To create the proxy, go to add -> Proxy service -> Custom Proxy service. There we'l give the name as MyMailProzy.
And you need to give set of service parameters as listed below.

 <parameter name="transport.PollInterval">5</parameter>

   <parameter name="mail.pop3.host">pop.gmail.com</parameter>
   <parameter name="mail.pop3.password">wso2mail</parameter>
   <parameter name="mail.pop3.user">wso2esb.mail</parameter>
   <parameter name="mail.pop3.socketFactory.port">995</parameter>
   <parameter name="transport.mail.ContentType">text/plain</parameter>
   <parameter name="mail.pop3.port">995</parameter>
   <parameter name="mail.pop3.socketFactory.fallback">false</parameter>
   <parameter name="transport.mail.Address">wso2esb.mail@gmail.com</parameter>
   <parameter name="transport.mail.Protocol">pop3</parameter>
   <parameter name="mail.pop3.socketFactory.class">javax.net.ssl.SSLSocketFactory</parameter>

And also under Transport Settings you need to enable mailto transport as shown in the below diagram.

Click on next and we will configure the in sequence.

There we will log the sender's email address by getting the sender email address to a property from the transports (get-property('transport', 'From')) and then using log mediator to log it

<inSequence xmlns="http://ws.apache.org/ns/synapse">

   <property name="senderAddress" expression="get-property('transport', 'From')" scope="default" type="STRING"/>
   <log level="full">
      <property name="Sender Address" expression="get-property('senderAddress')"/>
   </log>
   <drop/>
</inSequence>



Click on next to configure the out sequence.

In the out sequence we will be having a simple send mediator which will send the response back to the client.

The full ESB Configuration

<?xml version="1.0" encoding="UTF-8"?>

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="MyMailProzy"
       transports="mailto"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <property name="senderAddress" expression="get-property('transport', 'From')"/>
         <log level="full">
            <property name="Sender Address" expression="get-property('senderAddress')"/>
         </log>
         <drop/>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </target>
   <parameter name="mail.pop3.host">pop.gmail.com</parameter>
   <parameter name="transport.PollInterval">5</parameter>
   <parameter name="mail.pop3.password">wso2mail</parameter>
   <parameter name="transport.mail.ContentType">text/plain</parameter>
   <parameter name="mail.pop3.socketFactory.port">995</parameter>
   <parameter name="mail.pop3.user">wso2esb.mail</parameter>
   <parameter name="mail.pop3.socketFactory.fallback">false</parameter>
   <parameter name="mail.pop3.port">995</parameter>
   <parameter name="transport.mail.Address">wso2esb.mail@gmail.com</parameter>
   <parameter name="mail.pop3.socketFactory.class">javax.net.ssl.SSLSocketFactory</parameter>
   <parameter name="transport.mail.Protocol">pop3</parameter>
   <description/>
</proxy>
                               

Sending the EMAIL


I have used my gmail to send the email to ESB, my to address will be wso2esb.mail@gmail.com, and you can customize your own subject, and the message body.

Once we send the email, the email will be received to ESB, and you will see the following logs

[2013-10-13 09:06:45,038]  INFO - LogMediator To: , From: mailto:wso2esb.mail@gmail.com, WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: <CAMJOoU667WqnFbx-cCG=hgOy3E2i2qRqKaAUoAcETJE8a-2b+g@mail.gmail.com>, Direction: request, Sender Address = Amani Soysa <amani.soysa@gmail.com>, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">--001a11c233ea18b53d04e8970f15&#xd; Content-Type: text/plain; charset=ISO-8859-1&#xd; &#xd; Sending emails to  WSO2 Enterprise Service Bus!!!!&#xd; &#xd; --001a11c233ea18b53d04e8970f15&#xd; Content-Type: text/html; charset=ISO-8859-1&#xd; Content-Transfer-Encoding: quoted-printable&#xd; &#xd; &lt;div dir=3D"ltr">Sending emails to=A0&lt;span style=3D"color:rgb(68,68,68);fon=&#xd; t-family:arial,sans-serif;line-height:16px" >=A0&lt;/span>&lt;span style=3D"font-w=&#xd;eight:bold;color:rgb(68,68,68);font-family:arial,sans-serif;line-height:16p=&#xd;x" >WSO2 Enterprise Service Bus!!!!&lt;/span>=A0&lt;/div>&#xd;&#xd; --001a11c233ea18b53d04e8970f15--&#xd; </text></soapenv:Body></soapenv:Envelope>

How to call multiple soap operations using call out mediator - WSO2 ESB

In this post I will explain how we can call multiple operations inside the same proxy service using the  call out mediator in WSO2 ESB. For this sample I have used my distributed transaction sample where we need to call multiple operation in the same transaction.

In order to do this you need to have WSO2 ESB, as well as sample web service with multiple operations.  For that I will be using WSO2 DSS distributed transaction data service which I have explained in my previous blog post. Where we will insert customer information, in to multiple data bases in the same transaction.

Sample input 

<customer>
  <id>2</id>
  <name>Smith</name>
</customer>

Step 1 - Creating the proxy service.

To create the proxy, you need to start the ESB server under services -> add -> proxy service -> Custom Proxy service. Give an appropriate name. For this sample I am calling it MyTransactionProxy.


In the in sequence I will be calling several mediators.

  1. Log Mediator - To log the incoming request
  2. Property Mediators - To store the input parameters
  3. Payload Factory Mediator - to arrange the payload for each operation
  4. Call out mediator  - to call each operation of the given service 


Sample In sequence

<inSequence xmlns="http://ws.apache.org/ns/synapse">
   <log level="full">
      <property name="M1" value="***************HITTING Transaction PROXY****************"/>
   </log>
   <property name="OUT_ONLY" value="true"/>
   <property name="id" expression="//id/text()"/>
   <property name="name" expression="//name/text()"/>
   <payloadFactory media-type="xml">
      <format>
         <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:dat="http://ws.wso2.org/dataservice">            
            <soapenv:Header/>            
            <soapenv:Body/>      
         </soapenv:Envelope>
      </format>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:begin_boxcar">
      <source type="envelope"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
   <payloadFactory media-type="xml">
      <format>
         <p:my_insert xmlns:p="http://ws.wso2.org/dataservice">          
            <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>          
            <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>      
         </p:my_insert>
      </format>
      <args>
         <arg expression="get-property('id')" evaluator="xml"/>
         <arg expression="get-property('name')" evaluator="xml"/>
      </args>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:my_insert">
      <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
   <payloadFactory media-type="xml">
      <format>
         <p:pos_insert xmlns:p="http://ws.wso2.org/dataservice">          
            <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>          
            <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>      
         </p:pos_insert>
      </format>
      <args>
         <arg expression="get-property('id')" evaluator="xml"/>
         <arg expression="get-property('name')" evaluator="xml"/>
      </args>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:pos_insert">
      <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
   <payloadFactory media-type="xml">
      <format>
         <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:dat="http://ws.wso2.org/dataservice">          
            <soapenv:Header/>          
            <soapenv:Body/>      
         </soapenv:Envelope>
      </format>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:end_boxcar">
      <source type="envelope"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
</inSequence>


Here I have created payload factory mediator and a call out mediator to call each operation. In the call our service URL I have given the end point for my service. For example I have an operation called my_insert. So I created a payload factory mediator and defined the payload format inline as shown below (parameters for the operation is given as $1,$2 which are later replaced by the id, and the name properties.

<payloadFactory xmlns="http://ws.apache.org/ns/synapse" media-type="xml">
   <format>
      <p:my_insert xmlns:p="http://ws.wso2.org/dataservice">                                                
         <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>                                                
         <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>                                  
      </p:my_insert>
   </format>
   <args>
      <arg expression="get-property('id')" evaluator="xml"/>
      <arg expression="get-property('name')" evaluator="xml"/>
   </args>
</payloadFactory>

Likewise, I have called each operation with the same format.

In the out sequence I have used an empty send mediator which will send back the response to the same service.

Complete proxy xml is attached below.

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="TransactionProxy"
       transports="http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <log level="full">
            <property name="M1"
                      value="*************HITTING Transaction PROXY*************"/>
         </log>
         <property name="OUT_ONLY" value="true"/>
         <property name="id" expression="//id/text()"/>
         <property name="name" expression="//name/text()"/>
         <payloadFactory media-type="xml">
            <format>
               <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                                 xmlns:dat="http://ws.wso2.org/dataservice">
                  <soapenv:Header/>
                  <soapenv:Body/>
               </soapenv:Envelope>
            </format>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:begin_boxcar">
            <source type="envelope"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <payloadFactory media-type="xml">
            <format>
               <p:my_insert xmlns:p="http://ws.wso2.org/dataservice">
                  <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>
                  <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>
               </p:my_insert>
            </format>
            <args>
               <arg expression="get-property('id')" evaluator="xml"/>
               <arg expression="get-property('name')" evaluator="xml"/>
            </args>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:my_insert">
            <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <payloadFactory media-type="xml">
            <format>
               <p:pos_insert xmlns:p="http://ws.wso2.org/dataservice">
                  <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>
                  <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>
               </p:pos_insert>
            </format>
            <args>
               <arg expression="get-property('id')" evaluator="xml"/>
               <arg expression="get-property('name')" evaluator="xml"/>
            </args>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:pos_insert">
            <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <payloadFactory media-type="xml">
            <format>
               <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                                 xmlns:dat="http://ws.wso2.org/dataservice">
                  <soapenv:Header/>
                  <soapenv:Body/>
               </soapenv:Envelope>
            </format>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:end_boxcar">
            <source type="envelope"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </target>
   <description/>
</proxy>
                               

Calling the proxy service

Here I am calling the proxy service using the curl tool.

curl -v --request POST -d '2Smith' -H Content-Type:"text/xml" http://amani-ThinkPad-T520:8280/services/MyTransactionProxy




Wednesday, October 9, 2013

Distributed Transaction with WSO2 Data Services

What is distributed transaction ?

Distributed database transaction means executing multiple related actions/operations in a coordinated way. This is also known as  global transaction. Distributed transaction can occur in the same database level however, in most cases distributed transaction happens in different databases (typically different RDBMS types) and often in different locations. Distributed transactions are often described as ACID -- atomic, consistent, isolated, and durable.  Meaning changes made to the database during the transactions are tentative, if any of the operation fails then none of the other changes will get affected. In a typical distributed transaction, you have to make sure if one operation fails the whatever previously executed operation should roll back, undoing all the changes as if the transactions never took place. Even if your application crashes in the middle of the transaction, when it restarts, transaction recovery should roll back the open transaction.



A typical Distributed transaction example would be moving data from one data base to another database. For example lets say you want to delete customer from one location and add that same customer to the another location.You would not want either transaction committed without assurance that both will complete successfully. Therefore, for these kind of instances its important to have distributed transaction feature.

The above transactions involve the following steps:
  1. Begin a transaction. 
  2. Add Customer
  3. Delete Customer
  4. End transaction

Distributed Transaction with WSO2 Data Services Server

WSO2 data services server provide distributed transaction using Java Transaction API (JTA) which enables global level transaction across multiple X/Open XA resources in java environment.

When you use XA functionality, the transaction manager uses XA resource instances to prepare and coordinate each transaction branch and then to commit or roll back all each of individual transaction appropriately.

For each RDBMS type there is a specific XA-Datasource class and set of configuration properties. Therefore you need to know the XA-Datasource class and their configurations before creating the data service.

Lets see how we can create a data service for the above transaction.

For this example I will have two databases in postgres and mysql. For this demo I will be using a very simple customer table which has id and name and we will see how we can inset values to these two tables in each database in a coordinated manner

=========================== SQL script ==============================
CREATE TABLE customer (
cust_id int NOT NULL,
name varchar(255) NOT NULL,
PRIMARY KEY (cust_id)
)

========================data service configuration file======================

<data disableStreaming="true" enableBoxcarring="true" enableDTP="true" name="DTPDS">
   <config id="pos_ds">
      <property name="org.wso2.ws.dataservice.xa_datasource_class">org.postgresql.xa.PGXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="ServerName">localhost</property>
         <property name="PortNumber">5432</property>
         <property name="DatabaseName">MyDB</property>
         <property name="User">postgres</property>
         <property name="Password">root</property>
      </property>
   </config>
   <config id="my_ds">
     <property name="org.wso2.ws.dataservice.xa_datasource_class">com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="URL">jdbc:mysql://localhost:3306/MyDB</property>
         <property name="User">root</property>
         <property name="Password">root</property>
      </property>
   </config>
   <query id="pos_q" useConfig="pos_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <query id="my_q" useConfig="my_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <operation disableStreaming="true" name="pos_insert" returnRequestStatus="true">
      <call-query href="pos_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
   <operation disableStreaming="true" name="my_insert" returnRequestStatus="true">
      <call-query href="my_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
</data>

You need to add the configuration file to WSO2DS_HOME/repository/deployment/server/dataservices in order to deploy this file

Step by step explanation

Data source configuration

I have created two data source configuration in the data service descriptor file
 <config id="pos_ds">
      <property name="org.wso2.ws.dataservice.xa_datasource_class">org.postgresql.xa.PGXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="ServerName">localhost</property>
         <property name="PortNumber">5432</property>
         <property name="DatabaseName">MyDB</property>
         <property name="User">postgres</property>
         <property name="Password">root</property>
      </property>
   </config>
   <config id="my_ds">
     <property name="org.wso2.ws.dataservice.xa_datasource_class">com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="URL">jdbc:mysql://localhost:3306/MyDB</property>
         <property name="User">root</property>
         <property name="Password">root</property>
      </property>
   </config>
There we have specified the XA datasource classes "org.wso2.ws.dataservice.xa_datasource_class" in a property along with it's parameters. For each XA datasource class configuration properties may differ.

Query Configuration

  <query id="pos_q" useConfig="pos_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <query id="my_q" useConfig="my_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>

I have created two queries pointing each data source, and also two distinct opperations mapping to each query.

<operation disableStreaming="true" name="pos_insert" returnRequestStatus="true">
      <call-query href="pos_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
   <operation disableStreaming="true" name="my_insert" returnRequestStatus="true">
      <call-query href="my_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>

After you deploy this data service. You will see the deployed data service under the data service list. You can invoke this service using the try-it tool (or using your own class)

Invoke the operations in this order

  1. begin_boxcar 
  2. my_insert
  3. pos_insert
  4. end_boxcar

Please make sure you have set the  "max_prepared_transactions" to a non zero value in "/etc/postgres/postgres.conf in oorder this sample to work

In my next post, I will be explaining how we can call all these four operations using a single proxy service with the use of of WSO2 ESB.

Thursday, July 25, 2013

Configure HTTP Access Logging in WSO2 products


If you want to analyze your application's usage activities such as who is accessing your page, number of hits, errors ect, you can use the HTTP access log files monitor and analyze the above measures. WSO2 products uses embedded tomcat as it's primary runtime therefore, you can use apache tomcat access logger to monitor your activity and performance of the server as well as any errors that may be occurring. This post explain how you can configure HTTP access logging in WSO2 products in order to get efficient monitoring.


In WSO2 products you can customize the http access log configuration by editing the catalina-server.xml which is located {CARBON_HOME}/repository/conf/tomcat directory, which is the server descripter file for the embedded tomcat integration. In the catalina-server.xml, under valves you have the HTTP access log configuration as shown below
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="${carbon.home}/repository/logs"
               prefix="http_access_" suffix=".log"
               pattern="combined" />

The server access log records all requests processed by the server. You can modify the what to log and what not to log by customizing the pattern attribute.
In the pattern attribute we can define formatting layouts. A formatting layout identifying the various information fields from the request and response to be logged, or the word "common" or "combined" to select a standard format.
Values for the pattern attribute are made up of literal text strings, combined with pattern identifiers prefixed by the "%" character to cause replacement by the corresponding variable value from the current request and response. The following pattern codes are supported:

%a - Remote IP address
%A - Local IP address
%b - Bytes sent, excluding HTTP headers, or '-' if no bytes were sent
%B - Bytes sent, excluding HTTP headers
%h - Remote host name
%H - Request protocol
%l - Remote logical username from identd (always returns '-')
%m - Request method
%p - Local port
%q - Query string (prepended with a '?' if it exists, otherwise an empty string
%r - First line of the request
%s - HTTP status code of the response
%S - User session ID
%t - Date and time, in Common Log Format format
%u - Remote user that was authenticated
%U - Requested URL path
%v - Local server name
%D - Time taken to process the request, in millis
%T - Time taken to process the request, in seconds
%I - current request thread name (can compare later with stacktraces) 


In addition, the caller can specify one of the following aliases for commonly utilized patterns:

common - %h %l %u %t "%r" %s %b
combined - %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i"

Please note that the optimized access does only support common and combined as the value for this attribute.

If you want to change the format you can modify the as shown below

 <Valve className="org.apache.catalina.valves.AccessLogValve" directory="${carbon.home}/repository/logs"
               prefix="http_access_" suffix=".log"
               pattern="%h %l %u %t '%r' %s %b" />



You can further change the log file location by changing the directory, prefix is the name of the log file. By default logs are rotated daily and the date is appended to the log file name. However, you can disable the log rotation by adding the following attribute rotatable="false". If you want to not have the current date appended in the log file name however, need the date to the rotated log file you can add renameOnRotate="true" attribute the the valve configuration. Please refer Access Log Valve Attributes for the full list of supported attributes.