Thing which seemed very Thingish inside you is quite different when it gets out into the open and has other people looking at it

Saturday, October 12, 2013

How to Receive Emails to WSO2 ESB

In one of my previous blogs, I have explained how we can send emails using WSO2 ESB, in this post I am going to explain how we receive emails to WSO2 ESB, so that you can collect data from in coming emails and do some processing and send them to data storage or any other end point.


Before we start ESB server, you need to enable mail transport listeners in ESB. In order to do that you need to go to WSO2_ESB_HOME/repository/conf/axis2/axis2.xml and un comment the following.
<transportReceiver name="mailto" class="org.apache.axis2.transport.mail.MailTransportListener" />

Then start the server.

Creating the proxy service

To create the proxy, go to add -> Proxy service -> Custom Proxy service. There we'l give the name as MyMailProzy.
And you need to give set of service parameters as listed below.

 <parameter name="transport.PollInterval">5</parameter>

   <parameter name="mail.pop3.host">pop.gmail.com</parameter>
   <parameter name="mail.pop3.password">wso2mail</parameter>
   <parameter name="mail.pop3.user">wso2esb.mail</parameter>
   <parameter name="mail.pop3.socketFactory.port">995</parameter>
   <parameter name="transport.mail.ContentType">text/plain</parameter>
   <parameter name="mail.pop3.port">995</parameter>
   <parameter name="mail.pop3.socketFactory.fallback">false</parameter>
   <parameter name="transport.mail.Address">wso2esb.mail@gmail.com</parameter>
   <parameter name="transport.mail.Protocol">pop3</parameter>
   <parameter name="mail.pop3.socketFactory.class">javax.net.ssl.SSLSocketFactory</parameter>

And also under Transport Settings you need to enable mailto transport as shown in the below diagram.

Click on next and we will configure the in sequence.

There we will log the sender's email address by getting the sender email address to a property from the transports (get-property('transport', 'From')) and then using log mediator to log it

<inSequence xmlns="http://ws.apache.org/ns/synapse">

   <property name="senderAddress" expression="get-property('transport', 'From')" scope="default" type="STRING"/>
   <log level="full">
      <property name="Sender Address" expression="get-property('senderAddress')"/>
   </log>
   <drop/>
</inSequence>



Click on next to configure the out sequence.

In the out sequence we will be having a simple send mediator which will send the response back to the client.

The full ESB Configuration

<?xml version="1.0" encoding="UTF-8"?>

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="MyMailProzy"
       transports="mailto"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <property name="senderAddress" expression="get-property('transport', 'From')"/>
         <log level="full">
            <property name="Sender Address" expression="get-property('senderAddress')"/>
         </log>
         <drop/>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </target>
   <parameter name="mail.pop3.host">pop.gmail.com</parameter>
   <parameter name="transport.PollInterval">5</parameter>
   <parameter name="mail.pop3.password">wso2mail</parameter>
   <parameter name="transport.mail.ContentType">text/plain</parameter>
   <parameter name="mail.pop3.socketFactory.port">995</parameter>
   <parameter name="mail.pop3.user">wso2esb.mail</parameter>
   <parameter name="mail.pop3.socketFactory.fallback">false</parameter>
   <parameter name="mail.pop3.port">995</parameter>
   <parameter name="transport.mail.Address">wso2esb.mail@gmail.com</parameter>
   <parameter name="mail.pop3.socketFactory.class">javax.net.ssl.SSLSocketFactory</parameter>
   <parameter name="transport.mail.Protocol">pop3</parameter>
   <description/>
</proxy>
                               

Sending the EMAIL


I have used my gmail to send the email to ESB, my to address will be wso2esb.mail@gmail.com, and you can customize your own subject, and the message body.

Once we send the email, the email will be received to ESB, and you will see the following logs

[2013-10-13 09:06:45,038]  INFO - LogMediator To: , From: mailto:wso2esb.mail@gmail.com, WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: <CAMJOoU667WqnFbx-cCG=hgOy3E2i2qRqKaAUoAcETJE8a-2b+g@mail.gmail.com>, Direction: request, Sender Address = Amani Soysa <amani.soysa@gmail.com>, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">--001a11c233ea18b53d04e8970f15&#xd; Content-Type: text/plain; charset=ISO-8859-1&#xd; &#xd; Sending emails to  WSO2 Enterprise Service Bus!!!!&#xd; &#xd; --001a11c233ea18b53d04e8970f15&#xd; Content-Type: text/html; charset=ISO-8859-1&#xd; Content-Transfer-Encoding: quoted-printable&#xd; &#xd; &lt;div dir=3D"ltr">Sending emails to=A0&lt;span style=3D"color:rgb(68,68,68);fon=&#xd; t-family:arial,sans-serif;line-height:16px" >=A0&lt;/span>&lt;span style=3D"font-w=&#xd;eight:bold;color:rgb(68,68,68);font-family:arial,sans-serif;line-height:16p=&#xd;x" >WSO2 Enterprise Service Bus!!!!&lt;/span>=A0&lt;/div>&#xd;&#xd; --001a11c233ea18b53d04e8970f15--&#xd; </text></soapenv:Body></soapenv:Envelope>

How to call multiple soap operations using call out mediator - WSO2 ESB

In this post I will explain how we can call multiple operations inside the same proxy service using the  call out mediator in WSO2 ESB. For this sample I have used my distributed transaction sample where we need to call multiple operation in the same transaction.

In order to do this you need to have WSO2 ESB, as well as sample web service with multiple operations.  For that I will be using WSO2 DSS distributed transaction data service which I have explained in my previous blog post. Where we will insert customer information, in to multiple data bases in the same transaction.

Sample input 

<customer>
  <id>2</id>
  <name>Smith</name>
</customer>

Step 1 - Creating the proxy service.

To create the proxy, you need to start the ESB server under services -> add -> proxy service -> Custom Proxy service. Give an appropriate name. For this sample I am calling it MyTransactionProxy.


In the in sequence I will be calling several mediators.

  1. Log Mediator - To log the incoming request
  2. Property Mediators - To store the input parameters
  3. Payload Factory Mediator - to arrange the payload for each operation
  4. Call out mediator  - to call each operation of the given service 


Sample In sequence

<inSequence xmlns="http://ws.apache.org/ns/synapse">
   <log level="full">
      <property name="M1" value="***************HITTING Transaction PROXY****************"/>
   </log>
   <property name="OUT_ONLY" value="true"/>
   <property name="id" expression="//id/text()"/>
   <property name="name" expression="//name/text()"/>
   <payloadFactory media-type="xml">
      <format>
         <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:dat="http://ws.wso2.org/dataservice">            
            <soapenv:Header/>            
            <soapenv:Body/>      
         </soapenv:Envelope>
      </format>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:begin_boxcar">
      <source type="envelope"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
   <payloadFactory media-type="xml">
      <format>
         <p:my_insert xmlns:p="http://ws.wso2.org/dataservice">          
            <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>          
            <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>      
         </p:my_insert>
      </format>
      <args>
         <arg expression="get-property('id')" evaluator="xml"/>
         <arg expression="get-property('name')" evaluator="xml"/>
      </args>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:my_insert">
      <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
   <payloadFactory media-type="xml">
      <format>
         <p:pos_insert xmlns:p="http://ws.wso2.org/dataservice">          
            <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>          
            <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>      
         </p:pos_insert>
      </format>
      <args>
         <arg expression="get-property('id')" evaluator="xml"/>
         <arg expression="get-property('name')" evaluator="xml"/>
      </args>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:pos_insert">
      <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
   <payloadFactory media-type="xml">
      <format>
         <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:dat="http://ws.wso2.org/dataservice">          
            <soapenv:Header/>          
            <soapenv:Body/>      
         </soapenv:Envelope>
      </format>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:end_boxcar">
      <source type="envelope"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
</inSequence>


Here I have created payload factory mediator and a call out mediator to call each operation. In the call our service URL I have given the end point for my service. For example I have an operation called my_insert. So I created a payload factory mediator and defined the payload format inline as shown below (parameters for the operation is given as $1,$2 which are later replaced by the id, and the name properties.

<payloadFactory xmlns="http://ws.apache.org/ns/synapse" media-type="xml">
   <format>
      <p:my_insert xmlns:p="http://ws.wso2.org/dataservice">                                                
         <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>                                                
         <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>                                  
      </p:my_insert>
   </format>
   <args>
      <arg expression="get-property('id')" evaluator="xml"/>
      <arg expression="get-property('name')" evaluator="xml"/>
   </args>
</payloadFactory>

Likewise, I have called each operation with the same format.

In the out sequence I have used an empty send mediator which will send back the response to the same service.

Complete proxy xml is attached below.

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="TransactionProxy"
       transports="http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <log level="full">
            <property name="M1"
                      value="*************HITTING Transaction PROXY*************"/>
         </log>
         <property name="OUT_ONLY" value="true"/>
         <property name="id" expression="//id/text()"/>
         <property name="name" expression="//name/text()"/>
         <payloadFactory media-type="xml">
            <format>
               <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                                 xmlns:dat="http://ws.wso2.org/dataservice">
                  <soapenv:Header/>
                  <soapenv:Body/>
               </soapenv:Envelope>
            </format>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:begin_boxcar">
            <source type="envelope"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <payloadFactory media-type="xml">
            <format>
               <p:my_insert xmlns:p="http://ws.wso2.org/dataservice">
                  <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>
                  <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>
               </p:my_insert>
            </format>
            <args>
               <arg expression="get-property('id')" evaluator="xml"/>
               <arg expression="get-property('name')" evaluator="xml"/>
            </args>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:my_insert">
            <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <payloadFactory media-type="xml">
            <format>
               <p:pos_insert xmlns:p="http://ws.wso2.org/dataservice">
                  <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>
                  <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>
               </p:pos_insert>
            </format>
            <args>
               <arg expression="get-property('id')" evaluator="xml"/>
               <arg expression="get-property('name')" evaluator="xml"/>
            </args>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:pos_insert">
            <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <payloadFactory media-type="xml">
            <format>
               <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                                 xmlns:dat="http://ws.wso2.org/dataservice">
                  <soapenv:Header/>
                  <soapenv:Body/>
               </soapenv:Envelope>
            </format>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:end_boxcar">
            <source type="envelope"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </target>
   <description/>
</proxy>
                               

Calling the proxy service

Here I am calling the proxy service using the curl tool.

curl -v --request POST -d '2Smith' -H Content-Type:"text/xml" http://amani-ThinkPad-T520:8280/services/MyTransactionProxy




Wednesday, October 9, 2013

Distributed Transaction with WSO2 Data Services

What is distributed transaction ?

Distributed database transaction means executing multiple related actions/operations in a coordinated way. This is also known as  global transaction. Distributed transaction can occur in the same database level however, in most cases distributed transaction happens in different databases (typically different RDBMS types) and often in different locations. Distributed transactions are often described as ACID -- atomic, consistent, isolated, and durable.  Meaning changes made to the database during the transactions are tentative, if any of the operation fails then none of the other changes will get affected. In a typical distributed transaction, you have to make sure if one operation fails the whatever previously executed operation should roll back, undoing all the changes as if the transactions never took place. Even if your application crashes in the middle of the transaction, when it restarts, transaction recovery should roll back the open transaction.



A typical Distributed transaction example would be moving data from one data base to another database. For example lets say you want to delete customer from one location and add that same customer to the another location.You would not want either transaction committed without assurance that both will complete successfully. Therefore, for these kind of instances its important to have distributed transaction feature.

The above transactions involve the following steps:
  1. Begin a transaction. 
  2. Add Customer
  3. Delete Customer
  4. End transaction

Distributed Transaction with WSO2 Data Services Server

WSO2 data services server provide distributed transaction using Java Transaction API (JTA) which enables global level transaction across multiple X/Open XA resources in java environment.

When you use XA functionality, the transaction manager uses XA resource instances to prepare and coordinate each transaction branch and then to commit or roll back all each of individual transaction appropriately.

For each RDBMS type there is a specific XA-Datasource class and set of configuration properties. Therefore you need to know the XA-Datasource class and their configurations before creating the data service.

Lets see how we can create a data service for the above transaction.

For this example I will have two databases in postgres and mysql. For this demo I will be using a very simple customer table which has id and name and we will see how we can inset values to these two tables in each database in a coordinated manner

=========================== SQL script ==============================
CREATE TABLE customer (
cust_id int NOT NULL,
name varchar(255) NOT NULL,
PRIMARY KEY (cust_id)
)

========================data service configuration file======================

<data disableStreaming="true" enableBoxcarring="true" enableDTP="true" name="DTPDS">
   <config id="pos_ds">
      <property name="org.wso2.ws.dataservice.xa_datasource_class">org.postgresql.xa.PGXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="ServerName">localhost</property>
         <property name="PortNumber">5432</property>
         <property name="DatabaseName">MyDB</property>
         <property name="User">postgres</property>
         <property name="Password">root</property>
      </property>
   </config>
   <config id="my_ds">
     <property name="org.wso2.ws.dataservice.xa_datasource_class">com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="URL">jdbc:mysql://localhost:3306/MyDB</property>
         <property name="User">root</property>
         <property name="Password">root</property>
      </property>
   </config>
   <query id="pos_q" useConfig="pos_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <query id="my_q" useConfig="my_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <operation disableStreaming="true" name="pos_insert" returnRequestStatus="true">
      <call-query href="pos_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
   <operation disableStreaming="true" name="my_insert" returnRequestStatus="true">
      <call-query href="my_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
</data>

You need to add the configuration file to WSO2DS_HOME/repository/deployment/server/dataservices in order to deploy this file

Step by step explanation

Data source configuration

I have created two data source configuration in the data service descriptor file
 <config id="pos_ds">
      <property name="org.wso2.ws.dataservice.xa_datasource_class">org.postgresql.xa.PGXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="ServerName">localhost</property>
         <property name="PortNumber">5432</property>
         <property name="DatabaseName">MyDB</property>
         <property name="User">postgres</property>
         <property name="Password">root</property>
      </property>
   </config>
   <config id="my_ds">
     <property name="org.wso2.ws.dataservice.xa_datasource_class">com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="URL">jdbc:mysql://localhost:3306/MyDB</property>
         <property name="User">root</property>
         <property name="Password">root</property>
      </property>
   </config>
There we have specified the XA datasource classes "org.wso2.ws.dataservice.xa_datasource_class" in a property along with it's parameters. For each XA datasource class configuration properties may differ.

Query Configuration

  <query id="pos_q" useConfig="pos_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <query id="my_q" useConfig="my_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>

I have created two queries pointing each data source, and also two distinct opperations mapping to each query.

<operation disableStreaming="true" name="pos_insert" returnRequestStatus="true">
      <call-query href="pos_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
   <operation disableStreaming="true" name="my_insert" returnRequestStatus="true">
      <call-query href="my_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>

After you deploy this data service. You will see the deployed data service under the data service list. You can invoke this service using the try-it tool (or using your own class)

Invoke the operations in this order

  1. begin_boxcar 
  2. my_insert
  3. pos_insert
  4. end_boxcar

Please make sure you have set the  "max_prepared_transactions" to a non zero value in "/etc/postgres/postgres.conf in oorder this sample to work

In my next post, I will be explaining how we can call all these four operations using a single proxy service with the use of of WSO2 ESB.