tag:blogger.com,1999:blog-25923811821456063442024-03-13T21:21:01.495+01:00Maxence's technical cornerYou can find some articles about Java EE, WebLogic server but also about my experimentations about stuff I want to discoverMaxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.comBlogger63125tag:blogger.com,1999:blog-2592381182145606344.post-45886326188805903972019-04-28T19:24:00.000+02:002019-04-28T19:24:00.315+02:00Server-Sent Events with Quarkus and Kafka
<br/><div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-TQhK_u_YS20/XMVxkUySwSI/AAAAAAABf4Q/QzOGUNgQiwsNIXgRWcjfFoLbFNh2vMjhwCLcBGAs/s1600/article.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-TQhK_u_YS20/XMVxkUySwSI/AAAAAAABf4Q/QzOGUNgQiwsNIXgRWcjfFoLbFNh2vMjhwCLcBGAs/s1600/article.png" data-original-width="1600" data-original-height="320" height="80%" width="80%" /></a></div><br/>
<h1>Context</h1><br/>
I wanted to play with <a href="https://quarkus.io/">Quarkus</a> the day it came out. Unfortunately, I had other matters to deal with and today is the first day I can sit and concentrate :)<br/>
In this article, I will demonstrate a use case where I want to display on a webpage the events received on a Kafka topic, dynamically.<br/><br/>
<h1>Configuring the components</h1><br/>
<h2>Kafka</h2><br/>
That part is really easy.<br/>
You will need to run Kafka. If you already have a setup, that's perfect : launch Kafka the way you usually do<br/>
Else, I strongly encourage you to use the <a href="https://www.confluent.io/download/">Confluent Platform</a> (the most recent version, at the time of writing, is the 5.2.1).<br/>
Then, just type <b>bin/confluent start</b> and Kafka (+ the whole stack around it) will be up in seconds.<br/><br/>
Once Kafka is started, we will create a topic named <b>reactive</b> :
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
./kafka-topics --create --zookeeper localhost:2181 --partitions 3 --replication-factor 1 --topic <b>reactive</b>
</pre>
</div>
<h2>Quarkus</h2><br/>
In order to be able to run Quarkus, you will have to follow the prerequisites on this page :<br/>
<a href="https://quarkus.io/guides/getting-started-guide">https://quarkus.io/guides/getting-started-guide</a><br/><br/>
Quarkus doesn't have to be installed <i>stricto sensu</i> : the whole configuration will be done through the application (<b>POM</b>, <b>Maven plugin</b> and <b>application.properties</b>).<br/><br/>
<h2>NetBeans</h2><br/>
I'm also using the excellent NetBeans 11 (<a href="https://www.theserverside.com/news/252462360/NetBeans-Java-IDE-becomes-top-level-Apache-project">which is now a top-level Apache project !</a>), available at Apache :<br/>
<a href="https://netbeans.apache.org/download/nb110/nb110.html">https://netbeans.apache.org/download/nb110/nb110.html</a><br/><br/>
But there are a few things to know about NetBeans to be able to run this app smoothly.<br/>
Make sure that the Maven version you're using is not the bundled one (3.3.9) : to be compatible with Quarkus, you'll have to run a Maven 3.5.3 or later.<br/>
You can define it in <b>Options > Java > Maven > Maven Home</b><br/><br/>
And in order to be able to run Quarkus, you will have to tell NetBeans how to do so.<br/>
Right click on your project and select <b>"Properties"</b> then select <b>"Actions"</b> and click <b>"Add Custom"</b> and name it <b>"Quarkus Run"</b><br/>
In <b>"Execute Goals"</b>, type : <b>"compile quarkus:dev"</b> and click <b>"OK"</b>.<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-ji42R3B-Mx8/XMTOes6mB9I/AAAAAAABf3U/_aZkoTSY2O8EgzhMxvQmGM19xm3LvtKTQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2019-04-27%2Ba%25CC%2580%2B17.12.33.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-ji42R3B-Mx8/XMTOes6mB9I/AAAAAAABf3U/_aZkoTSY2O8EgzhMxvQmGM19xm3LvtKTQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2019-04-27%2Ba%25CC%2580%2B17.12.33.png" data-original-width="1600" data-original-height="1216" height="60%" width="60%" /></a></div><br/><br/>
Now, when you want to launch Quarkus, right click on your project and select <b>"Run Maven > Quarkus Run"</b><br/><br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-JD7NafCH-ZU/XMTO5ClPqjI/AAAAAAABf3c/HKXlOLc3_dE9T1UtmEfQzt6IJ4pZ1-ymwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2019-04-27%2Ba%25CC%2580%2B17.21.08.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-JD7NafCH-ZU/XMTO5ClPqjI/AAAAAAABf3c/HKXlOLc3_dE9T1UtmEfQzt6IJ4pZ1-ymwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2019-04-27%2Ba%25CC%2580%2B17.21.08.png" data-original-width="970" data-original-height="680" height="60%" width="60%"/></a></div><br/><br/>
<h1>Designing the application</h1><br/>
One thing I like about Quarkus is that it has been designed by developers who know what they are talking about. <br/>
For instance, some very nice improvements :
<ul>
<li>you don't have to ship an empty <b>beans.xml</b> to activate CDI</li>
<li>no need to declare an empty class extending <b>javax.ws.rs.core.Application</b> to be able to design JAX-RS WS</li>
<li>the time when you had to write an infinite loop to get messages is over</li>
</ul><br/>
To create a starter project, just follow the instructions provided on the <a href="https://quarkus.io/get-started/">Quarkus "Getting started" page</a>.
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
mvn io.quarkus:quarkus-maven-plugin:0.14.0:create \
-DprojectGroupId=fr.mbutton.blog \
-DprojectArtifactId=quarkus-kafka \
-DclassName="fr.mbutton.blog.KafkaResource" \
-Dpath="/consume"
</pre>
</div>
But some dependencies are missing. Here is the <i>dependency</i> section of my POM.
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
<dependencies>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-resteasy</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-smallrye-reactive-messaging-kafka</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-vertx</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-arc</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
</dependencies>
</pre>
</div>
The file <b>application.properties</b> is the file where the application is truly configured.<br/>
For instance, you will find the log configuration, as well as the Kafka bootstrap properties.
<div class="custom-html-block">
<pre class="prettyprint lang-properties">
smallrye.messaging.source.events.type=io.smallrye.reactive.messaging.kafka.Kafka
smallrye.messaging.source.events.topic=reactive
smallrye.messaging.source.events.bootstrap.servers=localhost:9092
smallrye.messaging.source.events.key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
smallrye.messaging.source.events.value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
smallrye.messaging.source.events.group.id=shipment-service-quarkus
quarkus.log.console.enable=true
quarkus.log.console.format=%d{HH:mm:ss} %-5p [%c{2.}]] (%t) %s%e%n
quarkus.log.console.level=DEBUG
quarkus.log.console.color=false
quarkus.log.category."fr.mbutton.blog".level=DEBUG
</pre>
</div>
And here is the code of the webservice :
<div class="custom-html-block">
<pre class="prettyprint lang-java">
package fr.mbutton.blog;
import io.smallrye.reactive.messaging.kafka.KafkaMessage;
import java.io.IOException;
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.time.LocalTime;
import java.util.concurrent.CompletionStage;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.sse.OutboundSseEvent;
import javax.ws.rs.sse.Sse;
import javax.ws.rs.sse.SseEventSink;
import org.eclipse.microprofile.reactive.messaging.Incoming;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@Path("/consume")
public class KafkaResource {
private OutboundSseEvent.Builder eventBuilder;
private Sse sse;
private SseEventSink sseEventSink = null;
private Logger logger = LoggerFactory.getLogger(this.getClass());
@Context
public void setSse (Sse sse) {
this.sse = sse;
this.eventBuilder = sse.newEventBuilder();
}
@GET
@Produces(MediaType.SERVER_SENT_EVENTS)
public void consume (@Context SseEventSink sseEventSink) {
this.sseEventSink = sseEventSink;
}
@Incoming("events")
public CompletionStage<Void> onMessage (KafkaMessage<String, String> message) throws IOException {
logger.debug("Message with content \"{}\" has been received.", message.getPayload());
if (sseEventSink != null) {
String display = String.valueOf(LocalTime.now().toString()) + " | " + getHost() + " | " + message.getPayload();
OutboundSseEvent sseEvent = this.eventBuilder
.name("message")
.id(message.getKey())
.mediaType(MediaType.TEXT_PLAIN_TYPE)
.data(display)
.reconnectDelay(3000)
.comment(message.getPayload())
.build();
sseEventSink.send(sseEvent);
}
return message.ack();
}
private String getHost () {
try {
return InetAddress.getLocalHost().getHostName();
} catch (UnknownHostException ex) {
logger.error("Problem while trying to get the hostname", ex);
return "Unknown";
}
}
}
</pre>
</div>
The mechanism is fairly simple : when loaded, thanks to the annotation <b>"Incoming"</b>, the class binds to the destination <b>"events"</b> whose configuration lies in the <b>"application.properties"</b>.<br/>
It's very important because that's here where the link between the destination known by the application and the Kafka topic is made.<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-properties">
[•••]
smallrye.messaging.source.<b>events</b>.topic=<b>reactive</b>
[•••]
</pre>
</div>
Then, when a GET request is received on path <b>/consume</b>, it initializes the SseEventSink.<br/>
Later, when Kafka messages are received, they are sent onto it.<br/><br/>
<h1>Sending events</h1><br/>
Just use one of the Kafka CLI tools, to send messages :<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
./kafka-console-producer --broker-list localhost:9092 --topic <b>reactive</b>
>test
>yet another test
</pre>
</div>
From a consumer point of view you have two ways of observing the incoming messages.<br/>
Either you simply query the <b>index.html</b>, which will show you the last received message.<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-EevqBlZRN-U/XMVldydvbPI/AAAAAAABf38/TdlKNBUVSaUd_XXo_3CkZ7RSTNxKi0AZgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2019-04-28%2Ba%25CC%2580%2B10.31.57.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-EevqBlZRN-U/XMVldydvbPI/AAAAAAABf38/TdlKNBUVSaUd_XXo_3CkZ7RSTNxKi0AZgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2019-04-28%2Ba%25CC%2580%2B10.31.57.png" data-original-width="878" data-original-height="326" height="30%" width="30%" /></a></div><br/>
Or you can also directly consume from the webservice (on path <b>/consume</b>), which will stack all the messages.<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-1szcrQye6aE/XMVlhkMLGoI/AAAAAAABf4A/r_U30C1UKPkIL2Xi9INPGDoV9eBd42XlQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2019-04-28%2Ba%25CC%2580%2B10.32.08.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-1szcrQye6aE/XMVlhkMLGoI/AAAAAAABf4A/r_U30C1UKPkIL2Xi9INPGDoV9eBd42XlQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2019-04-28%2Ba%25CC%2580%2B10.32.08.png" data-original-width="1022" data-original-height="492" height="30%" width="30%" /></a></div>
<h1>References</h1><br/>
<ul>
<li><a href="https://debezium.io/blog/2019/03/14/debezium-meets-quarkus/">https://debezium.io/blog/2019/03/14/debezium-meets-quarkus/</a> by Jiri Pechanec</li>
<li><a href="https://smallrye.io/smallrye-reactive-messaging/">https://smallrye.io/smallrye-reactive-messaging/</a> by Clément Escoffier</li>
<li><a href="https://quarkus.io/get-started/">https://quarkus.io/get-started/</a></li>
<li><a href="https://github.com/mbutton77/quarkus-kafka">Sources on GitHub</a></li>
</ul>
Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com0tag:blogger.com,1999:blog-2592381182145606344.post-73884955413906044882018-11-17T17:31:00.000+01:002018-11-17T17:31:46.495+01:00Building a real-time dashboard using Kafka & ElasticStack<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-zEfz8tq9A7o/W-8sBxSLPTI/AAAAAAABYTw/7UB2ootlc-4hQdorfz2TBkFJGcAfF_QQQCLcBGAs/s1600/Hotels.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-zEfz8tq9A7o/W-8sBxSLPTI/AAAAAAABYTw/7UB2ootlc-4hQdorfz2TBkFJGcAfF_QQQCLcBGAs/s1600/Hotels.png" data-original-width="512" data-original-height="663" /></a></div><br/><br/>
In this article I will demonstrate how to build a real-time dashboard thanks to the ElasticStack, fed by a CSV and a continuous flow coming from Kafka.<br/>
The CSV contains the description of some hotels, including their names and their GPS coordinates.<br/>
The Kafka flow simulates comments and ratings about the hotels, coming from imaginary customers.<br/>
Like Kafka Streams, the principle is to enrich data coming from this Kafka flow with the data contained in the CSV file.<br/><br />
<h1>About what you need</h1><br/>
For that article, I used :
<ul>
<li>Confluent OpenSource 5.0.1</li>
<li>ElasticStack 6.5.0</li>
<li>Mockaroo</li>
<li>cURL</li>
</ul>
<h1>Generating a consistent dataset</h1><br/>
To generate meaningful data, I'm using <b>Mockaroo</b> (a big thank to <a href="https://rmoff.net/2018/05/10/quick-n-easy-population-of-realistic-test-data-into-kafka-with-mockaroo-and-kafkacat/">Robin Moffatt's article</a> for pointing out that excellent website).<br/>
I strongly encourage you to register : you'll be able to save your work and to consume your data exposed via a REST endpoint (which will prove to be very useful).<br/>
I created two schemas, one for the CSV, containing the hotel information and another one for the comments/ratings.<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-kZ-sc11T64M/W-8vO6iGTyI/AAAAAAABYT8/f9SZFPJO45AvWtuOpU8xHQMXXL608xOjQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-11-16%2Ba%25CC%2580%2B21.53.05.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-kZ-sc11T64M/W-8vO6iGTyI/AAAAAAABYT8/f9SZFPJO45AvWtuOpU8xHQMXXL608xOjQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-11-16%2Ba%25CC%2580%2B21.53.05.png" data-original-width="1444" data-original-height="340" width="70%" height="70%" /></a></div><br/>
Nothing complex about the hotel schema : id / name / latitude / longitude.<br/>
I created 50 sample rows and then exported them to a CSV file.<br />
The ratings schema requires a bit more explanation.<br/><br/>
To generate credible comments, such as :<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
<b>My wife and I were delighted about the restaurant</b>
<i>or</i>
<b>The kids were disappointed about the pool</b>
</pre>
</div>
I used Mockaroo functionality called "<b>Custom List</b>" to create some parts of my sentence and I then assembled all of them into a comment.<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-gD9-1xEgJek/W-8x1VVR2wI/AAAAAAABYUI/EkDFNe4vQEorcitw5nh5m0kVhfqnGQ1PwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-11-16%2Ba%25CC%2580%2B21.53.53.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-gD9-1xEgJek/W-8x1VVR2wI/AAAAAAABYUI/EkDFNe4vQEorcitw5nh5m0kVhfqnGQ1PwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-11-16%2Ba%25CC%2580%2B21.53.53.png" data-original-width="1600" data-original-height="778" width="80%" height="80%" /></a></div> <br/>
That did the trick. But I was facing a little problem about the rating : as I first selected a random value between 1 and 10, it wasn't exactly related to the comment : a person who says he/she is amazed wouldn't normally rate the hotel a 1 or 2 out of 10 !<br/>
Mockaroo has another very cool feature called "<b>Scenario</b>" which will allow you to set a data according to the context. In my case, I chose to set some ratings based on the adjective.<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-qAf-3hGEDKU/W-8y3DZRzcI/AAAAAAABYUU/0FGE0x0KI-wLRhzTzchgeOm07Z7aaecSQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-11-16%2Ba%25CC%2580%2B21.53.19.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-qAf-3hGEDKU/W-8y3DZRzcI/AAAAAAABYUU/0FGE0x0KI-wLRhzTzchgeOm07Z7aaecSQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-11-16%2Ba%25CC%2580%2B21.53.19.png" data-original-width="1600" data-original-height="575" width="80%" height="80%" //></a></div><br/>
I generated 1000 comments which were about the 50 hotels that we created earlier.<br />
Make sure you have chosen the JSON format and that you disabled the "<b>array</b>" checkbox.<br />
You should have comments like this one :
<div class="custom-html-block">
<pre class="prettyprint lang-json">
{
"first_name": "Kaycee",
"last_name": "Doring",
"email": "kdoringrd@hhs.gov",
"gender": "F",
"age": 61,
"hotel_id": 37,
"subject": "The children were",
"level": "pretty",
"satisfaction": "satisfied",
"place": "activities",
"rating": 6,
"comment": "The children were pretty satisfied about the activities"
}
</pre>
</div>
Okay, now our data is ready to be used.<br /><br />
<h1>Sending CSV data to ElasticSearch</h1><br/>
To send our CSV data into ElasticSearch, I used the powerful Logstash and its CSV module.<br />
<div class="custom-html-block">
<pre class="prettyprint lang-json">
input {
file {
path => "20181115_HotelRatings/data/Hotels.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
skip_header => "true"
columns => ["id", "name", "latitude", "longitude"]
}
mutate {
rename => {
"latitude" => "[location][lat]"
"longitude" => "[location][lon]"
}
convert => {
"[location][lat]" => "float"
"[location][lon]" => "float"
}
remove_field => [
"host",
"@version",
"message",
"path"
]
}
}
output {
elasticsearch {
template => "20181115_HotelRatings/esk/hotels_mapping.json"
template_overwrite => "true"
index => "logstash-ratings-hotels"
hosts => ["localhost:9200"]
}
stdout { codec => rubydebug }
}
</pre>
</div>
To be able to use the location as a GeoPoint, you have to use a custom mapping because it's an ElasticSearch concept, completely unknown to Logtash.<br />
The super trick to keep in mind is to use the tag :
<div class="custom-html-block">
<pre class="prettyprint lang-json">
template_overwrite => "true"
</pre>
</div>
Else you could get stuck, wondering why your custom mapping is not taken into account. Super easy when you know it, but before that, it could be a bit foggy.<br />
To create your custom mapping, just follow the <a href="https://www.elastic.co/blog/logstash_lesson_elasticsearch_mapping">excellent (and only) tutorial</a> provided by <a href="https://twitter.com/theuntergeek">Aaron Mildenstein</a><br /><br />
To summarize, you first have to run the CSV import, so that ElasticSearch creates a default mapping. Then you get the mapping, update it and reference it in your logstash conf file.<br />
Then, you can relaunch the CSV injection.
<div class="custom-html-block">
<pre class="prettyprint lang-json">
{
"template": "logstash-ratings-hotels*",
"version": 1,
"settings": {
"index.refresh_interval": "5s"
},
"mappings": {
"_default_": {
"properties": {
"id": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"location": {
<b>"type": "geo_point"</b>
},
"name": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
</pre>
</div>
If everything went fine, you should see your updated mapping with the location as type "<b>geo_point</b>".
To check if everything is in order, create a <b>Coordinate Map</b> in Kibana, mapped on your index and you should have some data.<br/><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-wmUj95FXDpE/W-9AMYw3GPI/AAAAAAABYUg/DO8rXeJVDjA-CCc_82ECW89iQn55INKYwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-11-15%2Ba%25CC%2580%2B14.38.11.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-wmUj95FXDpE/W-9AMYw3GPI/AAAAAAABYUg/DO8rXeJVDjA-CCc_82ECW89iQn55INKYwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-11-15%2Ba%25CC%2580%2B14.38.11.png" data-original-width="1600" data-original-height="843" width="80%" height="80%" /></a></div><br />
<h1>Sending Kafka stream of events to ElasticSearch</h1><br/>
My first idea was to use the Kafka Connect for ElasticSearch. But after I got it working, I wasn't really happy with several facts :
<ul>
<li>It's quite complex</li>
<li>It uses Avro</li>
<li>No filtering before the sending</li>
</ul>
I then took a look at the Kafka input login of Logstash and all the points described above vanished in an instant ! Super easy, works well and fast setup !<br />
Moreover, I feel more comfortable with the fact that my central nervous system (Kafka) knows nothing about the producers / consumers.<br/>
To me, it's a basic principle of loose coupling.<br />
<div class="custom-html-block">
<pre class="prettyprint lang-json">
input {
kafka {
bootstrap_servers => "localhost:9092"
topics => [
"ratings"
]
}
}
filter {
json {
source => "message"
}
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-ratings-hotels"
query => "id:%{hotel_id}"
fields => {
"location" => "location"
"name" => "hotel_name"
}
}
mutate {
remove_field => [
"host",
"message",
"@version",
"path",
"subject",
"satisfaction",
"level",
"place",
"hotel_id"
]
}
}
output {
elasticsearch {
template => "20181115_HotelRatings/esk/comments_mapping.json"
index => "logstash-ratings-comments"
template_overwrite => "true"
hosts => ["localhost:9200"]
}
stdout { codec => rubydebug }
}
</pre>
</div>
I have to explain a bit what's going on here.<br />
The input section is simple enough. Then in the filter section, I describe the content of the message as JSON, which will index each pair of key/value.<br />
The elasticsearch plugin is used to retrieve data previously indexed in my elastic cluster. I just have to define which index I want to get data from and in the query field, the join condition. And once the join is done, I have to specify the fields I want to get copied in my record.<br /><br />
To be more concrete, each time a new comment record is received from Kafka, we isolate the <b>hotel_id</b> and look for the corresponding <b>id</b> in the index <b>logstash-ratings-hotels</b>. Once found, we copy the field <b>location</b> and <b>name</b>, to <b>location</b> and <b>hotel_name</b>, respectively.<br/><br/>
The mutate filter is used to remove all the unnecessary fields (all the fields used to create the final comment for instance).<br /><br />
The mechanism to transform a classical data into a <b>geo_point</b> is exactly the same as the one described above : through a <b>custom mapping</b>.<br />
Here's the custom mapping for the comments.
<div class="custom-html-block">
<pre class="prettyprint lang-json">
{
"template": "logstash-ratings-comments*",
"version": 1,
"settings": {
"index.refresh_interval": "5s"
},
"mappings": {
"_default_": {
"properties": {
"@timestamp": {
"type": "date"
},
"age": {
"type": "long"
},
"comment": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"email": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"first_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"gender": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"location": {
<b>"type": "geo_point"</b>
},
"hotel_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"last_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"rating": {
"type": "long"
}
}
}
}
}
</pre>
</div>
<h1>Designing our dashboard</h1><br/>
Before sending our stream of data, it's interesting to send only one record. That way, ElasticSearch will create an index, which will be used for creating a Kibana index.<br />
From that Kibana Index, we will be able to design our visualizations and create a dashboard.
First of all, we've got to summarize the information we have.<br/>
When we sent the sample record, the output Ruby plugin wrote it on the Logstash standard out.
<div class="custom-html-block">
<pre class="prettyprint lang-json">
{
"first_name" => "Kaycee",
"email" => "kdoringrd@hhs.gov",
"location" => {
"lat" => 21.938596,
"lon" => 110.552977
},
"last_name" => "Doring",
"rating" => 6,
"@timestamp" => 2018-11-17T13:16:22.120Z,
"age" => 61,
"gender" => "F",
"hotel_name" => "Konopelski Inc",
"comment" => "The children were pretty satisfied about the activities"
}
</pre>
</div>
I'll exploit 4 elements :
<ul>
<li>Hotel location => as a map</li>
<li>Hotel Name & rating => Top 20 of the best rated hotels</li>
<li>Age of the person => Age donut</li>
<li>Gender => Gender pie</li>
</ul><br/>
Now the dashboard is ready, it's time to send some data !<br/><br/>
<h1>Streaming Mockaroo JSON data to Kafka</h1><br/>
I created a simple topic ratings, with only one partition. This time, I'm not interested in scaling (that will be the subject of another blogpost)<br />
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
kafka-topics --create --zookeeper localhost:2181 --topic ratings --partitions 1 --replication-factor 1
</pre>
</div>
I could use <b>Kafkacat</b> to stream the data from the REST endpoint to my Kafka topic, but I prefer to use the platform, as <a href="https://twitter.com/AdamBien">Adam Bien</a> would say :)<br />
In order to see a progressive update of the dashboard, I used <b>awk</b> to send the messages at a slower rate.<br />
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
curl -s "https://api.mockaroo.com/api/fb43c4e0?count=1000&key=0673f940" \
| awk '{print $0; system("sleep 1");}' \
| kafka-console-producer --broker-list localhost:9092 --topic ratings
</pre>
</div>
That way, it will send a message every second.<br /><br/>
<h1>Results & Live demonstration</h1><br/>
You can see a sample of what it will be like.
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-eCy2guYjMHo/W_Asq-qYhOI/AAAAAAABYVA/4BWlB_hZNYAR46X6oH6JAMTIlL7ppE7XACLcBGAs/s1600/Hotels.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-eCy2guYjMHo/W_Asq-qYhOI/AAAAAAABYVA/4BWlB_hZNYAR46X6oH6JAMTIlL7ppE7XACLcBGAs/s1600/Hotels.gif" data-original-width="640" data-original-height="400" width="80%" height="80%" /></a></div><br/>
And here's the dashboard, once the 1000 comments have been processed.<br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-DwW28sgKhT0/W_AwUIJemmI/AAAAAAABYVM/0fD0h4qxq-gC7akYi3oIY-23yhrl4wD0ACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-11-17%2Ba%25CC%2580%2B15.45.22.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://1.bp.blogspot.com/-DwW28sgKhT0/W_AwUIJemmI/AAAAAAABYVM/0fD0h4qxq-gC7akYi3oIY-23yhrl4wD0ACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-11-17%2Ba%25CC%2580%2B15.45.22.png" data-original-width="1600" data-original-height="833" width="80%" height="80%" /></a></div><br/>
<h1>Source and references</h1><br/>
<ul>
<li><a href="https://www.elastic.co/blog/logstash_lesson_elasticsearch_mapping">https://www.elastic.co/blog/logstash_lesson_elasticsearch_mapping</a></li>
<li><a href="https://rmoff.net/2018/05/10/quick-n-easy-population-of-realistic-test-data-into-kafka-with-mockaroo-and-kafkacat/">https://rmoff.net/2018/05/10/quick-n-easy-population-of-realistic-test-data-into-kafka-with-mockaroo-and-kafkacat/</a></li>
<li><a href="https://www.kisspng.com/png-zipper-royalty-free-drawing-zip-zip-iron-316065/">Zip image</a>
</ul>
Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com0tag:blogger.com,1999:blog-2592381182145606344.post-25799299153097156442018-09-09T18:20:00.000+02:002018-09-09T18:20:08.911+02:00IoT / SmartHome - Send EnOcean sensor data to Kafka<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-s67n2JNXoVQ/W5PKlhtIg1I/AAAAAAABW1M/7JBdLETwQPwilquE2u1uJXHSaOOdZI1bgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-08%2Ba%25CC%2580%2B15.11.24.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-s67n2JNXoVQ/W5PKlhtIg1I/AAAAAAABW1M/7JBdLETwQPwilquE2u1uJXHSaOOdZI1bgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-08%2Ba%25CC%2580%2B15.11.24.png" data-original-width="1600" data-original-height="717" width="80%"/></a></div>
<br/>
Lately, like everyone else, I've been reading a lot about IoT and how this could change the world.<br/>
Even if it's been a while since I first heard that (1995 was the year when I first heard about the concepts of smart home), this time, I've got a feeling that it will soon be truly available to everyone.<br/><br/>
So I read about the different protocols that are available today.<br/>
Bluetooth LE, ZigBee, Z-Wave and EnOcean are among the most used / discussed, from my point of view.<br/><br/>
I've been seduced by <a href="https://www.enocean.com/en/">EnOcean</a>, since its technology is indeed well-suited to IoT : very low energy consumption and for most sensors, the ability to work without batteries !<br/>
In this article, as depicted in the first picture I will demonstrate a simple scenario : an EnOcean sensor will send data to Jeedom, which will feed a MQTT broker and then a synchronizer will send each received message to a Kafka topic. Then the latest value will be exposed to be displayed in a Grafana dashboard.<br/>
<!-- ######################################################## -->
<br/><h1>Before we start ...</h1><br/>
For this configuration, you will need :
<ul>
<li>a Raspberry Pi (with everything it needs to run)</li>
<li>a <a href="https://amzn.to/2wympLT">EnOcean sensor (wall switch from <a href="https://nodon.fr/en/wireless-protocols/enocean/">NodOn</a>)</a> </li>
<li>a <a href="https://amzn.to/2LKvWV6">EnOcean physical bridge (USB stick)</a></li>
<li>Jeedom, <a href="https://www.jeedom.com/market/index.php?v=d&p=market&type=plugin&&name=mqtt">MQTT plugin</a> and the <a href="https://www.jeedom.com/market/index.php?v=d&p=market&type=plugin&&name=enocean">EnOcean plugin</a> (6€)</li>
<li><a href="https://www.soapui.org/downloads/soapui.html">SoapUI Open Source</a></li>
<li>A Java IDE (I'm using <a href="https://netbeans.org/downloads/">Netbeans</a>)</li>
<li>Kafka and ZooKeeper (I'm using the <a href="https://www.confluent.io/product/confluent-open-source/">Confluent Open Source</a>)</li>
<li><a href="https://prometheus.io/download/">Prometheus</a></li>
<li><a href="https://grafana.com/grafana/download">Grafana</a></li>
</ul>
<!-- ######################################################## -->
<br/><h1>Installing Jeedom</h1><br/>
Well, there are plenty of ways to do so, so I won't guide you on this one, especially since I've installed it on a existing RPi.<br/>
Please refer to the <a href="https://jeedom.github.io/documentation/installation/en_US/index">installation guide, from the official documentation</a> (I think the translation has been started but not finished yet ...).<br/>
(Note that you can install it on Synology NAS, which can also be interesting)<br/>
<!-- ######################################################## -->
<br/><h1>Configuring the EnOcean sensor</h1><br/>
Once Jeedom is installed (as well as the plugins) and the EnOcean USB stick has been plugged in, it's time to bind the sensor.<br/>
First of all, we will add a group, which will allow an easier administration. You can think of it as the room your sensor is in.<br/>
Go to the menu <b>Tools > Objects</b> and click on <b>Add</b><br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-HtvAdQnqO3Q/W4ogu0V1DII/AAAAAAABWmw/-CLLxwpgU_46aUcTO56LVS-u_qO6lknIwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B06.56.02.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-HtvAdQnqO3Q/W4ogu0V1DII/AAAAAAABWmw/-CLLxwpgU_46aUcTO56LVS-u_qO6lknIwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B06.56.02.png" data-original-width="564" data-original-height="362" /></a></div><br/><br/>
Give it the name <b>Room</b> :
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-BdqiNkdM-gs/W4oiVoVCsuI/AAAAAAABWnM/YQ3X5cLCMe0hIlFIEcGhTV77kkoXkT5VACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B07.21.31.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-BdqiNkdM-gs/W4oiVoVCsuI/AAAAAAABWnM/YQ3X5cLCMe0hIlFIEcGhTV77kkoXkT5VACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B07.21.31.png" data-original-width="1600" data-original-height="766" height="50%" width="50%"/></a></div><br/><br/>
Then to add the sensor, go to the menu : <b>Plugins > Automation protocols > EnOcean</b> <br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-DHdGvMtAPlk/W4oc8pnrXeI/AAAAAAABWlo/vYpeEuOtX2cVCtcn3C_j4HVFyxMMq_XvwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B06.54.25.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-DHdGvMtAPlk/W4oc8pnrXeI/AAAAAAABWlo/vYpeEuOtX2cVCtcn3C_j4HVFyxMMq_XvwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B06.54.25.png" data-original-width="758" data-original-height="424" height="50%" width="50%"/></a></div><br/><br/>
Click on <b>Inclusion Mode</b> :<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-UnNNZsj6Rpk/W4odiU6Ag6I/AAAAAAABWl0/2TynPhfMhpQW8kBcKV8vjtoOW_RKm-2xQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B07.02.17.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-UnNNZsj6Rpk/W4odiU6Ag6I/AAAAAAABWl0/2TynPhfMhpQW8kBcKV8vjtoOW_RKm-2xQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B07.02.17.png" data-original-width="290" data-original-height="288" height="15%" width="15%"/></a></div><br/><br/>
And then choose the automatic mode on the next window :<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-GHFKDsHuhYw/W4ofFdNx_CI/AAAAAAABWmQ/2GpfasIM70sV6onyzY2ayQ2sktjhwKjUgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B06.55.23.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-GHFKDsHuhYw/W4ofFdNx_CI/AAAAAAABWmQ/2GpfasIM70sV6onyzY2ayQ2sktjhwKjUgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B06.55.23.png" data-original-width="1244" data-original-height="706" height="50%" width="50%"/></a></div><br/><br/>
Click on any button, then your device should show up in the EnOcean plugin homepage.<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-0Bhr89dr3N4/W4ofu9FiTMI/AAAAAAABWmc/1P4oY8sx3vcfHxEtYC360Vi1th-mN7adwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B06.55.36.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-0Bhr89dr3N4/W4ofu9FiTMI/AAAAAAABWmc/1P4oY8sx3vcfHxEtYC360Vi1th-mN7adwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B06.55.36.png" data-original-width="516" data-original-height="572" /></a></div><br/><br/>
If you want to see the first physical interactions or if you just want to make sure your device works properly, you can go to the <b>Health</b> section of the EnOcean plugin menu and you will see the current status of your sensor.
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-wvX9FPbGA9E/W4ogUHBFWdI/AAAAAAABWmk/O37izQ49VMYu76tp5pTa1pOPlkqsXuIpACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B06.58.12.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-wvX9FPbGA9E/W4ogUHBFWdI/AAAAAAABWmk/O37izQ49VMYu76tp5pTa1pOPlkqsXuIpACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B06.58.12.png" data-original-width="1600" data-original-height="127" height="70%" width="70%"/></a></div><br/>
<!-- ######################################################## -->
<br/><h1>Configuring Mosquitto (MQTT Broker)</h1><br/>
First, make sure that the MQTT plugin has been correctly installed. If not, please do so.<br/>
Normally, when you install the MQTT plugin, Mosquitto is automatically added. If not, install Mosquitto, following the <a href="https://mosquitto.org/download/">official documentation</a>.<br/>
Configure the MQTT plugin by specifying the default configuration (the default password being "<b>jeedom</b>") :<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-lVMfAoCYGcM/W4o4XkbGjYI/AAAAAAABWnY/8v5He6m1V5Y8-VfTTyqFsOcTUYyfxkJ4QCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B08.57.08.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-lVMfAoCYGcM/W4o4XkbGjYI/AAAAAAABWnY/8v5He6m1V5Y8-VfTTyqFsOcTUYyfxkJ4QCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B08.57.08.png" data-original-width="1222" data-original-height="911" height="50%" width="50%"/></a></div><br/><br/>
The <b>#</b> sign means the root, or all topics if you prefer. If you don't know how the MQTT protocol works : read <a href="https://mosquitto.org/man/mqtt-7.html">that</a>.<br/>
Like Kafka, if the message is sent to a non-existing topic, the Mosquitto broker will automatically create it.<br/>
<!-- ######################################################## -->
<br/><h1>Using SoapUI to test a scenario</h1><br/>
<a href="https://www.soapui.org/">SoapUI</a> is an excellent tool I've been using for many years now. I mostly use it to test functionalities built upon webservices.<br/>
If you are not familiar with it yet, I strongly encourage you to try it.<br/>
<a href="https://smartbear.com/">SmartBear</a> is also behind Swagger, which is part of the excellent <a href="https://microprofile.io/">Eclipse MicroProfile</a>.<br/><br/>
Create a TestSuite and add a <b>Publish using MQTT</b> step : <br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-gG0mel37Ie4/W4pQ6fyN4uI/AAAAAAABWoo/TFGTTIFWxzs3LIBOmBDWV6ERB5AwJ-z4QCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B10.41.10.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-gG0mel37Ie4/W4pQ6fyN4uI/AAAAAAABWoo/TFGTTIFWxzs3LIBOmBDWV6ERB5AwJ-z4QCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B10.41.10.png" data-original-width="1202" data-original-height="905" height="50%" width="50%"/></a></div><br/><br/>
Configure the MQTT broker and make sure you have the same information you have in the Jeedom MQTT plugin.<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/--qyfb2MI2yk/W4o7iaMfEWI/AAAAAAABWns/AwgnNFhaBektJSwv58_KwVU5LGQigJ3TgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B09.10.11.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/--qyfb2MI2yk/W4o7iaMfEWI/AAAAAAABWns/AwgnNFhaBektJSwv58_KwVU5LGQigJ3TgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B09.10.11.png" data-original-width="1170" data-original-height="708" height="50%" width="50%"/></a></div><br/><br/>
Send a sample value to the topic <b>sensor/counter</b> : <br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-fpEmWajgw7g/W4o8SpAlF6I/AAAAAAABWn4/u2fJtfgHoMQJW6E3mz8hK0BzOQySsByVACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B09.13.21.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-fpEmWajgw7g/W4o8SpAlF6I/AAAAAAABWn4/u2fJtfgHoMQJW6E3mz8hK0BzOQySsByVACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B09.13.21.png" data-original-width="1074" data-original-height="900" height="50%" width="50%"/></a></div><br/><br/>
If you go to your Room dashboard, in Jeedom, you should see the value reflected on the gauge, as well as the state of your buttons (each one has a on/off status) :<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-9rCqTLdeKhA/W4pBC6UkWII/AAAAAAABWoE/kmsAXAeRR_M34Wb9t5lvPqf_vntTjgnWQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B09.33.29.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-9rCqTLdeKhA/W4pBC6UkWII/AAAAAAABWoE/kmsAXAeRR_M34Wb9t5lvPqf_vntTjgnWQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B09.33.29.png" data-original-width="992" data-original-height="330" /></a></div><br/><br/>
Of course, in our simple case, you may also use the Mosquitto client, which is far simpler.<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bash">
mosquitto_pub -h localhost -t sensor/counter -m 25
</pre>
</div>
But keep in mind that to test a real-world scenario, with multi steps with correlation, pause, etc, SoapUI is, by far, the best solution to me.<br/>
<!-- ######################################################## -->
<br/><h1>Creating a Jeedom scenario</h1><br/>
All right, everything is set up, but for now, we've not used our EnOcean sensor yet !<br/>
The wall switch can be used in two configurations : 2-button configuration or 4-button configuration. We will use the first one.<br/>
The principle is pretty simple : when I click the upper button, I want the counter to increase and when I click the down button, I want it to decrease.<br/><br/>
Since you have sent a message to the topic sensor/counter (whether it was through SoapUI or Mosquitto client), it has already been created.<br/>
On the MQTT plugin configuration page, you should then have your topic displayed. <br/>
Bind it to the Room object (choose it as the parent).<br/><br/>
If you go to the <b>Commands</b> tab, you should see the subtopic.<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-oDTVh69bW5k/W4pNGaFQE5I/AAAAAAABWoQ/P2-GV5npaB4CMD1FV6CqfJZBK1zGrGmPwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B10.25.29.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-oDTVh69bW5k/W4pNGaFQE5I/AAAAAAABWoQ/P2-GV5npaB4CMD1FV6CqfJZBK1zGrGmPwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B10.25.29.png" data-original-width="1600" data-original-height="128" height="80%" width="80%"/></a></div><br/><br/><br/>
Tick the checkbox <b>Historiser</b> (not translated !) which will allow you to track the value changes and to display a time series in the <b>Analysis > History</b> menu.<br/>
You can also check the current value by clicking <b>Tester</b>.<br/><br/>
We will add two actions to that component. One for increasing the value, and the other one to decrease the value.<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-OKkH7sS4uek/W4pOnbVkw9I/AAAAAAABWoc/XvzPKAsFqrgbxjJTxoa0pkX5ANQoeYdsQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B08.03.29.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-OKkH7sS4uek/W4pOnbVkw9I/AAAAAAABWoc/XvzPKAsFqrgbxjJTxoa0pkX5ANQoeYdsQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B08.03.29.png" data-original-width="1600" data-original-height="390" height="80%" width="80%"/></a></div><br/><br/>
Name them as you like, as long as it stays meaningful to you.<br/>
What's important is the <b>Payload</b> field : in the <b>Up</b> action, it will take the value <b>#[Room][sensor][counter]#+1</b> which is the current value of the topic (represented by the final #) plus one.<br/>
And in the <b>Down</b> section, you can guess :)<br/><br/>
Now we need to create scenarios and decide when to execute them.<br/>
To do so, go to the <b>Tools > Scenario</b> menu and then click <b>Add</b>.<br/><br/>
Create two scenarios, one for the increase, the other one for the decrease.<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/--mqweZWtv1s/W4pSUrJDE3I/AAAAAAABWo0/DIEAz8wmFUAZWU2QZ9WgdnHt1Tn4x6tbwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B10.45.42.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/--mqweZWtv1s/W4pSUrJDE3I/AAAAAAABWo0/DIEAz8wmFUAZWU2QZ9WgdnHt1Tn4x6tbwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B10.45.42.png" data-original-width="750" data-original-height="348" height="50%" width="50%"/></a></div><br/><br/>
For a scenario to be properly defined, you need to decide when it's triggered and what it will do.<br/>
For the trigger, it will be the click on a EnOcean sensor button. (Button 1 is the one which will be used to the increase action).<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-5UrXc_6tI4U/W4pS1hH-gwI/AAAAAAABWo8/aHwXrNSHrREG_YtXxZEKDbaPtsIGp1wygCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B10.46.26.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-5UrXc_6tI4U/W4pS1hH-gwI/AAAAAAABWo8/aHwXrNSHrREG_YtXxZEKDbaPtsIGp1wygCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B10.46.26.png" data-original-width="834" data-original-height="210" height="50%" width="50%"/></a></div><br/><br/>
For the actions, go to the <b>Scenario</b> pane and click on <b>+Add block</b> button on the right side of the screen.<br/>
Chose <b>Action</b> :
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-Wu7tNMNW-XU/W4pTk1q16sI/AAAAAAABWpI/yInIWQdCs8ALdo136xaSdcTcH90kKWQVACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B10.51.47.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-Wu7tNMNW-XU/W4pTk1q16sI/AAAAAAABWpI/yInIWQdCs8ALdo136xaSdcTcH90kKWQVACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B10.51.47.png" data-original-width="1290" data-original-height="718" height="50%" width="50%"/></a></div><br/><br/>
Once the action block has been created, we will call one of the action that we previously added on the MQTT component.<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-nq6Kb_vAAbc/W4pT4DFYZPI/AAAAAAABWpQ/ZDzS31h-txMn4ltKMnlCgrIYOVty9HHXwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B10.46.40.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-nq6Kb_vAAbc/W4pT4DFYZPI/AAAAAAABWpQ/ZDzS31h-txMn4ltKMnlCgrIYOVty9HHXwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-01%2Ba%25CC%2580%2B10.46.40.png" data-original-width="1130" data-original-height="270" height="50%" width="50%"/></a></div><br/><br/>
Repeat the process for the <b>Down</b> button (whose number is 3) and invoke the <b>Down</b> action.<br/>
<!-- ######################################################## -->
<br/><h1>Coding the MQTT / Kafka synchronizer</h1><br/>
I haven't had the time to code a Kafka connector yet, that's why I wrote a very small piece of code.<br/><br/>
As usual, I used Maven to create my project.
Three dependencies are needed, here's an excerpt of my POM :
<div class="custom-html-block">
<pre class="prettyprint lang-xml">
<dependency>
<groupId>org.eclipse.paho</groupId>
<artifactId>org.eclipse.paho.client.mqttv3</artifactId>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.0.13</version>
</dependency>
</pre>
</div>
And here's the code :
<div class="custom-html-block">
<pre class="prettyprint lang-java">
package fr.mbutton.kafka.sync;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
import java.util.logging.Level;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.header.Header;
import org.apache.kafka.common.header.internals.RecordHeader;
import org.eclipse.paho.client.mqttv3.IMqttDeliveryToken;
import org.eclipse.paho.client.mqttv3.MqttCallback;
import org.eclipse.paho.client.mqttv3.MqttClient;
import org.eclipse.paho.client.mqttv3.MqttConnectOptions;
import org.eclipse.paho.client.mqttv3.MqttException;
import org.eclipse.paho.client.mqttv3.MqttMessage;
import org.eclipse.paho.client.mqttv3.persist.MemoryPersistence;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class MQTT2Kafka {
private final transient Logger logger = LoggerFactory.getLogger(this.getClass());
public static String MQTT_BROKER_URL = "tcp://192.168.0.232:1883";
public static String MQTT_BROKER_USER = "jeedom";
public static String MQTT_BROKER_PASSWORD = "jeedom";
public static String KAFKA_BOOTSTRAP = "localhost:9092";
public static String KAFKA_TOPIC = "sensor.counter";
public void synchronize() {
final Properties props = new Properties();
props.put("bootstrap.servers", KAFKA_BOOTSTRAP);
props.put("key.serializer" , "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer" , "org.apache.kafka.common.serialization.StringSerializer");
MemoryPersistence persistence = new MemoryPersistence();
MqttClient mqttClient = null; // not auto-closeable
try {
mqttClient = new MqttClient(MQTT_BROKER_URL, "Synchronizer", persistence);
MqttConnectOptions connOpts = new MqttConnectOptions();
connOpts.setCleanSession(true);
mqttClient.setCallback(new MqttCallback() {
@Override
public void connectionLost(Throwable cause) {
logger.warn("Connection lost ¯\\_(ツ)_/¯ : please restart me !");
}
@Override
public void messageArrived(String topic, MqttMessage message) throws Exception {
logger.debug("Message: " + message.toString());
sendMessageOverKafka(topic, message);
}
@Override
public void deliveryComplete(IMqttDeliveryToken token) {
logger.debug("Job's done.");
}
private void sendMessageOverKafka(String topic, MqttMessage message) {
logger.debug("Sending the message to my Kafka cluster.");
List<Header> headers = new ArrayList<>();
headers.add(new RecordHeader("mqtt-topic", topic.getBytes()));
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
ProducerRecord<String, String> record = new ProducerRecord<>(KAFKA_TOPIC, 0, "1", message.toString(), headers);
producer.send(record);
}
}
});
mqttClient.connect(connOpts);
mqttClient.subscribe("#");
} catch (MqttException e) {
logger.error("Little problem while trying to synchronize MQTT gateway with Kafka broker.", e);
} finally {
if (mqttClient != null) {
try {
mqttClient.close();
} catch (MqttException ex) {
logger.error("Error while liberating the resources.", ex);
}
}
}
}
public static void main(String[] args) {
MQTT2Kafka bridge = new MQTT2Kafka();
bridge.synchronize();
}
}
</pre>
</div><br/>
And that's it.<br/>
<!-- ######################################################## -->
<br/><h2>Trying to build a native image thanks to GraalVM</h2><br/>
As this lib seems to be a perfect use case, I tried to use GraalVM, just out of curiosity, to generate a native image.<br/>
Unfortunately, I encountered some generation problems that seem unresolvable for now.<br/>
On my Maven project, I first copied the dependencies to make the compilation easier :<br/><br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bash">
mvn dependency:copy-dependencies
[INFO] --- maven-dependency-plugin:2.6:copy-dependencies (default-cli) @ KafkaMQTTSynchronizer ---
[INFO] Copying slf4j-api-1.7.25.jar to sources/KafkaMQTTSynchronizer/target/dependency/slf4j-api-1.7.25.jar
[INFO] Copying org.eclipse.paho.client.mqttv3-1.2.0.jar to sources/KafkaMQTTSynchronizer/target/dependency/org.eclipse.paho.client.mqttv3-1.2.0.jar
[INFO] Copying snappy-java-1.1.7.1.jar to sources/KafkaMQTTSynchronizer/target/dependency/snappy-java-1.1.7.1.jar
[INFO] Copying logback-core-1.0.13.jar to sources/KafkaMQTTSynchronizer/target/dependency/logback-core-1.0.13.jar
[INFO] Copying javaee-web-api-7.0.jar to sources/KafkaMQTTSynchronizer/target/dependency/javaee-web-api-7.0.jar
[INFO] Copying lz4-java-1.4.1.jar to sources/KafkaMQTTSynchronizer/target/dependency/lz4-java-1.4.1.jar
[INFO] Copying logback-classic-1.0.13.jar to sources/KafkaMQTTSynchronizer/target/dependency/logback-classic-1.0.13.jar
[INFO] Copying kafka-clients-1.1.1.jar to sources/KafkaMQTTSynchronizer/target/dependency/kafka-clients-1.1.1.jar
</pre>
</div><br/>
Compiled the source :
<div class="custom-html-block">
<pre class="prettyprint lang-bash">
graalvm-ce-1.0.0-rc5/bin/javac -cp "../../../target/dependency/*" fr/mbutton/kafka/sync/MQTT2Kafka.java
</pre>
</div><br/>
Tried to generate the native image :<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bash">
graalvm-ce-1.0.0-rc5/bin/native-image -cp ".:../../../target/dependency/*" fr.mbutton.kafka.sync.MQTT2Kafka
Build on Server(pid: 47807, port: 52391)
classlist: 827.05 ms
(cap): 1,428.57 ms
setup: 2,142.86 ms
Detected unnecessary RecomputeFieldValue.ArrayBaseOffset
com.oracle.svm.core.jdk.Target_java_nio_DirectByteBuffer.arrayBaseOffset substitution field for java.nio.DirectByteBuffer.arrayBaseOffset.
The annotated field can be removed. This ArrayBaseOffset computation can be detected automatically.
Use option -H:+UnsafeAutomaticSubstitutionsLogLevel=2 to print all automatically detected substitutions.
analysis: 8,346.29 ms
error: unsupported features in 3 methods
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException:
Invoke with MethodHandle argument could not be reduced to at most a single call:
java.lang.invoke.LambdaForm$MH.2127087555.invoke_MT(Object, Object, Object)
Trace:
at parsing org.apache.kafka.common.record.CompressionType$3.wrapForInput(CompressionType.java:90)
[...]
</pre>
</div><br/>
I'm disappointed. Maybe next time !<br/>
<!-- ######################################################## -->
<br/><h1>Setting up Kafka</h1><br/>
Thanks to the Confluent Suite, starting Kafka for testing can be done in a blink of an eye.<br/>
<a href="https://www.confluent.io/download/">Download Confluent Open Source</a> and start it with the following command : <br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bash">
bin/confluent start
</pre>
</div>
Then create a mirror topic that will receive messages from the MQTT broker.<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bash">
bin/kafka-topics --zookeeper localhost:2181 --create --partitions 1 --replication-factor 1 --topic sensor.counter
</pre>
</div>
Run the KafkaSync (Java bridge) either via the CLI or via your IDE<br/><br/>
Click on your buttons, to see if the changes are correctly reflected in Kafka.<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bash">
bash-4.4$ bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic sensor.counter
26
27
28
29
28
27
26
25
24
25
</pre>
</div>
<!-- ######################################################## -->
<br/><h1>Exposing the information to Prometheus</h1><br/>
<!-- ######################################################## -->
<br/><h2>Coding the exporter</h2><br/>
To expose the data to Prometheus, you only have to expose text composed of a name and a value.<br/>
In our case, what I'm interested in, is to expose the current (or last known) value of a topic.<br/>
As Kafka stores the values in a log, I thought it would be pretty simple to take a quick peek. But it's not and the product has clearly not been made to be queryable as a database.<br/>
(Even if it's possible to see the current state of your topics in the Kafka Control Center)<br/><br/>
I then thought about coding a consumer that would write the change in a database. But it sounded too oversized ...<br/>
Same thing about writing in a file ...<br/>
I then found a neat in-memory solution.<br/><br/>
It's really simple : I used Payara Micro (I love it) to deploy a simple app with a WebListener that launches a thread, in charge of listening for new Kafka messages.<br/>
Then on each received message, it updates a static variable, on a JAXRS service, whose value is returned when a GET query is made.<br/><br/>
JAX-RS Service :<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-java">
package fr.mbutton.prometheus;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
@Path("metrics")
public class KafkaCounterExporter {
static String counterValue = "0";
@GET
public String getCounter() {
return "counter " + counterValue;
}
}
</pre>
</div>
Can it be simpler ? <br/>
And as for the WebListener :
<div class="custom-html-block">
<pre class="prettyprint lang-java">
package fr.mbutton.prometheus;
import java.util.Arrays;
import java.util.Properties;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;
import javax.servlet.annotation.WebListener;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@WebListener
public class KafkaListener implements ServletContextListener {
private transient Logger logger = LoggerFactory.getLogger(this.getClass());
@Override
public void contextInitialized(ServletContextEvent sce) {
logger.info("Starting Kafka Listener");
Runnable listener = () -> {
Properties props = new Properties();
props.put("bootstrap.servers" , "localhost:9092");
props.put("group.id" , "confluent");
props.put("key.deserializer" , "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(Arrays.asList("sensor.counter"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(10);
for (ConsumerRecord<String, String> record : records) {
KafkaCounterExporter.counterValue = record.value();
}
}
}
};
Thread t = new Thread(listener);
t.start();
}
@Override
public void contextDestroyed(ServletContextEvent sce) {
logger.info("Disconnecting");
}
}
</pre>
</div>
Then use your favorite IDE (NetBeans for me) to build your project. <br/>
Once it's built, run it with Payara Micro with the following line :
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
java -jar payara-micro-5.183.jar --deploy /<path>/KafkaPrometheusExporter-1.0-SNAPSHOT.war --port 8090
</pre>
</div>
<!-- ######################################################## -->
<br/><h2>Configuring Prometheus</h2><br/>
Now we have a MicroProfile application running, the endpoint is something like <b>http://localhost:8090/KafkaPrometheusExporter-1.0-SNAPSHOT/prometheus/metrics</b><br/>
But by default, you just can't use a random endpoint : it has to be <b>/metrics</b><br/><br/>
There are several ways to fix that. The simplest way is to change the metrics path.<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
- job_name: kafka-mqtt
metrics_path: KafkaPrometheusExporter-1.0-SNAPSHOT/prometheus/metrics
static_configs:
- targets: ['localhost:8090']
</pre>
</div><br/>
But if like me, you're using NGinx as a reverse proxy, you may configure it to perform URL rewriting.<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
location /metrics {
proxy_pass http://localhost:8090/KafkaPrometheusExporter-1.0-SNAPSHOT/prometheus/metrics;
}
</pre>
</div>
The following URL (<b>http://localhost/metrics</b>) is then available and ready to be scraped.<br/><br/>
To be sure, open Prometheus and try to display the counter stat.<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-Ta-hGrQ6HWk/W5NsiBMwLAI/AAAAAAABW1A/Mk4cuXralC0FE1xNAJdV9UyIJ1uSj--HQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-08%2Ba%25CC%2580%2B08.01.22.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-Ta-hGrQ6HWk/W5NsiBMwLAI/AAAAAAABW1A/Mk4cuXralC0FE1xNAJdV9UyIJ1uSj--HQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-08%2Ba%25CC%2580%2B08.01.22.png" data-original-width="1600" data-original-height="440" width=80%"/></a></div><br/>
<!-- ######################################################## -->
<br/><h1>Displaying it via Grafana</h1><br/>
I won't spend too much time on setting up Grafana, which is not the topic of this post.<br/>
Just as a reminder, you'll have to define a DataSource, pointing at your Prometheus.<br/>
Then, look for the stat <b>counter</b> that we exposed.<br/>
And create a new dashboard, based on a single stat, make it a gauge, configure it more or less like that :<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-IW1dGdCIerA/W5NsBFaHbEI/AAAAAAABW0w/_KnfJZs-xVkVTvjYYrg9LKe44tmO-RjCgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-08%2Ba%25CC%2580%2B08.20.34.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-IW1dGdCIerA/W5NsBFaHbEI/AAAAAAABW0w/_KnfJZs-xVkVTvjYYrg9LKe44tmO-RjCgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-08%2Ba%25CC%2580%2B08.20.34.png" data-original-width="1600" data-original-height="808" width=80%" /></a></div><br/>
And when you're satisfied, you can live-test it (with the auto-refresh option).<br/><br/>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-Sj6AQekTaVY/W5NsPJSh5NI/AAAAAAABW00/M8FKxPiMxzMyf6oMgD_601QXRpo7qs1-QCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-08%2Ba%25CC%2580%2B08.22.39.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-Sj6AQekTaVY/W5NsPJSh5NI/AAAAAAABW00/M8FKxPiMxzMyf6oMgD_601QXRpo7qs1-QCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2018-09-08%2Ba%25CC%2580%2B08.22.39.png" data-original-width="1600" data-original-height="1030" width=80%" /></a></div><br/>
<!-- ######################################################## -->
<br/><h1>References</h1><br/>
<ul>
<li><a href="https://www.enocean.com/en/">EnOcean</a></li>
<li><a href="https://smartbear.com/plugins/mqtt-test-steps-page/">SoapUI & MQTT</a></li>
<li><a href="https://mosquitto.org/">Mosquitto</a></li>
<li><a href="https://jeedom.github.io/documentation/howto/en_US/raspberrypi3">Jeedom on a RPi 3</a></li>
<li><a href="http://mqtt.org/">MQTT</a></li>
<li><a href="https://www.eclipse.org/paho/">Eclipse Paho</a></li>
<li><a href="https://thenewstack.io/apache-kafka-cornerstone-iot-data-platform/">Apache Kafka: The Cornerstone of an Internet-of-Things Data Platform</a></li>
<li><a href="https://kafka.apache.org/documentation/#api">Apache Kafka APIs</a></li>
<li><a href="https://www.graalvm.org/docs/getting-started/">GraalVM</a></li>
<li><a href="https://www.payara.fish/payara_micro">Payara Micro</a></li>
<li><a href="https://prometheus.io/docs/instrumenting/writing_exporters/">Writing Prometheus exporters</a></li>
<li><a href="http://docs.grafana.org/guides/getting_started/">First steps with Grafana</a></li>
<li><a href="https://github.com/mbutton77">My GitHub on which to find the sources</a></li>
</ul>
Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com0tag:blogger.com,1999:blog-2592381182145606344.post-46049869562700228662018-04-22T19:29:00.000+02:002018-11-16T20:56:33.414+01:00KSQL, a step beyond Kafka Streams<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-sQUoiriWpk0/WtykmhGYQnI/AAAAAAABPyk/p4O0CuWqoFMZ5rHRfQ1bP2szXFnsKzGKQCLcBGAs/s1600/ksql.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-sQUoiriWpk0/WtykmhGYQnI/AAAAAAABPyk/p4O0CuWqoFMZ5rHRfQ1bP2szXFnsKzGKQCLcBGAs/s1600/ksql.png" data-original-width="650" data-original-height="450" /></a></div><br/><br/>
I'm currently learning Kafka Streams and I found a <a href="http://blog.ippon.fr/2017/04/18/kafka-streams-101/">very interesting article (in French)</a> from <a href="https://www.linkedin.com/in/william-conti-1b6374109/">William Conti</a>.<br/><br/>
In his article, William shows how to exploit the power of Kafka Streams to get the number of <a href="http://www.meetup.com">meetups</a> per city, using a KTable.<br/><br/>
I first took his repo, adjusted the sources to be compliant with the latest version of Kafka Streams, and then I thought it could be a good candidate to test KSQL, which has been <a href="https://www.confluent.io/blog/confluent-platform-4-1-with-production-ready-ksql-now-available/">officially certified for production</a> not so long ago and whose presentation by <a href="https://www.linkedin.com/in/robinmoffatt/">Robin Moffatt</a>, during a Paris Kafka Meetup made me want to play with it.<br/><br/>
That's what I'm going to demonstrate in this article.<br/>
<br/>
<h1>Start Confluent Platform</h1><br/>
First, <a href-"https://www.confluent.io/download/">download Confluent Platform <b>4.1</b></a> to be able to play with KSQL.<br/>
Follow the <a href="https://docs.confluent.io/4.1.0/control-center/docs/quickstart.html">instructions</a> to start it (if you're using defaults, it should be up in seconds).<br/><br/>
<h1>Feed a Kafka topic with RVSPs</h1><br/>
To have the full project, check <a href="https://github.com/Wconti/kstream-meetup">Williams' GitHub</a> out.<br/>
But I will only use the class "<b>Producer</b>".
<div class="custom-html-block">
<pre class="prettyprint lang-java">
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class Producer {
public static void main (String... args) throws MalformedURLException, IOException {
URL url = new URL("http://stream.meetup.com/2/rsvps");
URLConnection connection = url.openConnection();
JsonFactory jsonFactory = new JsonFactory(new ObjectMapper());
JsonParser parser = jsonFactory.createParser(connection.getInputStream());
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG , "localhost:9092");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG , StringSerializer.class.getName());
try (KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(props)) {
while (parser.nextToken() != null) {
String record = parser.readValueAsTree().toString();
<b>String updatedRecord = record.replace("\"group\"", "\"rvsp_group\"");</b>
ProducerRecord<String, String> producerRecord = new ProducerRecord<>("meetups", updatedRecord);
kafkaProducer.send(producerRecord);
}
}
}
}
</pre>
</div>
Note the instruction in bold : I had to replace the "<b>group</b>" element as it is a keyword in KSQL (as it is in SQL).<br/>
A <a href="https://github.com/confluentinc/ksql/issues/273">bug</a> has been opened on this subject.<br/><br/>
Once you've run this class, you can verify that your topic "<b>meetups</b>" is well fed, by running the console consumer.
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic meetups
</pre>
</div>
You should have some RVSPs popping.<br/>
For instance :
<div class="custom-html-block">
<pre class="prettyprint lang-json">
{
"venue": {
"venue_name": "HSBXL (Hackerspace Brussels)",
"lon": 4.328797,
"lat": 50.848236,
"venue_id": 24593226
},
"visibility": "public",
"response": "yes",
"guests": 0,
"member": {
"member_id": 35492902,
"photo": "https://secure.meetupstatic.com/photos/member/b/9/9/1/thumb_243587505.jpeg",
"member_name": "Jo"
},
"rsvp_id": 1724491047,
"mtime": 1524412889175,
"event": {
"event_name": "DIY home-automation v2",
"event_id": "249274076",
"time": 1524916800000,
"event_url": "https://www.meetup.com/hackerspace-Brussels-hsbxl/events/249274076/"
},
"<b>rvsp_group</b>": {
"group_topics": [{
"urlkey": "opensource",
"topic_name": "Open Source"
}, {
"urlkey": "robotics",
"topic_name": "Robotics"
}, {
"urlkey": "gameprog",
"topic_name": "Game Programming"
}, {
"urlkey": "softwaredev",
"topic_name": "Software Development"
}, {
"urlkey": "electronics",
"topic_name": "Electronics"
}, {
"urlkey": "microcontroller",
"topic_name": "Microcontrollers"
}, {
"urlkey": "web-development",
"topic_name": "Web Development"
}, {
"urlkey": "hacking",
"topic_name": "Hacking"
}, {
"urlkey": "arduino",
"topic_name": "Arduino"
}, {
"urlkey": "computer-programming",
"topic_name": "Computer programming"
}, {
"urlkey": "makerspaces",
"topic_name": "Makerspaces"
}, {
"urlkey": "3d-printing",
"topic_name": "3D Printing"
}],
"group_city": "Brussels",
"group_country": "be",
"group_id": 20346523,
"group_name": "Hackerspace Brussels - HSBXL",
"group_lon": 4.33,
"group_urlname": "hackerspace-Brussels-hsbxl",
"group_lat": 50.83
}
}
</pre>
</div>
We can see that the "<b>group</b>" has been successfully renamed to "<b>rvsp_group</b>".<br/><br/>
<h1>Time to bring KSQL in the game</h1><br/>
My goal is to count the number of meetups per city, as a KTable would allow, thanks to its <i>state store</i> (RocksDB).</br>
First of all, launch the <a href="https://docs.confluent.io/current/ksql/docs/installation/cli-config.html">KSQL CLI</a>.<br/><br/>
To be able to query, we first have to create a stream (in KSQL terminology, not to be mixed up with KStreams).
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
CREATE STREAM meetups ( \
venue VARCHAR, \
visibility VARCHAR, \
response VARCHAR, \
guests BIGINT, \
member VARCHAR, \
rsvp_id BIGINT, \
mtime BIGINT, \
event VARCHAR, \
rvsp_group VARCHAR \
) \
WITH (KAFKA_TOPIC='meetups', VALUE_FORMAT='JSON');
</pre>
</div>
I guess it's pretty self-explanatory : simply the definition of our first-level elements.<br/>
But note that you have to specify that our records are in JSON and are in the "meetups" topic.<br/><br/>
Then, we create another stream, which extracts only the relevant information, that is to say, cities.<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
CREATE STREAM meetup_cities AS SELECT EXTRACTJSONFIELD(rvsp_group,'$.group_city') AS city FROM meetups;
</pre>
</div>
Now we can perform some aggregation ...
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
SELECT city, COUNT(*) FROM meetup_cities GROUP BY city;
</pre>
</div>
And here is the result I was hoping for :)
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
Longmont | 3
Zürich | 16
Calgary | 44
Helsinki | 9
Essex Fells | 1
London | 392
Philadelphia | 35
Brooklyn | 19
Columbus | 17
Las Vegas | 34
</pre>
</div>
<b><u>Note</u></b> : if you want the first records to be taken into account, just update the following parameter and re-run the query.
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
ksql> SET 'auto.offset.reset' = 'earliest';
Successfully changed local property 'auto.offset.reset' from 'null' to 'earliest'
</pre>
</div>
<h1>References</h1><br/>
<ul>
<li><a href="http://blog.ippon.fr/2017/04/18/kafka-streams-101">http://blog.ippon.fr/2017/04/18/kafka-streams-101/</a></li>
<li><a href="https://docs.confluent.io/4.1.0/control-center/docs/quickstart.html">https://docs.confluent.io/4.1.0/control-center/docs/quickstart.html</a></li>
<li><a href="https://www.confluent.io/blog/using-ksql-to-analyse-query-and-transform-data-in-kafka">https://www.confluent.io/blog/using-ksql-to-analyse-query-and-transform-data-in-kafka</a> (Excellent article by <a href="https://www.linkedin.com/in/robinmoffatt/">Robin Moffatt</a>, as usual.)</li>
<li><a href="https://github.com/confluentinc/ksql/blob/0.1.x/docs/syntax-reference.md">https://github.com/confluentinc/ksql/blob/0.1.x/docs/syntax-reference.md</a></li>
</ul>
Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com0tag:blogger.com,1999:blog-2592381182145606344.post-80960534543790296732018-03-18T20:54:00.000+01:002018-03-18T20:58:17.448+01:00Simple example of using Avro in Kafka<br/><br/><div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-jL5Th4PnNQE/Wq6n45Br3PI/AAAAAAABOnE/aACQB9pthd0d09p3brNWN8mYxDtY9FJigCLcBGAs/s1600/Avro-Kafka.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-jL5Th4PnNQE/Wq6n45Br3PI/AAAAAAABOnE/aACQB9pthd0d09p3brNWN8mYxDtY9FJigCLcBGAs/s1600/Avro-Kafka.png" data-original-width="570" data-original-height="256" /></a></div><br/><br/>
Kafka has been designed to reach the best performance possible, as it is very well explained in the <a href="https://kafka.apache.org/documentation/#design">official documentation</a>.<br/>
(If you haven't read it yet, I strongly encourage you to do so).<br/><br/>
It has come pretty clear that to stay on the path of performance, some exchange formats were to be excluded, such as XML and JSON, as <a href="http://labs.criteo.com/2017/05/serialization/">nicely exposed by Criteo</a>.<br/><br/>
I then read about <a href="https://developers.google.com/protocol-buffers/">Protobuff</a>, <a href="https://thrift.apache.org/">Thrift</a> and <a href="http://avro.apache.org/">Avro</a>. But <a href="https://www.confluent.io/blog/avro-kafka-data/">the article from Confluent about Avro for Kafka Data</a> made the choice clear to me : it was going to be Avro.<br/><br/>
So, here I am, sending Avro messages to a Kafka topic.<br/><br/>
<h1>Setting up the project</h1><br/>
For that article, I will use NetBeans and Maven, as well as my <a href="http://m-button.blogspot.fr/2018/02/setting-up-kafka-on-raspberry-pi.html">Raspberry Pi cluster on which I deployed my Kafka cluster</a>.<br/>
Just create a Java Maven project.<br/><br/>
Edit the <b>pom.xml</b> so it's like the description below (or get it from <a href="https://github.com/mbutton77/kafka-playground/blob/master/pom.xml">my GitHub</a>) :
<div class="custom-html-block">
<pre class="prettyprint lang-xml">
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>fr.mbutton</groupId>
<artifactId>Kafka</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<repositories>
<repository>
<id>apache-repo</id>
<name>Apache Repository</name>
<url>https://repository.apache.org/content/repositories/releases</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<id>confluent</id>
<url>http://packages.confluent.io/maven/</url>
</repository>
</repositories>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>1.8.2</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
</goals>
<configuration>
<sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory>
<outputDirectory>${project.basedir}/src/main/java</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.0.13</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.9.0.1</version>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.8.2</version>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>2.0.0</version>
</dependency>
</dependencies>
</project>
</pre>
</div>
We're using four dependencies : <b>logback-classic</b> (logback for SL4J), <b>kafka-clients</b> + <b>avro</b>, obviously, and <b>kafka-avro-serializer</b> from Confluent.<br/><br/>
We're also configuring the Avro build plugin that will generate the Java stubs from the Avro schema.<br/><br/>
<h1>Manipulating Avro</h1><br/>
<h2>Schema definition</h2><br/>
First of all, you have to define a schema "<b>avsc</b>" which is gonna be your contract (OK, I may have worked a little bit too much with WSDL and XML).<br/>
You can also work with dynamic schema (which are only defined in your code), but that's not what I'm interested in : to me, a schema has to be shared / exposed.<br/><br/>
In our case, it's going to be a simple message.<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-yaml">
{"namespace": "fr.mbutton.avro",
"type": "record",
"name": "Message",
"fields": [
{"name": "code" , "type": "string"},
{"name": "priority", "type": ["int" , "null"]},
{"name": "extra" , "type": ["string", "null"]}
]
}
</pre>
</div>
I quote the <a href="https://avro.apache.org/docs/1.8.2/gettingstartedjava.html">Avro official documentation</a> :
<pre>
At minimum, a record definition must include its type ("type": "record"), a name ("name": "Message") [...].
Fields are defined via an array of objects, each of which defines a name and type, which is another schema object, primitive or complex type
(record, enum, array, map, union, and fixed).
<b>unions</b> are a complex type that can be any of the types listed in the array.
e.g., <i>priority</i> can either be an int or null, essentially making it an optional field.
</pre><br/>
Then, just build your Maven project and it should generate the Java source for your object. (It reminds me a lot of the dead but yet excellent <a href="https://xmlbeans.apache.org/">XMLBeans project</a>, by the way).<br/><br/>
<h2>Testing the serialization / deserialization</h2><br/>
Following the Avro documentation, I wrote a sample Java class, <i>AvroManipulator</i>, to understand how the serialization / deserialization process works.<br/><br/>
<div class="custom-html-block">
<pre class="prettyprint lang-java">
package fr.mbutton.kafka;
import fr.mbutton.avro.Message;
import java.io.File;
import java.io.IOException;
import org.apache.avro.file.DataFileReader;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.io.DatumReader;
import org.apache.avro.io.DatumWriter;
import org.apache.avro.specific.SpecificDatumReader;
import org.apache.avro.specific.SpecificDatumWriter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class AvroManipulator {
public static void main(String... args) {
Logger logger = LoggerFactory.getLogger(AvroManipulator.class);
Message message1 = new Message("Test1", 1, "This");
Message message2 = new Message("Test2", 1, "is");
Message message3 = new Message("Test3", 1, "super");
Message message4 = new Message("Test4", 1, "dope");
try {
File file = new File("messages.avro");
// Serialization
DatumWriter<Message> datumWriter = new SpecificDatumWriter<>(Message.class);
try (DataFileWriter<Message> dfWriter = new DataFileWriter<>(datumWriter)) {
dfWriter.create(message1.getSchema(), file);
dfWriter.append(message1);
dfWriter.append(message2);
dfWriter.append(message3);
dfWriter.append(message4);
}
// Deserialization
DatumReader<Message> messageDatumReader = new SpecificDatumReader<>(Message.class);
DataFileReader<Message> dfReader = new DataFileReader<>(file, messageDatumReader);
dfReader.forEach(readMessage -> logger.info(readMessage.toString()));
} catch (IOException e) {
logger.error("Something obviously went wrong !", e);
}
}
}
</pre>
</div>
Perfect ! I can now manipulate my Avro objects. <br/><br/>
<b>Note</b> : If you take a look at the file <b>messages.avro</b>, which contains the serialized version of your messages, you can see the schema description in JSON, and then your message in a binary format.<br/><br/>
<h1>Sending Avro object to a Kafka topic</h1><br/>
Based on <a href="http://aseigneurin.github.io/2016/03/02/kafka-spark-avro-kafka-101.html">Alexis Seigneurin's article</a>, I tweaked his classes to send my Avro object instead.
I just updated the bootstrap servers to reflect <a href="http://m-button.blogspot.fr/2018/02/setting-up-kafka-on-raspberry-pi.html">my Kafka/Rpi cluster</a> and replaced the original <b>StringSerializer</b> by Confluent Avro serializer / deserializer.<br/>
But when you want those Confluent objects, you also need to setup a schema registry, whose definition follows :
<pre>
Schema Registry provides a serving layer for your metadata. It provides a RESTful interface for storing and retrieving Avro schemas.
It stores a versioned history of all schemas, provides multiple compatibility settings and allows evolution of schemas according to the configured
compatibility setting. It provides serializers that plug into Kafka clients that handle schema storage and retrieval for Kafka messages
that are sent in the Avro format.
</pre>
<b><i>Source</i></b> : <a href="https://docs.confluent.io/current/schema-registry/docs/intro.html">https://docs.confluent.io/current/schema-registry/docs/intro.html</a>
<br/><br/>
Then, I configured Confluent platform, updated the <b>schema-registry.properties</b> to have my Kafka cluster configuration instead of localhost ... <br/>
<div class="custom-html-block">
<pre class="prettyprint lang-xml">
listeners=http://0.0.0.0:8081
kafkastore.connection.url=<b>192.168.0.231:2181,192.168.0.232:2181,192.168.0.233:2181,192.168.0.234:2181</b>
kafkastore.topic=_schemas
debug=false
</pre>
</div>
... and simply launched the schema registry.
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
bin/schema-registry-start etc/schema-registry/schema-registry.properties
</pre>
</div>
(By the way, I never could get it started from the GitHub repo).<br/><br/>
Be sure that the <b>bootstrap.servers</b> and <b>schema.registry.url</b> properties are correctly set up for your Java consumer and producer :
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
props.put("bootstrap.servers" , "192.168.0.231:9092,192.168.0.232:9092,192.168.0.233:9092,192.168.0.234:9092");
props.put("schema.registry.url", "http://localhost:8081");
</pre>
</div>
And that the serializer / deserializer objects are those from Confluent (io.confluent.kafka.serializers), so that your consumer looks like :<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-java">
package fr.mbutton.kafka;
import fr.mbutton.avro.Message;
import java.util.Arrays;
import java.util.Properties;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
public class AvroConsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "192.168.0.231:9092,192.168.0.232:9092,192.168.0.233:9092,192.168.0.234:9092");
props.put("group.id", "mygroup");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "io.confluent.kafka.serializers.KafkaAvroDeserializer");
props.put("schema.registry.url", "http://localhost:8081");
try (KafkaConsumer<String, Message> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(Arrays.asList("testopic"));
while (true) {
ConsumerRecords<String, Message> records = consumer.poll(100);
for (ConsumerRecord<String, Message> record : records) {
System.out.println(record.value());
}
}
}
}
}
</pre>
</div>
And your producer :
<div class="custom-html-block">
<pre class="prettyprint lang-java">
package fr.mbutton.kafka;
import fr.mbutton.avro.Message;
import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
public class AvroProducer {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "192.168.0.231:9092,192.168.0.232:9092,192.168.0.233:9092,192.168.0.234:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", "http://localhost:8081");
try (KafkaProducer<String, Message> producer = new KafkaProducer<>(props)) {
for (int i = 0; i < 100; i++) {
ProducerRecord<String, Message> record = new ProducerRecord<>("testopic", new Message("Message-" + i, 1, "extra"));
producer.send(record);
}
}
}
}
</pre>
</div>
<h1>Running the project</h1><br/>
Simply run both Java files, first the <b>AvroConsumer</b> and then the <b>AvroProducer</b>.<br/><br/>
Then, observe the console output.<br/><br/>
<u>From the producer</u>:<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
16:30:24.633 [main] DEBUG i.c.k.s.client.rest.RestService - Sending POST with input
{"schema":
"{\"type\":\"record\",
\"name\":\"Message\",
\"namespace\":\"fr.mbutton.avro\",
\"fields\":[
{\"name\":\"code\",\"type\":\"string\"},
{\"name\":\"priority\",\"type\":[\"int\",\"null\"]},
{\"name\":\"extra\",\"type\":[\"string\",\"null\"]}]}"
} to http://localhost:8081/subjects/testopic-value/versions
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
</pre>
</div>
<u>From the schema registry</u>:<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
[2018-03-18 16:30:24,647] INFO 127.0.0.1 - "POST /subjects/testopic-value/versions HTTP/1.1" 200 9 2 (io.confluent.rest-utils.requests:77)
[2018-03-18 16:30:25,100] INFO 127.0.0.1 - "GET /schemas/ids/21 HTTP/1.1" 200 246 4 (io.confluent.rest-utils.requests:77)
</pre>
</div>
<u>From the consumer</u> :<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
16:30:25.084 [main] DEBUG i.c.k.s.client.rest.RestService - Sending GET with input null to http://localhost:8081/schemas/ids/21
{"code": "Message-0", "priority": 1, "extra": "extra"}
{"code": "Message-1", "priority": 1, "extra": "extra"}
{"code": "Message-2", "priority": 1, "extra": "extra"}
[...]
{"code": "Message-97", "priority": 1, "extra": "extra"}
{"code": "Message-98", "priority": 1, "extra": "extra"}
{"code": "Message-99", "priority": 1, "extra": "extra"}
</pre>
</div>
All the code is available on <a href="https://github.com/mbutton77/kafka-playground">my GitHub</a>.<br/><br/>
<h1>References</h1>
<ol>
<li><a href="https://avro.apache.org/docs/1.8.2/gettingstartedjava.html">https://avro.apache.org/docs/1.8.2/gettingstartedjava.html</a></li>
<li><a href="http://aseigneurin.github.io/2016/03/02/kafka-spark-avro-kafka-101.html">http://aseigneurin.github.io/2016/03/02/kafka-spark-avro-kafka-101.html</a></li>
<li><a href="https://docs.confluent.io/current/schema-registry/docs/serializer-formatter.html">https://docs.confluent.io/current/schema-registry/docs/serializer-formatter.html</a></li>
<li><a href="https://github.com/gwenshap/kafka-examples/tree/master/AvroProducerExample">https://github.com/gwenshap/kafka-examples/tree/master/AvroProducerExample</a></li>
</ol>Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com0tag:blogger.com,1999:blog-2592381182145606344.post-51696046609059688962018-02-21T22:15:00.000+01:002018-02-21T22:15:52.466+01:00Setting up Kafka on a Raspberry Pi cluster via Ansible<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-apxMrnit1xA/WotIPhmTVjI/AAAAAAABNq8/DO4pA9jRAQQY8MZ0J2xpwka2QbxiAUXAgCLcBGAs/s1600/logo-article.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;">
<img border="0" src="https://1.bp.blogspot.com/-apxMrnit1xA/WotIPhmTVjI/AAAAAAABNq8/DO4pA9jRAQQY8MZ0J2xpwka2QbxiAUXAgCLcBGAs/s1600/logo-article.png" width="90%" height="90%" data-original-width="1280" data-original-height="1024" />
</a>
</div><br/>
After <a href="https://m-button.blogspot.fr/2018/02/deploying-docker-and-kubernetes-on.html">installing Docker and Kubernetes on my RPi cluster</a>, I wanted to go on, with Kafka.<br/>
<br/><br/><h1>Prerequisites</h1><br/><br/>
First of all, I assume that you have a RPi cluster already configured to work with Ansible. If not, please refer to <a href="https://m-button.blogspot.fr/2018/02/building-small-raspberry-pi-cluster-and.html">my previous article on the subject</a>.<br/><br/>
I could have refer to online archives, but as you need Java 8 and as Oracles forces you to acknowledge their license, I decided to start with archives previously downloaded.<br/>
You need :
<ol>
<li><a href="https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.0/kafka_2.11-1.0.0.tgz">Kafka 1.0.0</a></li>
<li><a href="http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html">Java 8 (recent update) for ARM</a></li>
</ol>
You will then have to fill the <i>var section</i> in the playbook with the values matching your environment.<br/>
<br/><br/><h1>Ansible Playbook</h1><br/><br/>
The goal of this playbook is to provide a way to learn about using Kafka in a cluster, without having to cope with the installation. Therefore, you can decide whether it suits your needs or not.<br/>
And if it's the case, you still can learn about the installation process by simply reading the playbook.<br/><br/>
The playbook is here for information, but if it were to be modified, it would be on <a href="https://github.com/mbutton77/ansible4raspcluster">my github</a>.
<br/><br/><h3>Zookeeper and Kafka cluster configuration</h3><br/><br/>
One thing to have in mind when reading the Ansible playbook is that my nodes have fixed IPs that are following each other.<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
[pi-cluster]
192.168.0.23[1:4]
[masters]
192.168.0.231
[slaves]
192.168.0.232
192.168.0.233
192.168.0.234
</pre>
</div>
As you can see, my first node IP last number is 1. The second one is 2 and so on. I took that as a fact when I wrote my playbook. If you have a different strategy, you'll have to modify the playbook accordingly.<br/>
This playbook is not that hard, but there is a little tricky part : the configuration of the ZooKeeper and Kafka clusters : I had to compose the cluster configuration from the information available through the Ansible inventory (= Ansible hosts).<br/><br/>
For instance, the zookeeper cluster config looks like :
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
server.1=192.168.0.231:2888:3888
server.2=192.168.0.232:2888:3888
server.3=192.168.0.233:2888:3888
server.4=192.168.0.234:2888:3888
</pre>
</div>
To reach that result, here is what I used :
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
- name: (ZOOKEEPER CONFIG) Adding cluster nodes to the zookeeper configuration
lineinfile:
path: "{{ zookeper_config }}"
line: "server.{{ item[-1] }}={{ item + ':2888:3888' }}"
insertafter: EOF
with_items:
- "{{ groups['pi-cluster'] }}"
</pre>
</div>
And regarding the Kafka config, it looks like :
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
192.168.0.231:2181,192.168.0.232:2181,192.168.0.233:2181,192.168.0.234:2181
</pre>
</div>
That was done by a simple var declaration :
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
"{{ groups['pi-cluster'] | join(':2181,') }}:2181"
</pre>
</div>
<br/><br/><h3>Idempotent, yes but how ?</h3><br/><br/>
One of Ansible great strengths is the fact that most of its command are idempotent. Yet, in certain cases, I did not use <b>lineinfile</b> because I wanted to keep the original files intact to be able to start the configuration all over again without having to go to the process of copying archives and installing them from scratch. Maybe there's a better way to do it. <br/>
If so, leave a comment or better, open a PR !
<br/><br/><h3>The playbook</h3><br/><br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
---
- name: Set Kafka up on the RPi cluster
hosts: pi-cluster
remote_user: pi
tasks:
- block:
- name: (INSTALL) Checking if Java is already installed
stat:
path: "/opt/{{ java_version }}"
register: javadir
- name: (INSTALL) Checking if Kafka is already installed
stat:
path: "/opt/{{ kafka_version }}"
register: kafkadir
- name: (INSTALL) Unarchiving Java and Kafka
unarchive:
src: "{{ item }}"
dest: /opt
owner: pi
group: pi
mode: 0755
with_items:
- "{{ java_installer_path }}"
- "{{ kafka_installer_path }}"
when: javadir.stat.exists == false or kafkadir.stat.exists == false
- name: (INSTALL) Fixing permissions for Java (unarchive user/group modification does not work with that one)
file:
path: /opt/{{ java_version }}
owner: pi
group: pi
mode: 0755
recurse: yes
when: javadir.stat.exists == false
- name: (INSTALL) Adding symbolic link for Java
file:
src: "/opt/{{ java_version }}/bin/java"
dest: /usr/bin/java
owner: pi
group: pi
state: link
when: javadir.stat.exists == false
- name: (INSTALL) Removing Kafka "windows" directory
file:
path: "/opt/{{ kafka_version }}/bin/windows"
state: absent
when: kafkadir.stat.exists == false
- name: (BACKUP) Checking if previous config backups already exist
stat:
path: "{{ item }}"
register: backup
with_items:
- "{{ zookeper_config }}.original"
- "{{ kafka_config }}.original"
- debug:
var: backup
- name: (BACKUP) Making backup copies of the zookeper and kafka config files, if never been backed up before
copy:
src: "{{ item }}"
dest: "{{ item }}.original"
owner: pi
group: pi
mode: 0755
remote_src: yes
with_items:
- "{{ zookeper_config }}"
- "{{ kafka_config }}"
when: backup.results[0].stat.exists == false
- name: (BACKUP) Restoring original file to be truly idempotent
copy:
src: "{{ item }}.original"
dest: "{{ item }}"
remote_src: true
with_items:
- "{{ zookeper_config }}"
- "{{ kafka_config }}"
when: backup.results[0].stat.exists == true
- name: (ZOOKEEPER CONFIG) Creating zookeeper work directory
file:
path: /var/zookeeper
owner: pi
group: pi
state: directory
mode: 0755
- name: (ZOOKEEPER CONFIG) Replacing the default config which sets the zookeeper workdir under var
lineinfile:
path: "{{ zookeper_config }}"
regexp: '^dataDir=.*$'
line: 'dataDir={{ zookeeper_workdir }}'
- name: (ZOOKEEPER CONFIG) Adding useful configuration
lineinfile:
path: "{{ zookeper_config }}"
line: "{{ item }}"
insertafter: EOF
with_items:
- "tickTime=2000"
- "initLimit=10"
- "syncLimit=5"
- name: (ZOOKEEPER CONFIG) Adding cluster nodes to the zookeeper configuration
lineinfile:
path: "{{ zookeper_config }}"
line: "server.{{ item[-1] }}={{ item + ':2888:3888' }}"
insertafter: EOF
with_items:
- "{{ groups['pi-cluster'] }}"
- name: (ZOOKEEPER CONFIG) Removing a previous idFile
file:
path: "{{ zookeeper_workdir }}/myid"
state: absent
- name: (ZOOKEEPER CONFIG) Creating zookeeper id file
file:
path: "{{ zookeeper_workdir }}/myid"
state: touch
owner: pi
group: pi
mode: 0755
- name: (ZOOKEEPER CONFIG) Filling id file with respecting id
lineinfile:
path: "{{ zookeeper_workdir }}/myid"
line: "{{ inventory_hostname[-1] }}"
insertafter: EOF
- name: (KAFKA CONFIG) Defining the broker ID
lineinfile:
path: "{{ kafka_config }}"
regexp: '^broker.id=.*$'
line: 'broker.id={{ inventory_hostname[-1] }}'
- name: (KAFKA CONFIG) Setting the listen address
lineinfile:
path: "{{ kafka_config }}"
regexp: '^#listeners=.*$'
line: 'listeners=PLAINTEXT://{{ inventory_hostname }}:9092'
- name: (KAFKA CONFIG) Setting the zookeeper cluster address
lineinfile:
path: "{{ kafka_config }}"
regexp: '^zookeeper.connect=.*$'
line: 'zookeeper.connect={{ zookeeper_cluster_address }}'
- name: (STARTUP) Starting ZooKeeper
shell: "nohup /opt/{{ kafka_version }}/bin/zookeeper-server-start.sh {{ zookeper_config }} &"
async: 10
poll: 0
- name: (STARTUP) Starting Kafka
shell: "nohup /opt/{{ kafka_version }}/bin/kafka-server-start.sh {{ kafka_config }} &"
async: 10
poll: 0
become: true
vars:
installer_dir: "YourPathToTheDownloadedArchives"
java_version: "jdk1.8.0_162"
kafka_version: "kafka_2.11-1.0.0"
java_installer_path: "{{ installer_dir }}/jdk-8u162-linux-arm32-vfp-hflt.tar.gz"
kafka_installer_path: "{{ installer_dir }}/{{ kafka_version }}.tgz"
zookeper_config: "/opt/{{ kafka_version }}/config/zookeeper.properties"
kafka_config: "/opt/{{ kafka_version }}/config/server.properties"
zookeeper_workdir: "/var/zookeeper"
zookeeper_cluster_address: "{{ groups['pi-cluster'] | join(':2181,') }}:2181"
</pre>
</div>
Then to run it, use the following command :<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
ansible-playbook nameOfYourFile.yml --ask-become-pass
</pre>
</div>
Then you will be prompted the password (that's what the option --ask-become-pass do) for issuing commands as root.
<br/><br/><h1>Testing the cluster</h1><br/>
Now, it's time to check that everything went smoothly. To do so, I'm going to use the test command lines shipped with the distribution.<br/><br/>
First, start a producer from any of the nodes (or even on another machine, as long as Kafka is installed) :
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
/opt/kafka_2.11-1.0.0/bin/kafka-console-producer.sh \
--broker-list 192.168.0.231:9092,192.168.0.232:9092,192.168.0.233:9092,192.168.0.234:9092 --topic testopic
</pre>
</div>
This will get you a command prompt in which anything you type will be sent across the cluster.<br/><br/>
Then, start as many consumers as you want and just observe what comes up in the terminal. (You can use something like Terminator to handle multi dynamic display, even if I never managed to get it working on my Mac)
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
/opt/kafka_2.11-1.0.0/bin/kafka-console-consumer.sh \
--zookeeper 192.168.0.231 --topic testopic from-beginning
</pre>
</div>
Then type something in the producer prompt, and it should be displayed on all consumer terminals.<br/>
<br/><br/><h1>Troubleshooting</h1><br/>
But if something does not work, I strongly suggest that you refer to these commands :
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
echo dump | nc localhost 2181
/opt/kafka_2.11-1.0.0/bin/zookeeper-shell.sh localhost:2181 <<< "get /brokers/ids/1"
</pre>
</div>
Source : <a href="https://stackoverflow.com/questions/46158296/kafka-broker-not-available-at-starting">https://stackoverflow.com/questions/46158296/kafka-broker-not-available-at-starting</a>
<br/><br/><h1>References</h1><br/>
<ul>
<li><a href="https://medium.com/@oliver_hu/set-up-a-kafka-cluster-with-raspberry-pi-2859005a9bed"><b>Set up a Kafka cluster with Raspberry Pi</b> by Oliver HU</a></li>
<li><a href="https://www.thegeekstuff.com/2016/10/zookeeper-cluster-install/"><b>Installing ZooKeeper</b> by Ramesh NATARAJAN</a></li>
</ul>Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com0tag:blogger.com,1999:blog-2592381182145606344.post-61327984004569335142018-02-04T18:43:00.000+01:002018-02-20T18:06:25.132+01:00Deploying Docker and Kubernetes on a Raspberry Pi cluster using Ansible<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-weCJqxakOvc/Wnc7oLwWcWI/AAAAAAABMkA/Wh3dXFGxRVMmZyANkIVzz-yl6ysJMv2FACLcBGAs/s1600/logo-article.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;">
<img border="0" src="https://2.bp.blogspot.com/-weCJqxakOvc/Wnc7oLwWcWI/AAAAAAABMkA/Wh3dXFGxRVMmZyANkIVzz-yl6ysJMv2FACLcBGAs/s1600/logo-article.png" width="90%" height="90%" data-original-width="1280" data-original-height="1024" />
</a>
</div><br/>
<h1>Disclaimer</h1><br/>
I would like to thank <a href="https://www.alexellis.io/">Alex Ellis</a> for his wonderful work in the container world.<br/>
From the moment I first read his blog, I wanted to build a RPi cluster and test stuff :)<br/><br/>
And what you are going to find in that article is no more no less than his work that I "translated" in a playbook.<br/>
Why Ansible ? Simply because it is probably the tool that has impressed me most these past years by its simplicity, efficiency and power.<br/>
Nothing new, nothing fancy : just a playbook I wrote for me and I thought could be interesting to others.<br/><br/>
For the playbook to run smoothly, I'm assuming you already have a configured RPi cluster and Ansible set up to interact with it. If not, you still can check <a href="https://m-button.blogspot.fr/2018/02/building-small-raspberry-pi-cluster-and.html">my previous article</a> on the topic.<br/>
If you are just willing to install Ansible, go to the <a href="https://m-button.blogspot.fr/2018/02/building-small-raspberry-pi-cluster-and.html#ansible">dedicated section</a>.<br/><br/>
<h1>The playbook</h1><br/>
This playbook, along with some others to come (I'm really eager to play with Kafka and Pulsar) is available on <a href="https://github.com/mbutton77/ansible4raspcluster">my Github</a>.<br/><br/>
It's here for you to have a quick overview, but if some changes were to occur, I don't think I will reflect them here. You've been warned.<br/><br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
---
- name: Install Docker and K8s
hosts: pi-cluster
remote_user: pi
tasks:
- block:
- name: Add encryption key for the Docker and K8s repository
apt_key:
url: "{{ item }}"
state: present
with_items:
- https://download.docker.com/linux/raspbian/gpg
- https://packages.cloud.google.com/apt/doc/apt-key.gpg
- name: Clean Docker and K8s repository files to be idempotent
file:
name: "{{ item }}"
state: absent
with_items:
- /etc/apt/sources.list.d/docker.list
- /etc/apt/sources.list.d/kubernetes.list
- name: Recreate Docker and K8s repository files
file:
name: "{{ item }}"
state: touch
with_items:
- /etc/apt/sources.list.d/docker.list
- /etc/apt/sources.list.d/kubernetes.list
- name: Add Docker and K8s repository to the list of repositories
lineinfile:
path: /etc/apt/sources.list.d/{{ item.category }}.list
line: "{{ item.url }}"
with_items:
- { url: 'deb [arch=armhf] https://download.docker.com/linux/raspbian stretch stable', category: 'docker' }
- { url: 'deb http://apt.kubernetes.io/ kubernetes-xenial main' , category: 'kubernetes' }
- name: Install packages to allow apt to use HTTPS repositories
apt:
name: "{{ item }}"
state: present
with_items:
- apt-transport-https
- ca-certificates
- software-properties-common
- name: Update list of available repositories
apt:
update_cache: yes
- name: Update all packages to the latest version
apt:
upgrade: dist
- name: Install Docker and K8s binaries
apt:
name: "{{ item }}"
state: present
with_items:
- docker-ce
- kubelet
- kubeadm
- kubectl
- kubernetes-cni
- name: Turn off swap
shell: dphys-swapfile swapoff && dphys-swapfile uninstall && update-rc.d dphys-swapfile remove
- name: Activating cgroup
lineinfile:
path: /boot/cmdline.txt
backrefs: true
regexp: '^(.*rootwait)$'
line: '\1 cgroup_enable=cpuset cgroup_memory=1'
- name: Rebooting
shell: reboot now
ignore_errors: true
become: true
</pre>
</div>
Then to run it, use the following command :
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
ansible-playbook <i>nameOfYourFile.yml</i> --ask-become-pass
</pre>
</div>
Then you will be prompted the password (that's what the option <b>--ask-become-pass</b> do) for executing commands as root.<br/><br/>
Once you've run it (without any errors I hope), Docker and Kubernetes should be installed and ready to use.<br/><br/>
Have fun !Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com0tag:blogger.com,1999:blog-2592381182145606344.post-1451504426677996152018-02-03T10:28:00.000+01:002018-02-05T22:42:19.354+01:00Building a small Raspberry Pi cluster and configure Ansible to administer it<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-pF8Bn3T1WUo/WnR3-dCxVlI/AAAAAAABMgs/D492X-FqYaY3UPMgohVm0UczvQFGL8RuwCPcBGAYYCw/s1600/20180202_153700.jpg" imageanchor="1" >
<img border="0" src="https://3.bp.blogspot.com/-pF8Bn3T1WUo/WnR3-dCxVlI/AAAAAAABMgs/D492X-FqYaY3UPMgohVm0UczvQFGL8RuwCPcBGAYYCw/s640/20180202_153700.jpg" width="387" height="640" data-original-width="968" data-original-height="1600" />
</a>
</div>
<br/><br/>
Lately, everybody has been talking about microservices, Docker, Kubernetes, Kafka, Pulsar, ... <br/>
All those subjects are highly appealing but to really be able to learn them, you have to eventually test them in some real conditions. By "<i>real</i>", I mean, you can for instance learn K8s with <a href="https://github.com/kubernetes/minikube">minikube</a>, but to see how it reacts when a node is gone, is something you truly have to experiment to correctly apprehend it. VMs are cool but RPis are fun !<br/><br/>
For that article, I was inspired by <a href="https://blog.alexellis.io/your-serverless-raspberry-pi-cluster/">Alex Ellis' blogpost</a>.<br/><br/>
<h1>Gathering the hardware</h1><br/>
As you can see in the first picture, my cluster is composed of 4 units. I bought two Raspberry Pi 3 and recycled two Raspberry Pi 2.<br/>
4 is the perfect number to me : if you take a RPi as the master, it leaves 3 units which offers multiple use cases to test. <br/>
Two is too limited. More is better but also more expensive.<br/><br/>
As I already have too many cables running around, I also bought two WiFi dongles for my RPi2.<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.amazon.fr/gp/product/B003MTTJOY/ref=as_li_ss_il?ie=UTF8&psc=1&linkCode=li1&tag=mbuton77-21&linkId=b8ebdccf4a960a6f0752c70fc5ecf773" target="_blank">
<img border="0" src="https://4.bp.blogspot.com/-1Vx-DK03ooA/WnSREcuXM7I/AAAAAAABMhM/vEuYSF3yJDEz86hyKCzrYRY2WQT0sZ9lQCLcBGAs/s320/wifi_dongle.jpg" width="200" height="200" data-original-width="1500" data-original-height="1255" />
</a>
</div>
<br/><br/>
To power the whole stuff, I found a cool alim from RavPower<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.amazon.fr/gp/product/B00NBMFX6U/ref=as_li_ss_il?ie=UTF8&psc=1&linkCode=li1&tag=mbuton77-21&linkId=3701fc52be8ce9a3a14de73673df622c" target="_blank">
<img border="0" src="https://4.bp.blogspot.com/-30SuS0MpANA/WnSRn_fddkI/AAAAAAABMhU/QJ-r4Ih2OA0EheDR_uSTq1nU1GArWzI0QCLcBGAs/s200/ravpower.jpg" width="200" height="200" data-original-width="1000" data-original-height="1000" />
</a>
</div>
<br/><br/>
And of course, a pack of USB cables<br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.amazon.fr/chargeur-vitesse-IMLEZON-Station-Android/dp/B0784JQC68/ref=as_li_ss_il?s=computers&ie=UTF8&qid=1517584415&sr=1-2&keywords=cable+micro+usb+pack+4+blanc&linkCode=li2&tag=mbuton77-21&linkId=43b680ad94266e42842a6ce34d30fa8d" target="_blank">
<img border="0" src="https://2.bp.blogspot.com/-A7216u5USag/WnScmZsTrQI/AAAAAAABMhk/EIWAFoGgsEYkCewgsT6AmI9h-CEDPXIFACLcBGAs/s200/usb_cable_pack.jpg" width="200" height="200" data-original-width="1500" data-original-height="1500" />
</a>
</div>
<br/><br/>
Once your cluster is ready from a hardware point of view, it's time to proceed to the software part.<br/><br/>
<h1>Prepare Raspbian</h1><br/><br/>
For that part, it's pretty much the same than Alex's. <br/><br/>
First, just get the Raspbian Stretch Lite image :<br/>
<a href="https://www.raspberrypi.org/downloads/raspbian/">https://www.raspberrypi.org/downloads/raspbian/</a><br/><br/>
Then flash it with <a href="https://etcher.io">Etcher.io</a><br/><br/>
Don't put it in your RPi yet ! First thing to do, before booting, is to create an empty "ssh" file in the boot partition. This will allow you to login remotely.<br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
sudo touch /boot/ssh
</pre>
</div>
Then put the card inside the RPi and boot it.<br/><br/>
<h1>Configure Raspbian</h1><br/><br/>
<h2>Keyboard layout (optional) / hostname / WIFI</h2><br/><br/>
As I'm French, my keyboard layout is AZERTY. It's really easy to switch from QWERTY (default) to the desired layout.<br/>
In fact, the three next steps are all done in the Raspbian menu.
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
sudo raspi-config
</pre>
</div>
<div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-PTu1BAY60Ag/WnTZR59bZuI/AAAAAAABMh8/L_bplmzg4DApcG09q7ivUy7H-PIc7gizACLcBGAs/s1600/raspi-config.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-PTu1BAY60Ag/WnTZR59bZuI/AAAAAAABMh8/L_bplmzg4DApcG09q7ivUy7H-PIc7gizACLcBGAs/s1600/raspi-config.gif" data-original-width="800" data-original-height="414" /></a></div><br/>
4th item <i>"Localisation Options"</i> is the part where you will be able to select the keyboard layout you want.<br/><br/>
<h2>Network 1/2</h2><br/>
As you can guess from the screenshot above, the menu <i>"Network Options"</i> (second item) will allow you to configure the hostname, the WiFi network and change the user password.<br/>
Configuring the WiFi access will update the file <b>/etc/wpa_supplicant/wpa_supplicant.conf</b> with your configuration.<br/><br/>
Once you're done, you'll be asked if you want to reboot, click "<i>Yes</i>".<br/><br/>
<h2>Network 2/2</h2><br/>
In order to make the administration of the cluster easier , we're going to set a fix IP to each node.<br/>
To do so, just update the file <b>/etc/dhcpcd.conf</b> following the example below, but pay attention to reflect your network configuration.
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
sudo vi /etc/dhcpcd.conf
interface wlan0
static ip_address=192.168.0.23X
static routers=192.168.0.1
static domain_name_servers=192.168.0.1
</pre>
</div>
When complete, reboot once again (this is the last time).<br/><br/>
<h1><a id="ansible">SSH key-based authentication</a></h1><br/><br/>
To be able to let Ansible interact with your nodes, you have to set a SSH key-based authentication for each one.
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
ssh-copy-id pi@192.168.0.23X
</pre>
</div>
Try to connect to your node(s), and you shouldn't be prompted any password.
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
ssh pi@192.168.0.23X
</pre>
</div>
Okay, we're ready to let Ansible come join the party.<br/>
(I'm expecting you to have it already installed. If not, simply follow <a href="http://docs.ansible.com/ansible/latest/intro_installation.html">the official documentation</a>.).<br/><br/>
If you're using a Linux OS, your inventory file should be <b>/etc/ansible/hosts</b> and you may then skip the next part.<br/>
Since I'm working on a Mac, I configured it manually.<br/><br/>
First, edit or create a file <b>~/.ansible.cfg</b><br/>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
# config file for ansible -- https://ansible.com/
# ===============================================
# nearly all parameters can be overridden in ansible-playbook
# or with command line flags. ansible will read ANSIBLE_CONFIG,
# ansible.cfg in the current working directory, .ansible.cfg in
# the home directory or /etc/ansible/ansible.cfg, whichever it
# finds first
[defaults]
inventory=/your/path/to/ansible/hosts
</pre>
</div>
Then, once you've defined where to find it, edit it, paste the following configuration and update the IP values.
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
[pi-cluster]
192.168.0.23[1:4]
[masters]
192.168.0.231
[slaves]
192.168.0.232
192.168.0.233
192.168.0.234
</pre>
</div>
Create a playbook (a simple YAML file) <b>test-cluster.yml</b> with the following content.
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
---
- name: Test my raspberry cluster
hosts: pi-cluster
remote_user: pi
tasks:
- name: Check Rasp model
command: cat /proc/device-tree/model
register: piversion
- name: Debug each entry of my hosts
debug:
msg: "System {{ inventory_hostname }} is a {{ piversion.stdout }}"
</pre>
</div>
And then just execute it :
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
ansible-playbook test-cluster.yml
</pre>
</div>
You should have a result such as the one below.
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
PLAY [Test my raspberry cluster] ***************************************************
TASK [Gathering Facts] *************************************************************
ok: [192.168.0.233]
ok: [192.168.0.232]
ok: [192.168.0.234]
ok: [192.168.0.231]
TASK [Check Rasp model] ************************************************************
changed: [192.168.0.233]
changed: [192.168.0.232]
changed: [192.168.0.234]
changed: [192.168.0.231]
TASK [Debug each entry of my hosts] ************************************************
ok: [192.168.0.231] => {
"msg": "System 192.168.0.231 is a Raspberry Pi 2 Model B Rev 1.1\u0000"
}
ok: [192.168.0.232] => {
"msg": "System 192.168.0.232 is a Raspberry Pi 3 Model B Rev 1.2\u0000"
}
ok: [192.168.0.233] => {
"msg": "System 192.168.0.233 is a Raspberry Pi 3 Model B Rev 1.2\u0000"
}
ok: [192.168.0.234] => {
"msg": "System 192.168.0.234 is a Raspberry Pi 2 Model B Rev 1.1\u0000"
}
PLAY RECAP *************************************************************************
192.168.0.231 : ok=3 changed=1 unreachable=0 failed=0
192.168.0.232 : ok=3 changed=1 unreachable=0 failed=0
192.168.0.233 : ok=3 changed=1 unreachable=0 failed=0
192.168.0.234 : ok=3 changed=1 unreachable=0 failed=0
</pre>
</div>
And you're done : your RPi cluster is all set up, and Ansible is ready to perform various actions on it.
Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com0tag:blogger.com,1999:blog-2592381182145606344.post-58970936688397324222017-11-18T17:51:00.000+01:002017-11-19T17:27:21.332+01:00Monitoring traceroute through Prometheus and Grafana<p dir="auto">
Last week, we faced a network outage at my client's.<br>
We tried to gather facts to know more precisely when it started, to help the netops diagnose what was wrong.</p>
<p dir="auto">And when we got all we could, it was mainly stats and figures from our application point of view, which is insufficient (and often considered as unreliable) for netops.<br>
And needless to say that we have a proper monitoring but as we do not run it, it’s not for us to analyze ...</p>
<p dir="auto">That’s where we, the project team, decided to setup our own monitoring.<br>And as I recently read a lot about Prometheus and Grafana, I decided it was high time to play with these new promising tools.<br>
I simply installed Prometheus on my machine and node exporters on the machines I wanted to monitor. Then I plugged Grafana on top and using some dashboards available from the Grafana community, I quickly displayed relevant information about my servers.</p>
<p dir="auto">But we were missing something, important to us : how to monitor the route our packets were following ?<br>
Traceroute is obviously the answer, but I didn’t want to run a traceroute manually and on top of that, I had to be capable of saying when the route had changed.</p>
<h1>Install and configure Prometheus</h1>
<p dir="ltr">Just as a reminder on how Prometheus works, here is the diagram taken from the official documentation</p><div class="separator" style="clear: both; text-align: center;">
<img src="https://camo.githubusercontent.com/7cc17b981938e40974542fbfa9c34172fd92eccd/68747470733a2f2f63646e2e7261776769742e636f6d2f70726f6d6574686575732f70726f6d6574686575732f633334323537643036396336333036383564613335626365663038343633326666643564363230392f646f63756d656e746174696f6e2f696d616765732f6172636869746563747572652e737667" width="699" style="max-width: 100%;"></div><p><br></p>
<p dir="ltr">We’re not going to use the docker image since I didn't find in the documentation how to mount a volume to be able to keep my data.<br>
No need to configure something special, but the general config file.</p>
<p dir="ltr"><b><u>Source</u></b> : <a href="https://prometheus.io/docs/prometheus/latest/installation/" target="_blank">https://prometheus.io/docs/prometheus/latest/installation/</a></p>
<p dir="ltr">The <a href="https://prometheus.io/docs/prometheus/latest/getting_started/"><i>prometheus.yml</i></a> should look like that :<br>
<div class="custom-html-block">
<pre class="prettyprint lang-bsh">
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
</pre></div>
<div class="custom-html-block">
<br>Once started, you should have the following log :
<pre class="prettyprint lang-bsh">
level=info ts=2017-11-18T16:40:40.3794356Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0, branch=HEAD)"
level=info ts=2017-11-18T16:40:40.3794809Z caller=main.go:216 build_context="(go=go1.9.2, user=root@615b82cb36b6, date=20171108-07:11:59)"
level=info ts=2017-11-18T16:40:40.3794976Z caller=main.go:217 host_details="(Linux 4.9.49-moby #1 SMP Wed Sep 27 23:17:17 UTC 2017 x86_64 (none))"
level=info ts=2017-11-18T16:40:40.3819182Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2017-11-18T16:40:40.3819422Z caller=main.go:314 msg="Starting TSDB"
level=info ts=2017-11-18T16:40:40.381969Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager..."
level=info ts=2017-11-18T16:40:40.387602Z caller=main.go:326 msg="TSDB started"
level=info ts=2017-11-18T16:40:40.3876796Z caller=main.go:394 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2017-11-18T16:40:40.389357Z caller=main.go:371 msg="Server is ready to receive requests."</pre>
</div>
<p dir="ltr">
<h1>Configuring metrics scraping</h1><p><br>Now it’s time to collect metrics.<br/>
The most usual way to do it is to use exporter (node exporter is a standard way to do it) or to code a collection point. This is the principle where Prometheus asks for metrics.<br/>
I've then setup a "<b>node_exporter</b>" on one of my Rasp (you can follow <a href="https://blog.alexellis.io/prometheus-nodeexporter-rpi/">Alex Ellis' article</a> if you want to know how) and I simply added an entry in my <i>prometheus.yml</i>, such as :<br/>
<pre class="prettyprint lang-bsh">
- job_name: rpi
static_configs:
- targets: ['192.168.0.69:9100']
</pre>
After a restart, Prometheus starts gathering information about the machine you're collecting data from.<br/>
You can verify this by using the Prometheus console, which now shows lots of metrics starting with "<b>node</b>"<br/><br/>
<a href="https://2.bp.blogspot.com/--0059L8zDT0/WhFHzDsKmsI/AAAAAAABGRM/lDZS0jVRtQgw1DU4E-nP9UWnVl3OXePHgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B09.57.19.png" imageanchor="1" >
<img border="0" data-original-width="1600" data-original-height="803" src="https://2.bp.blogspot.com/--0059L8zDT0/WhFHzDsKmsI/AAAAAAABGRM/lDZS0jVRtQgw1DU4E-nP9UWnVl3OXePHgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B09.57.19.png" width="1400" style="max-width: 100%;" />
</a>
<br></p>
<p dir="ltr">Now, what we are going to do is slightly different : we want to monitor the traceroute from the local machine (also the machine where Prometheus server is running) to the destination machine.<br>Thus, we are not going to « scrape » a distant machine but the local machine thanks to the <a href="https://github.com/prometheus/pushgateway" target="_blank" title="Project on GitHub">Pushgateway</a>, in its dockerized form.</p>
<pre class="prettyprint lang-bsh">
docker run -d -p 9091:9091 --name pushgateway prom/pushgateway
</pre>
Then let's configure Prometheus to scrape from the "<b>pushgateway</b>".
Just add the following section at the end of your <i>prometheus.yml</i> and restart Prometheus.
<pre class="prettyprint lang-yaml">
- job_name: <b>traceroute</b>
static_configs:
- targets: ['localhost:9091']
</pre>
But in order to be able to use the keyword "<b>instance</b>", you will have to add an option to the job section, in your <i>prometheus.yml</i>
<pre class="prettyprint lang-bsh">
- job_name: traceroute
<b>honor_labels: true</b>
static_configs:
- targets: ['localhost:9091']
</pre>
To know why using the keyword "<b>instance</b>" is important, I strongly encourage you to read the doc, by following the link below !<br/>
(And there's an explanation at the end of the article)<br/><br/>
<b><u>Source</b></u> : <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config" target="_blank">https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config</a><br/><br/>
We are going to test our system by sending random values.
<pre class="prettyprint lang-bsh">
max$ echo "fake_number 16" | curl --data-binary @- http://localhost:9091/metrics/job/<b>traceroute</b>
max$ echo "fake_number 11" | curl --data-binary @- http://localhost:9091/metrics/job/traceroute
max$ echo "fake_number 31" | curl --data-binary @- http://localhost:9091/metrics/job/traceroute
max$ echo "fake_number 1" | curl --data-binary @- http://localhost:9091/metrics/job/traceroute
</pre>
<b>Note that you have to define the same job name at the end of the URL and in the <i>prometheus.yml</i></b>.<br/><br/>
Now, if you look at the Prometheus console, you can see your graph with the information we added thanks to our HTTP requests.<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-QJAsRNpdqMU/WhE7TJgHCjI/AAAAAAABGQ8/jByX1rWbmlcJY1NN009zDFKtZuRlNVplgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B09.04.59.png" imageanchor="1" >
<img border="0" src="https://3.bp.blogspot.com/-QJAsRNpdqMU/WhE7TJgHCjI/AAAAAAABGQ8/jByX1rWbmlcJY1NN009zDFKtZuRlNVplgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B09.04.59.png" data-original-width="1509" data-original-height="1600" width="1400" style="max-width: 100%;" /></a><br/><br/>
</div>
Test values are fine, but we want something meaningful. Let's replace the test command by a real traceroute, but where we will just count the number of hops.<br/>
<pre class="prettyprint lang-bsh">
echo "traceroute_hops{instance=\"192.168.0.69:9100\"} $(traceroute 192.168.0.69 | grep -e "^ [0-9]" -e "^[0-9]" | wc -l)" \
| curl --data-binary @- http://localhost:9091/metrics/job/traceroute
</pre>
This line adds a metric to our job with the number of hops to reach our destination.<br/><br/>
<b>I'm aware that it is not useful to monitor a Rasp on the same LAN :) it's just for demonstration purposes</b><br/><br/>
And in order to have some history, just add this line to your crontab, with the desired frequency of execution.<br/><br/>
<h1>Pushing the monitoring to the next level with Grafana</h1>
<br/>First, let's install Grafana in its dockerized form
<pre class="prettyprint lang-bsh">
docker run -d --name=grafana -p 3000:3000 grafana/grafana
</pre>
<u><b>Source</b></u> : <a href="https://hub.docker.com/r/grafana/grafana/">https://hub.docker.com/r/grafana/grafana/</a><br/><br/>
Log in to the <a href="http://localhost:3000">Grafana console</a> (admin/admin) and click on "<b>Add data source</b>"<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-OOF6k5SSXYU/WhGBx_Z1pQI/AAAAAAABGRk/IX-OylzTLXg9hazZisa2TN3QIi71-OtpwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.05.46.png" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/-OOF6k5SSXYU/WhGBx_Z1pQI/AAAAAAABGRk/IX-OylzTLXg9hazZisa2TN3QIi71-OtpwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.05.46.png" data-original-width="1600" data-original-height="730" width="1400" style="max-width: 100%;" /></a><br/><br/>
</div>
Add the Prometheus datasource, such as :<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-XY3loLr-dlE/WhGCXXVBhrI/AAAAAAABGRw/5-Mb5X9GX_cZIsJu5PNx2CBxy4_Va_DSQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.08.19.png" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/-XY3loLr-dlE/WhGCXXVBhrI/AAAAAAABGRw/5-Mb5X9GX_cZIsJu5PNx2CBxy4_Va_DSQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.08.19.png" data-original-width="1104" data-original-height="1600" width="800" style="max-width: 100%;"/></a><br/><br/>
</div>
Then, import a <a href="https://grafana.com/dashboards/405">dashboard from the community</a> (the link points to one of the most downloaded dashboards), by first downloading the JSON file :<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-SBDJc7CNTKU/WhGFLcbFNzI/AAAAAAABGSE/DUa8rWF1gYwOB3OPiplxvAyLcPdNiXilACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.19.36.png" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-SBDJc7CNTKU/WhGFLcbFNzI/AAAAAAABGSE/DUa8rWF1gYwOB3OPiplxvAyLcPdNiXilACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.19.36.png" data-original-width="1600" data-original-height="1348" width="1000" style="max-width: 100%;"/></a>
<br/><br/>
</div>
And then by feeding Grafana with it (and then select the Prometheus data source) :<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-_pRdDOndbQc/WhGDBG7EdPI/AAAAAAABGR4/nhCSVPwNBLwz9BAKxF8b5GN_TMqbyTATgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.10.53.png" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-_pRdDOndbQc/WhGDBG7EdPI/AAAAAAABGR4/nhCSVPwNBLwz9BAKxF8b5GN_TMqbyTATgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.10.53.png" data-original-width="758" data-original-height="890" width="500" style="max-width: 100%;"/></a>
<a href="https://4.bp.blogspot.com/-UWAcjdC_tyU/WhGGArT_o4I/AAAAAAABGSM/WmAxUYOtXSQ0fXUrZ7Am8bkLHdaA13HHQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.23.30.png" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/-UWAcjdC_tyU/WhGGArT_o4I/AAAAAAABGSM/WmAxUYOtXSQ0fXUrZ7Am8bkLHdaA13HHQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.23.30.png" data-original-width="1500" data-original-height="862" width="600" style="max-width: 100%;" /></a>
<br/><br/>
</div>
<i>Et voilà</i> ! Your first dashboard ! And yes, the Rasp I'm monitoring is 2B+ :)<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-M4lljOwBxb8/WhGHLpc-QfI/AAAAAAABGSY/REa5OTrAcckAJ4imsLw3ztelOzNzzxhOgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.27.50.png" imageanchor="1" ><img border="0" src="https://2.bp.blogspot.com/-M4lljOwBxb8/WhGHLpc-QfI/AAAAAAABGSY/REa5OTrAcckAJ4imsLw3ztelOzNzzxhOgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.27.50.png" data-original-width="1600" data-original-height="1500" width="1400" style="max-width: 100%;" /></a>
<br/><br/>
</div>
Now, we need to add the last piece : the monitoring of the traceroute command.<br/>
To do so, go to the end of the page and click "<b>+ Add row</b>"<br/>
Then click on the panel title and choose "<b>Edit</b>"<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-hkxY5qvcbWM/WhGNRK95VCI/AAAAAAABGSs/dEBUZkTlfjQFZYKm7C_4hR8TJ59lKDX9ACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.37.05.png" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-hkxY5qvcbWM/WhGNRK95VCI/AAAAAAABGSs/dEBUZkTlfjQFZYKm7C_4hR8TJ59lKDX9ACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.37.05.png" data-original-width="706" data-original-height="302" width="600" style="max-width: 100%;" /></a><br/><br/>
</div>
In the "<b>Metrics</b>" tab, choose the prometheus datasource : <br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-fQQ812BsVx4/WhGNRGqyMbI/AAAAAAABGSw/M6h8ahVMPecZF40tmuFRGSHykAwbEtsAQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.37.21.png" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/-fQQ812BsVx4/WhGNRGqyMbI/AAAAAAABGSw/M6h8ahVMPecZF40tmuFRGSHykAwbEtsAQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.37.21.png" data-original-width="862" data-original-height="618" width="600" style="max-width: 100%;" /></a><br/><br/>
</div>
And then start to type "<b>traceroute</b>" in the A field. The autocompletion should do the rest. Select the name "<b>traceroute_hops</b>" :<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-G0HUex0jQh8/WhGNRNQA7pI/AAAAAAABGS0/zw-x07lI56oUAWnATSAOo4yW_NFCEuryACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.41.25.png" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/-G0HUex0jQh8/WhGNRNQA7pI/AAAAAAABGS0/zw-x07lI56oUAWnATSAOo4yW_NFCEuryACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.41.25.png" data-original-width="1542" data-original-height="790" width="600" style="max-width: 100%;" /></a><br/><br/>
</div>
Now, we have the stat based on the number of hops showing !<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-9jmZHCxEpKw/WhGNRpyDghI/AAAAAAABGS4/uCA6oA5A1ek1naZcKwYb1KhPKYTqodgsACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2011-19%2Ba%25CC%2580%2B14.43.02.png" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-9jmZHCxEpKw/WhGNRpyDghI/AAAAAAABGS4/uCA6oA5A1ek1naZcKwYb1KhPKYTqodgsACLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.43.02.png" data-original-width="1600" data-original-height="1254" width="600" style="max-width: 100%;" /></a><br/><br/>
</div>
You can change the panel title in the "<b>General</b>" tab, and the "<b>legend</b>" in the "<b>Legend Format</b>" textfield.<br/><br/>
To make it even more clear on our dashboard, we will add a "<b>single stat</b>" panel.<br/>
In the "<b>Options</b>" tab, define "<b>Stat</b>" to "<b>Current</b>", check the "<b>Background</b>" box and set the thresholds to "<b>1,1</b>", which means that the only valid value is 1.<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-wSsox4bNUVs/WhGNSJllvnI/AAAAAAABGTA/yOZrt43nBEER8zMWY9o5xBCkBIR560lBgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.50.36.png" imageanchor="1" ><img border="0" src="https://2.bp.blogspot.com/-wSsox4bNUVs/WhGNSJllvnI/AAAAAAABGTA/yOZrt43nBEER8zMWY9o5xBCkBIR560lBgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.50.36.png" data-original-width="1310" data-original-height="1600" width="600" style="max-width: 100%;" /></a><br/><br/>
</div>
If the counter differs from 1, the background turns red : <br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-gs70GDf5UwQ/WhGNSf5jItI/AAAAAAABGTE/_IC1sDT4SFMTJaa13b-eN8bkc7P5Fs-QQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.51.00.png" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-gs70GDf5UwQ/WhGNSf5jItI/AAAAAAABGTE/_IC1sDT4SFMTJaa13b-eN8bkc7P5Fs-QQCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.51.00.png" data-original-width="1261" data-original-height="1600" width="600" style="max-width: 100%;" /></a><br/><br/>
</div>
Viewed from the final dashboard (not the edition view). <br/>
KO : <br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-WpVIlk0pswo/WhGNSj6A4RI/AAAAAAABGTI/CyJkQKu1sEgoYH5OG7Rx1Uhqz-Fj6-4pwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.51.36.png" imageanchor="1" ><img border="0" src="https://2.bp.blogspot.com/-WpVIlk0pswo/WhGNSj6A4RI/AAAAAAABGTI/CyJkQKu1sEgoYH5OG7Rx1Uhqz-Fj6-4pwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.51.36.png" data-original-width="1600" data-original-height="627" width="1000" style="max-width: 100%;" /></a><br/><br/>
</div>
And then OK again : <br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-9SK1FXymJ_E/WhGNSz0ghvI/AAAAAAABGTM/oum0yjn1lrUz_mmu4hvrFdYduUeAJJ_vwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.52.33.png" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-9SK1FXymJ_E/WhGNSz0ghvI/AAAAAAABGTM/oum0yjn1lrUz_mmu4hvrFdYduUeAJJ_vwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B14.52.33.png" data-original-width="1600" data-original-height="621" width="1000" style="max-width: 100%;" /></a>
</div>
<br/><br/>
If more than one server is being monitored, we need to modify the metric query, so we only view information from the Grafana current node (the machine(s) selected from the top dropdown menu):<br/>
<pre class="prettyprint lang-bsh">
traceroute_hops<b>{instance=~'$node'}</b>
</pre>
That's the reason why we had to add the option "<b>honor_labels: true</b>". And if you enter "<b>{{instance}}</b>" in the "<b>Legend format</b>" textfield, it will display the name of the current node.<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-8YYR5DLXafA/WhGvtBe2RUI/AAAAAAABGTs/opunWsT9ApwDkNP_xZm_d3GMgLanmTXOwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B17.21.30.png" imageanchor="1" ><img border="0" src="https://2.bp.blogspot.com/-8YYR5DLXafA/WhGvtBe2RUI/AAAAAAABGTs/opunWsT9ApwDkNP_xZm_d3GMgLanmTXOwCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-19%2Ba%25CC%2580%2B17.21.30.png" data-original-width="1600" data-original-height="625" width="1000" style="max-width: 100%;" /></a>
</div>
Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com0tag:blogger.com,1999:blog-2592381182145606344.post-67639189060709020782017-11-11T09:21:00.000+01:002018-07-21T17:30:14.204+02:00Using a dockerized Nexus as a Docker registry<p dir="auto">Recently, I was playing with Docker Swarm and I decided to setup a containerized Nexus as my Docker registry.<br>But to be able to work as a Docker registry, you need to use HTTPS.<br>I found lots of article about using Nexus as a docker registry, but not a containerized Nexus. That's what I'm going to describe.<br/>A solution is to use a proxy, let it handle the security and use plain HTTP between the proxy and Nexus, but that’s not what I’m interested in.</p><h1>Install dockerized Nexus</h1><p dir="ltr">That part is the easiest one :)<br>Although it’s not the recommended way to do it, I’m going to use a mounted volume. I find it easier to manipulate what's inside that way.</p><div class="custom-html-block"><pre class="prettyprint lang-bsh">$ mkdir /some/dir/nexus-data && chown -R 200 /some/dir/nexus-data
$ docker run -d -p 8081:8081 <b>-p 8082:8082</b> --name nexus -v /some/dir/nexus-data:/nexus-data sonatype/nexus3
</pre></div><p dir="ltr">Note that, despite of the command line given in the doc, we are opening two ports since we need another one for HTTPS. Choose it carefully as it will be used after.</p><p dir="ltr"><strong><u>Source</u></strong> : <a href="https://github.com/sonatype/docker-nexus3" target="_blank">https://github.com/sonatype/docker-nexus3</a></p><p dir="ltr">Once you start Nexus, you can see the <strong>/some/dir/nexus-data</strong> directory growing.<br>Content of nexus-data :</p><div class="custom-html-block"><pre class="prettyprint lang-bsh">total 56
drwxr-xr-x 22 max staff 748B 4 nov 10:13 .
drwxr-xr-x 4 max staff 136B 2 nov 23:15 ..
-rw-r--r--@ 1 max staff 8,0K 2 nov 23:15 .DS_Store
-rw------- 1 max staff 524B 4 nov 09:49 .bash_history
-rw-rw-r-- 1 max staff 53B 2 nov 22:06 .install4j
drwxr-xr-x 3 max staff 102B 2 nov 22:06 .oracle_jre_usage
drwxr-xr-x 2 max staff 68B 2 nov 22:06 backup
drwxr-xr-x 3 max staff 102B 2 nov 22:06 blobs
drwxr-xr-x 260 max staff 8,6K 4 nov 10:14 cache
drwxr-xr-x 11 max staff 374B 2 nov 22:06 db
drwxr-xr-x 4 max staff 136B 2 nov 22:29 elasticsearch
drwxr-xr-x 6 max staff 204B 5 nov 09:53 <b>etc</b>
drwxr-xr-x 2 max staff 68B 2 nov 22:06 generated-bundles
drwxr-xr-x 2 max staff 68B 2 nov 22:06 health-check
drwxr-xr-x 3 max staff 102B 2 nov 22:06 instances
drwxr-xr-x 3 max staff 102B 2 nov 22:06 javaprefs
drwxr-xr-x 4 max staff 136B 2 nov 22:17 <b>keystores</b>
-rw-r--r-- 1 max staff 14B 4 nov 10:13 lock
drwxr-xr-x 10 max staff 340B 5 nov 01:07 <b>log</b>
drwxr-xr-x 2 max staff 68B 2 nov 22:06 orient
-rw-r--r-- 1 max staff 5B 4 nov 10:13 port
drwxr-xr-x 49 max staff 1,6K 4 nov 10:14 tmp
</pre></div><p dir="ltr">The 3 directories in bold are the ones we are going to use next.</p><h1>Adding a valid name for our certificate</h1><p dir="ltr">In real life, your certificate should match the DNS or machine on which you're hosting Nexus. <br />But in my case, it's just a local environment for testing purpose. <br>So I just added a value in <strong>/etc/hosts</strong> <br></p><div class="custom-html-block"><pre class="prettyprint lang-bsh">127.0.0.1 nexus
</pre></div><h1>Creating JKS</h1><p dir="ltr">As Jetty is the underlying web container, a JKS (Java KeyStore) is needed to contain your certificate.<br>Here, we are going to use a CSR.</p><p dir="ltr">First, go to the directory /some/dir/nexus-data/keystores and then create your CSR.</p><div class="custom-html-block"><pre class="prettyprint lang-bsh">keytool
-genkeypair
-keystore keystore.jks
-storepass changeit
-keypass changeit
-alias jetty
-keyalg RSA
-keysize 2048
-validity 5000
-dname "CN=nexus, OU=MyUnit, O=MyOrg, L=MyLoc, ST=MyState, C=FR"
-ext "SAN=DNS:nexus,IP:127.0.0.1"
-ext "BC=ca:true"
</pre></div><br/>
<p><strong><u>Source</u></strong> : <em><a href="https://support.sonatype.com/hc/en-us/articles/217542177-Using-Self-Signed-Certificates-with-Nexus-Repository-Manager-and-Docker-Daemon" target="_blank">https://support.sonatype.com/hc/en-us/articles/217542177-Using-Self-Signed-Certificates-with-Nexus-Repository-Manager-and-Docker-Daemon</a><br><br></em></p>
<h1>Updating the configuration</h1><br/><h2>jetty-https.xml</h2><p dir="ltr">Since the dockerized image of Nexus expose only the directory <strong>/nexus-data</strong> through a volume, we don’t have access to the HTTPS configuration, which is located in <strong>/opt/sonatype/nexus/etc/jetty/jetty-https.xml</strong>.<br>To do so, let’s create a writable jetty configuration directory.<br></p><div class="custom-html-block"><pre class="prettyprint lang-bsh">mkdir /some/dir/nexus-data/etc/jetty
</pre></div><p>Then start your dockerized Nexus and launch a bash prompt.<br></p><div class="custom-html-block"><pre class="prettyprint lang-bsh">docker start nexus
docker exec -it nexus bash
</pre></div><p>Once in the container, copy the file <strong>jetty-https.xml</strong> so that it’ll be accessible from the host and not deleted every time we relaunch the container.</p><div class="custom-html-block"><pre class="prettyprint lang-bsh">bash> cp /opt/sonatype/nexus/etc/jetty/jetty-https.xml /nexus-data/etc/jetty/</pre></div><p dir="ltr">Edit the file <strong>jetty-https.xml</strong> and modify the section accordingly :</p><div class="custom-html-block"><pre class="prettyprint lang-html"><new id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
<set name="KeyStorePath">/nexus-data/keystores/keystore.jks</set>
<set name="KeyStorePassword">changeit</set>
<set name="KeyManagerPassword">changeit</set>
<set name="TrustStorePath">/nexus-data/keystores/keystore.jks</set>
<set name="TrustStorePassword">changeit</set>
[ … ]</pre></div><p>As you can see, we deleted the property name « <strong>ssl.etc</strong> » since it tells jetty to look for the path under <strong>/etc/ssl</strong>, which a directory we cannot modify within the container.</p><h2>nexus.properties</h2><p>Then edit the file <strong>/some/dir/nexus-data/etc/nexus.properties</strong> to uncomment the line enabling the HTTPS port, and write the value you defined when you created your container.</p><div class="custom-html-block"><pre class="prettyprint lang-bsh">application-port-ssl=8082
</pre></div><p>Uncomment and modify the line allowing to tweak jetty configuration.<br></p><div class="custom-html-block"><pre class="prettyprint lang-xml">nexus-args=${jetty.etc}/jetty.xml,${jetty.etc}/jetty-http.xml,${jetty.etc}/jetty-requestlog.xml,<b>/nexus-data/etc/jetty/jetty-https.xml</b>
</pre></div><p dir="ltr">Then restart your container and tail the log.<br></p><div class="custom-html-block"><pre class="prettyprint lang-bsh">docker start nexus;tail -500f /some/dir/nexus-data/log/nexus.log
</pre></div><p>You should observe the correct opening of both HTTP and HTTPS ports.<br></p><div class="custom-html-block"><pre class="prettyprint lang-bsh">2017-11-04 09:14:16,900+0000 INFO [jetty-main-1] *SYSTEM
org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@276df136{HTTP/1.1,[http/1.1]}{0.0.0.0:<b>8081</b>}
2017-11-04 09:14:16,910+0000 INFO [jetty-main-1] *SYSTEM
org.eclipse.jetty.util.ssl.SslContextFactory - x509=X509@4f12e7b6(jetty,h=[nexus],w=[])
for SslContextFactory@5fc3c41c(file:///nexus-data/keystores/keystore.jks,file:///nexus-data/keystores/keystore.jks)
2017-11-04 09:14:16,946+0000 INFO [jetty-main-1] *SYSTEM
org.eclipse.jetty.server.AbstractConnector
- Started ServerConnector@287f5f5{SSL,[ssl, http/1.1]}{0.0.0.0:<b>8082</b>}
</pre></div><p><strong><u>Source</u></strong> : <em><a href="http:// https://help.sonatype.com/display/NXRM3/Configuring+SSL#ConfiguringSSL-InboundSSL-ConfiguringtoServeContentviaHTTPS" target="_blank">https://help.sonatype.com/display/NXRM3/Configuring+SSL#ConfiguringSSL-InboundSSL-ConfiguringtoServeContentviaHTTPS</a><br><br></em></p><p dir="ltr"><div class="separator" style="clear: both; text-align: center;"><img border="0" src="https://4.bp.blogspot.com/-zPCJruU_sGY/WgcdnYpZ9JI/AAAAAAABF7w/wmx2gOJf4Os-xSaHN5YEgF7rAVSDET7MgCLcBGAs/s1600/Capture%2Bd%25E2%2580%2599e%25CC%2581cran%2B2017-11-11%2Ba%25CC%2580%2B16.46.56.png" data-original-width="1316" data-original-height="619" /></div>
<p>Our certificate is self-signed, that's why it is considered not valid by the browser.</p>
<h1>Configure Docker Daemon to trust the certificate</h1>
<p>You'll now have to configure the docker daemon to trust your certificate. <br/>
I'm not going to describe something that has already have been : just look at the <a href="https://support.sonatype.com/hc/en-us/articles/217542177-Using-Self-Signed-Certificates-with-Nexus-Repository-Manager-and-Docker-Daemon">description at the end of this article</a> and you're good to go.</p>Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com0tag:blogger.com,1999:blog-2592381182145606344.post-26922396670683026192013-07-08T22:46:00.001+02:002013-07-08T22:46:53.179+02:00Book review : WLS 12c Admin Cookbook<p> </p> <p>It’s been a pretty long time since I last wrote an article …</p> <p>I’ve got some subjects in mind, but unfortunately no time to cover them !</p> <p>Anyway, I will soon review a book from Packt which sounds promising : <a href="http://www.packtpub.com/oracle-weblogic-server-12c-advanced-administration-cookbook/book">http://www.packtpub.com/oracle-weblogic-server-12c-advanced-administration-cookbook/book</a></p> <p>It’s going to be interesting : I will check the differences between 12c and good ol’ 11g :)</p> <p>Stay tuned !</p> Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com1tag:blogger.com,1999:blog-2592381182145606344.post-12355951321525021492012-09-21T16:26:00.001+02:002012-09-21T16:50:44.893+02:00Send your JMS messages to the Event Delivery Network (EDN) with a simple MDB<br />
If you have read some of my previous articles, you should know that I’m using the SOA Suite on my project as a BPM, based on the coordination of JMS messages.<br />
I naturally went for the solution proposed by Oracle : the JMS adapter. But it took me some time to figure out how to make it work :) The official documentation is complete, but when you need to really implement the concept, it’s often too light.<br />
Then I got it working, as demonstrated in a <a href="http://m-button.blogspot.fr/2011/06/how-to-connect-bpel-process-manager-on.html" target="_blank">previous blogpost</a> but it was quite disappointing. In fact, I found it quite strenuous and complex. To me, the need for JCA (and deployment plan) is not mandatory : I just wanted to KISS (keep it simple & stupid)<br />
Then I started looking for some content on that and I found some interesting stuff on <a href="http://biemond.blogspot.fr/" target="_blank">Edwin’s blog</a> & <a href="http://guidoschmutz.wordpress.com/" target="_blank">Guido’s blog</a>. But once again, I could not go with a complete implementation. Then I tried several things, tested and finally, I came up with an implementation that was satisfying (and which is used every single day in production).<br />
To keep you from going the same loong path I went, here is my code :<br />
<br />
<pre class="csharpcode">package fr.mbutton.blog.mdb;
import java.io.StringWriter;
import javax.ejb.ActivationConfigProperty;
import javax.ejb.MessageDriven;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.Queue;
import javax.jms.QueueConnectionFactory;
import javax.jms.TextMessage;
import javax.naming.InitialContext;
import javax.sql.DataSource;
import javax.transaction.UserTransaction;
import javax.xml.<span class="kwrd">namespace</span>.QName;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import oracle.fabric.blocks.<span class="kwrd">event</span>.BusinessEventConnection;
import oracle.fabric.blocks.<span class="kwrd">event</span>.BusinessEventConnectionFactory;
import oracle.fabric.common.BusinessEvent;
import oracle.integration.platform.blocks.<span class="kwrd">event</span>.BusinessEventBuilder;
import oracle.integration.platform.blocks.<span class="kwrd">event</span>.jms.JmsRemoteBusinessEventConnectionFactory;
import oracle.integration.platform.blocks.<span class="kwrd">event</span>.saq.SAQRemoteBusinessEventConnectionFactory;
import oracle.soa.common.util.XMLUtil;
import org.w3c.dom.Document;
@MessageDriven(mappedName = <span class="str">"your.topic.jndi.name"</span>,
activationConfig = {
@ActivationConfigProperty(propertyName = <span class="str">"destinationType"</span>, propertyValue = <span class="str">"javax.jms.Topic"</span>),
@ActivationConfigProperty(propertyName = <span class="str">"acknowledgeMode"</span>, propertyValue = <span class="str">"Auto-acknowledge"</span>),
@ActivationConfigProperty(propertyName = <span class="str">"topicMessagesDistributionMode"</span>, propertyValue = <span class="str">"One-Copy-Per-Application"</span>) })
<span class="kwrd">public</span> <span class="kwrd">class</span> JMSBridgeToEDN implements MessageListener {
<span class="rem">// EDN JMS environment</span>
<span class="kwrd">private</span> final String connFactName = <span class="str">"jms/fabric/EDNConnectionFactory"</span>;
<span class="kwrd">private</span> final String xaConnFactName = <span class="str">"jms/fabric/xaEDNConnectionFactory"</span>;
<span class="kwrd">private</span> final String queueName = <span class="str">"jms/fabric/EDNQueue"</span>;
<span class="kwrd">public</span> <span class="kwrd">void</span> onMessage(Message message) {
<span class="kwrd">if</span> (message instanceof TextMessage) {
TextMessage txt = (TextMessage) message;
BusinessEventConnection connection = <span class="kwrd">null</span>;
<span class="kwrd">try</span> {
<span class="rem">// You can choose between two methods : Oracle AQ or JMS</span>
<span class="rem">// Default is Oracle AQ</span>
connection = getOracleAQConnection ();
<span class="rem">// But you can choose the other one :</span>
<span class="rem">// connection = getWebLogicJMSConnection();</span>
<span class="rem">// Publish to EDN</span>
BusinessEvent <span class="kwrd">event</span> = updateAndConvertBody(txt.getText());
<span class="rem">// System.out.println("Element : " + displayXMLDoc(event.getAsDoc()));</span>
connection.publishEvent(<span class="kwrd">event</span>, 3); <span class="rem">// The number represents message priority (3 = default)</span>
} <span class="kwrd">catch</span> (Exception e) {
e.printStackTrace();
} <span class="kwrd">finally</span> {
connection.close();
}
}
}
<span class="kwrd">private</span> BusinessEventConnection getWebLogicJMSConnection () throws Exception {
InitialContext context = <span class="kwrd">new</span> InitialContext();
UserTransaction userTransaction = (UserTransaction) context.lookup(<span class="str">"javax.transaction.UserTransaction"</span>);
QueueConnectionFactory queueConnectionFactory = ((QueueConnectionFactory)context.lookup(connFactName));
QueueConnectionFactory xaQueueConnectionFactory =
((QueueConnectionFactory)context.lookup(xaConnFactName));
Queue jmsQueue = ((Queue)context.lookup(queueName));
BusinessEventConnectionFactory factory =
<span class="kwrd">new</span> JmsRemoteBusinessEventConnectionFactory(queueConnectionFactory,
xaQueueConnectionFactory,
jmsQueue,
userTransaction);
<span class="kwrd">return</span> factory.createBusinessEventConnection();
}
<span class="kwrd">private</span> BusinessEventConnection getOracleAQConnection () throws Exception {
InitialContext context = <span class="kwrd">new</span> InitialContext();
UserTransaction userTransaction = (UserTransaction) context.lookup(<span class="str">"javax.transaction.UserTransaction"</span>);
<span class="rem">// Working with both EDN datasources (as seen in class SAQRemoteBusinessEventConnectionFactory)</span>
DataSource ds = (DataSource) context.lookup(<span class="str">"jdbc/EDNDataSource"</span>);
DataSource localTxDs = (DataSource) context.lookup(<span class="str">"jdbc/EDNLocalTxDataSource"</span>);
BusinessEventConnectionFactory factory =
<span class="kwrd">new</span> SAQRemoteBusinessEventConnectionFactory(ds, localTxDs, userTransaction);
<span class="kwrd">return</span> factory.createBusinessEventConnection();
}
<span class="kwrd">private</span> BusinessEvent updateAndConvertBody (String body) throws Exception {
<span class="rem">// [...]</span>
<span class="rem">// Creation of the message to be sent over the EDN</span>
String TCEvenement = <span class="str">"TCEvenement"</span>;
<span class="rem">// A common mistake is to use the namespace of your message : IT'S NOT THE ONE YOU SHOULD USE !</span>
<span class="rem">// You have to provide the namespace of your message as defined IN THE EDL FILE !</span>
String NameSpace = <span class="str">"http://fr.edf.doaat.coctos/events/edl/EventDefinition"</span>;
BusinessEventBuilder builder = BusinessEventBuilder.newInstance();
builder.setEventName(<span class="kwrd">new</span> QName(NameSpace, TCEvenement));
builder.setBody(XMLUtil.parseDocumentFromXMLString(updatedBody).getDocumentElement());
<span class="kwrd">return</span> builder.createEvent();
}
<span class="kwrd">private</span> String displayXMLDoc (Document doc) throws Exception {
TransformerFactory transFactory = TransformerFactory.newInstance();
Transformer transformer = transFactory.newTransformer();
StringWriter buffer = <span class="kwrd">new</span> StringWriter();
transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, <span class="str">"yes"</span>);
transformer.transform(<span class="kwrd">new</span> DOMSource(doc), <span class="kwrd">new</span> StreamResult(buffer));
<span class="kwrd">return</span> buffer.toString();
}
}</pre>
<br />
The method displayXMLDoc is completely optionnal : I used it to help me debug my code but that’s all.<br />
Feel free to improve those codelines, as long as they help you, I’m glad.<br />
<br />
<div class="wlWriterEditableSmartContent" id="scid:0767317B-992E-4b12-91E0-4F059A8CECA8:5bcb8f00-2969-47b7-98c5-3fa77d8a8e8c" style="display: inline; float: none; margin: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">
Mots clés Technorati : <a href="http://technorati.com/tags/JMS" rel="tag">JMS</a>,<a href="http://technorati.com/tags/EDN" rel="tag">EDN</a>,<a href="http://technorati.com/tags/MDB" rel="tag">MDB</a>,<a href="http://technorati.com/tags/Oracle+AQ" rel="tag">Oracle AQ</a></div>
<br />
<br />
Useful links:<br />
<a href="http://docs.oracle.com/cd/E21764_01/integration.1111/e10231/adptr_jms.htm">http://docs.oracle.com/cd/E21764_01/integration.1111/e10231/adptr_jms.htm</a><br />
<a href="http://docs.oracle.com/cd/E21764_01/integration.1111/e10224/obe_intro.htm#BABHBGAG">http://docs.oracle.com/cd/E21764_01/integration.1111/e10224/obe_intro.htm#BABHBGAG</a><br />
<a href="http://guidoschmutz.wordpress.com/2010/01/12/using-the-event-api-to-publish-an-event-to-the-event-delivery-network-edn-the-spring-way/">http://guidoschmutz.wordpress.com/2010/01/12/using-the-event-api-to-publish-an-event-to-the-event-delivery-network-edn-the-spring-way/</a><br />
<a href="http://biemond.blogspot.fr/2011/06/configure-en-test-jms-based-edn-in-soa.html">http://biemond.blogspot.fr/2011/06/configure-en-test-jms-based-edn-in-soa.html</a>Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com2tag:blogger.com,1999:blog-2592381182145606344.post-79999508427621849262012-09-09T17:47:00.001+02:002012-09-09T18:39:53.016+02:00Why is my MDB not reconnecting ?<br />
I have recently encountered a problem I thought fixed for ages : the problem of a JMS consumer which is not reconnecting when the distant queue recovers from a momentary service loss.<br />
<br />
As far as I can remember, I guess the first time I noticed that problem was with WebLogic Server 7 !<br />
Now that I’m working with a recent (but not the most recent) version of WLS, I sincerely thought this bug was history …<br />
<br />
To sum up the context in which I got that error : we’re using WLS, as the center of our messaging system, and BPEL PM (which has been live for more than a month now, by the way), which is used to coordinate messages (just as WLI did).<br />
<br />
And when we had to restart WLS, I could see that no more messages were received by BPEL PM … (and you could think that the main source of the problem comes from the JCA adapter, but it’s not because I bypassed it and created my own JMS Bridge, based on a simple MDB)<br />
<br />
So, in this blogpost, I’m going to expose how the error looks like (it’s not that obvious) and what to do to get rid of it.<br />
<br />
<h1>
Context</h1>
<br />
My project is based on WLS 10.3.4 so that’s the version I’m going to use.<br />
<br />
The use case has been voluntarily simplified : it consists in two WLS domains (Producer / Consumer), with only one admin server in each domain.<br />
<br />
I’ll create a JMS queue on the Producer domain, and deploy a MDB (obviously connecting to that queue through a foreign JMS server) on the Consumer domain.<br />
<br />
<a href="http://lh4.ggpht.com/-HzXKAB5v-Rw/UEy5kuuJAUI/AAAAAAAAFCo/YOXjbeIMxfU/s1600-h/image10.png"><img alt="image" border="0" height="464" src="http://lh3.ggpht.com/-K7jCpJASUC8/UEy5logsRlI/AAAAAAAAFCs/S_OOm4EvJYg/image_thumb4.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="979" /></a><br />
<br />
Normally, everything should go according to plan. Then I’m going to restart the “Producer” domain. The behavior I’m expecting is that I should see some error messages from the “Consumer” domain, but when the distant server is back up & running, I’d like my MDB to reconnect and the error messages to stop flooding my logs.<br />
<h1>
Problem</h1>
First of all, if you’re trying to reproduce the problem yourself, please note that you may <a href="http://m-button.blogspot.fr/2009/02/working-with-two-weblogic-domains-on.html" target="_blank">change the console cookie name</a> if you don’t want to end up with a logout each time you click on a link in the other console. And you could also unable the on-demand deployment for the console.<br />
<h2>
Producer domain setup</h2>
First of all, I’ve got to create the JMS structure, composed of :<br />
<ul>
<li>a JMS server </li>
<li>a JMS module, with a sub-deployment aiming at the previously created JMS server </li>
<li>a connection factory (JNDI name : <strong>fr.mbutton.blog.jms.cf</strong>) </li>
<li>and finally a jms queue (JNDI name : <strong>fr.mbutton.blog.jms.queue</strong>) </li>
</ul>
<a href="http://lh5.ggpht.com/-JvCKGgNks1g/UEy5mlWGWOI/AAAAAAAAFC0/aDSflUzXkZ8/s1600-h/image14.png"><img alt="image" border="0" height="205" src="http://lh6.ggpht.com/-OrtNjIxitxM/UEy5npr3AkI/AAAAAAAAFC8/-gZt7eUV76g/image_thumb61.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="620" /></a><br />
<br />
That being done, I will check if everything is theoretically callable from the outside, by checking the JNDI tree.<br />
<br />
If not, you can untarget / retarget your JMS server, or restart your server. Anyway, make sure you have both your objects visible in the JNDI tree.<br />
<br />
<a href="http://lh4.ggpht.com/-MBC6_pk8Xkw/UEy5osR1pSI/AAAAAAAAFDE/8rwrn6Jucbg/s1600-h/image18.png"><img alt="image" border="0" height="279" src="http://lh6.ggpht.com/-2oyliFXQTgA/UEy5pxNKUcI/AAAAAAAAFDM/aUSlYHia19s/image_thumb81.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="698" /></a><br />
<h2>
Consumer domain setup</h2>
Once you first domain is set up, we have to configure the foreign JMS server, and deploy our MDB.<br />
<em><strong><u>Note </u></strong>: in order to be able to better understand what’s going on behind the scene, add the two options (where you want, as long as it is taken into account) : </em><br />
<blockquote>
<em>-Dweblogic.debug.DebugEjbMdbConnection=true <br />-Dweblogic.StdoutDebugEnabled=true</em></blockquote>
I’ve set up a foreign server, connecting on “t3://localhost:7001”, with a foreign connection factory and foreign jms destination.<br />
<br />
<table border="1" cellpadding="2" cellspacing="0" style="width: 641px;"><tbody>
<tr> <td valign="top" width="189"><div align="center">
<strong>Name</strong></div>
</td> <td valign="top" width="145"><div align="center">
<strong>Local JNDI</strong></div>
</td> <td valign="top" width="305"><div align="center">
<strong>Distant JNDI</strong></div>
</td> </tr>
<tr> <td valign="top" width="189">ForeignQueue</td> <td valign="top" width="145">blog.foreign.queue</td> <td valign="top" width="305">fr.mbutton.blog.jms.queue</td> </tr>
<tr> <td valign="top" width="189">ForeignConnectionFactory</td> <td valign="top" width="145">blog.foreign.cf</td> <td valign="top" width="305">fr.mbutton.blog.jms.cf</td> </tr>
</tbody></table>
<strong><u><br /></u></strong>
<strong><u>Note</u></strong> : I chosed to have a different local JNDI name for you not to mix up the domains, but I generally chose the same name.<br />
<br />
Everything normally went fine, once again, by checking the JNDI tree, you will be sure :<br />
<br />
<a href="http://lh4.ggpht.com/-nFD9tBaKmNc/UEy5rFtHHFI/AAAAAAAAFDU/5RzNkwDkWgI/s1600-h/image22.png"><img alt="image" border="0" height="281" src="http://lh4.ggpht.com/-UYFQEpOFG9o/UEy5sAXXn3I/AAAAAAAAFDc/1sitm1ZASRM/image_thumb10.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="727" /></a><br />
<br />
If a exclamation point is visible near your ressources, it proves that something is not right. Double check your configuration.<br />
<br />
Then, I created a MDB, thanks to NetBeans and since I configured my IDE to work with WLS, it should automatically propose the queue I defined, but it doesn’t work with foreign destination. It doesn’t matter : you may even skip the assistant and annotate yourself your pojo !<br />
<pre style="background-color: white; margin: 0em; width: 100%;"><span style="color: blue; font-family: consolas, Courier New, courier, monospace;"><span style="font-size: 12px;">import javax.ejb.ActivationConfigProperty;
package fr.mbutton.blog.jms;
import javax.ejb.MessageDriven;
import javax.jms.Message;
import javax.jms.MessageListener;
/**
* @author Maxence
*/
@MessageDriven(mappedName = "blog.foreign.queue", activationConfig = {
@ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge"),
@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue")
})
public class NewMessageBean implements MessageListener {
@Override
public void onMessage(Message message) {
System.out.println("A message has been received");
}
}</span></span><span style="font-family: consolas, 'Courier New', courier, monospace; font-size: 12px;">
</span></pre>
Package your MDB so that you can deploy it on your server. <br />
<br />
Once deployed, check that everything is normal in the “Monitoring” view :<br />
<br />
<br />
<a href="http://lh3.ggpht.com/-H4yaOeYdUjY/UEy5tTahp-I/AAAAAAAAFDo/Ktvub8qxrxs/s1600-h/image28.png"><img alt="image" border="0" height="395" src="http://lh4.ggpht.com/-_qBQMOe0EuM/UEy5ughd6dI/AAAAAAAAFDw/fIQtidRD99Y/image_thumb14.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="1021" /></a><br />
<br />
<br />
Connection Status should be “Connected” and status “running”.<br />
<br />
You should see, thanks to the two options we added before, some messages in your logs :<br />
<blockquote>
+++++++++++ Got JMSConnection ++++++++++ <br />
<17/04/2012#10:15:45.853#+0200> <Info> <EJB> <BEA-010060> <The Message-Driven EJB: NewMessageBean has connected/reconnected to the JMS destination: blog.foreign.queue.> <br />
<17/04/2012#10:15:45.885#+0200> <Debug> <EjbMdbConnection> <BEA-000000> <[MDConnectionManager] MDB NotificationSender: State = CONNECTED> <br />
<17/04/2012#10:15:45.886#+0200> <Debug> <EjbMdbConnection> <BEA-000000> <[MDConnectionManager] MDB NotificationSender: State = CONNECTED> <br />
<17/04/2012#10:15:45.887#+0200> <Debug> <EjbMdbConnection> <BEA-000000> <[MDConnectionManager] MDB NotificationSender: State = UNDEPLOYING> <br />
<17/04/2012#10:15:45.887#+0200> <Debug> <EjbMdbConnection> <BEA-000000> <[JMSConnectionPoller] disconnect is called> <br />
<17/04/2012#10:15:45.887#+0200> <Debug> <EjbMdbConnection> <BEA-000000> <[MDConnectionManager] MDB NotificationSender: State = UNDEPLOYING> <br />
<17/04/2012#10:15:45.888#+0200> <Debug> <EjbMdbConnection> <BEA-000000> <[MDConnectionManager] MDB NotificationSender: State = UNDEPLOYED> <br />
<17/04/2012#10:15:45.986#+0200> <Debug> <EjbMdbConnection> <BEA-000000> <[MDConnectionManager] MDB TaskNotificationSender: State = CONNECTED> <br />
<17/04/2012#10:15:45.986#+0200> <Debug> <EjbMdbConnection> <BEA-000000> <[JMSConnectionPoller] </blockquote>
Just for fun, let’s send a message : you can send it in many ways, but I’m a fan of Hermes JMS :)<br />
<br />
<br />
<a href="http://lh4.ggpht.com/-I0UWutFf_Ag/UEy5v8ycsGI/AAAAAAAAFD0/4ehBw4XehYU/s1600-h/image3.png"><img alt="image" border="0" height="420" src="http://lh4.ggpht.com/-Y3GKs39Y_z0/UEy5yGQj0VI/AAAAAAAAFEA/bJs1IF2bjCk/image_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="499" /></a><br />
<br />
Then you can check your message has been received by checking the monitoring of your MDB …<br />
<br />
<a href="http://lh5.ggpht.com/-rl_2CUULN7Y/UEy51bjT88I/AAAAAAAAFEI/exnwJyVMLe8/s1600-h/image12.png"><img alt="image" border="0" height="190" src="http://lh4.ggpht.com/-U9YF1A9pX8I/UEy52TkOO4I/AAAAAAAAFEQ/yHD4K-TtxhM/image_thumb6.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="1017" /></a><br />
<br />
… or more simply, check your server logs :<br />
<br />
<a href="http://lh5.ggpht.com/-okSFHXFCuoU/UEy53kRWMoI/AAAAAAAAFEY/oHh2g2mJk18/s1600-h/image7.png"><img alt="image" border="0" height="128" src="http://lh4.ggpht.com/-HOUZmulE4vk/UEy54y_cizI/AAAAAAAAFEc/XwL3Psb7Q68/image_thumb3.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="323" /></a><br />
<br />
Okay, everything works just fine (fortunately).<br />
<br />
Now shut down the server from the “Producer” domain.<br />
<br />
The MDB monitoring now indicates that :<br />
<br />
<a href="http://lh5.ggpht.com/-j5PLMK3m7IU/UEy558K1b6I/AAAAAAAAFEk/57c9g9MeGhw/s1600-h/image16.png"><img alt="image" border="0" height="287" src="http://lh5.ggpht.com/-F0pPWJToOf0/UEy56z8Z1UI/AAAAAAAAFEs/4iwnJTBtOwo/image_thumb8.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="464" /></a><br />
<br />
<br />
And in the logs, you should see some error messages, such as :<br />
<blockquote>
####<15/06/2012#14:53:30.638#+0000> <Debug> <EjbMdbConnection> <localhost> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1339772010638> <BEA-000000> <[MDConnectionManager] MDB NewMessageBean: State = UNDEPLOYED><br />
####<15/06/2012#14:53:30.638#+0000> <Debug> <EjbMdbConnection> <localhost> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1339772010638> <BEA-000000> <[MDConnectionManager] MDB NewMessageBean: State = UNDEPLOYED> <br />
####<15/06/2012#14:55:21.736#+0000> <Warning> <EJB> <localhost> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1339772121736> <BEA-010061> <The Message-Driven EJB: NewMessageBean is unable to connect to the JMS destination: blog.foreign.queue. The Error was: <br />
<span style="color: red;">java.lang.SecurityException: [Security:090398]Invalid Subject: principals=[weblogic, Administrators]</span> <br />
at weblogic.security.service.SecurityServiceManager.seal(SecurityServiceManager.java:833) <br />
at weblogic.security.service.IdentityUtility.authenticatedSubjectToIdentity(IdentityUtility.java:30) <br />
at weblogic.security.service.RoleManager.getRoles(RoleManager.java:183) <br />
at weblogic.security.service.AuthorizationManager.isAccessAllowed(AuthorizationManager.java:375) <br />
at weblogic.jndi.internal.ServerNamingNode.checkPermission(ServerNamingNode.java:442) <br />
at weblogic.jndi.internal.ServerNamingNode.checkLookup(ServerNamingNode.java:423) <br />
at weblogic.jndi.internal.ServerNamingNode.lookupHere(ServerNamingNode.java:180) <br />
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:206) <br />
at weblogic.jndi.internal.WLEventContextImpl.lookup(WLEventContextImpl.java:254) <br />
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:411) <br />
at javax.naming.InitialContext.lookup(InitialContext.java:392) <br />
at weblogic.jms.common.CDS$2.run(CDS.java:486) <br />
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363) <br />
at weblogic.jms.common.CrossDomainSecurityManager.runAs(CrossDomainSecurityManager.java:131) <br />
at weblogic.jms.common.CDS.lookupDestination(CDS.java:480) <br />
at weblogic.jms.common.CDS.lookupDDAndCalloutListener(CDS.java:345) <br />
at weblogic.jms.common.CDS.ddLookup(CDS.java:1393) <br />
at weblogic.jms.common.CDS.access$600(CDS.java:41) <br />
at weblogic.jms.common.CDS$DDLookupTimerListener.timerExpired(CDS.java:1267) <br />
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273) <br />
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528) <br />
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207) <br />
at weblogic.work.ExecuteThread.run(ExecuteThread.java:176) </blockquote>
At first, I was completely surprised by that stack ! What in the world have CDS (Cross Domain Security) and security to do with my MDB reconnection ?<br />
<br />
<h1>
Resolution </h1>
<div>
<br /></div>
I then open a case to the Oracle support and together, we found a similar issue which had been resolved thanks to a patch (I love when there’s a patch !).<br />
<br />
That patch is “PIEF” in case you need it. I applied it and … nothing. I triple-checked that is had been taken into account, and yes it was …<br />
<br />
<strong><u>Note</u></strong> : To play with patches, you can take a look on <a href="http://m-button.blogspot.fr/2012/09/bsu-opatch-commands.html" target="_blank">my previous blopost</a>.<br />
<blockquote>
<Jun 13, 2012 11:52:54 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING><br />
<Jun 13, 2012 11:52:54 AM UTC> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool> <br />
<13/06/2012#11:52:54.645#+0000> <Info> <WebLogicServer> <BEA-000214> <WebLogic Server "AdminServer" version: <br />
WebLogic Server Temporary Patch for <span style="color: red;">11733525</span> Thu Feb 17 12:22:04 IST 2011 <br />
WebLogic Server Temporary Patch for BUG8268373 Wed May 04 11:35:00 IST 2011 <br />
WebLogic Server 10.3.4.0.1 PSU Patch for Bug11677325 Mon Apr 11 15:38:05 IST 2011 <br />
WebLogic Server Temporary Patch for 9923849 Mon Aug 09 15:28:06 MDT 2010 <br />
WebLogic Server 10.3.4.0 Fri Dec 17 20:47:33 PST 2010 1384255 Copyright (c) 1995, 2009, Oracle and/or its affiliates. All rights reserved.> <br />
<br /></blockquote>
Well, I thought and thought and couldn’t stop thinking about the stack which was dealing with security & CDS.<br />
<br />
As I’m a bit lazy when dealing with a proper CDS configuration, I first came with the easiest solution : domain trust.<br />
<br />
I then entered the same password on both domains in the dedicated field (Domain > Security > General > Advanced > Credential)<br />
<br />
<br />
<a href="http://lh3.ggpht.com/-gEx6fNNCd1M/UEy58Gay1DI/AAAAAAAAFE4/m4QRzFmC4qc/s1600-h/image%25255B4%25255D.png"><img alt="image" border="0" height="687" src="http://lh4.ggpht.com/-C-7gAIlVHpk/UEy599bALhI/AAAAAAAAFFA/I1aegiHJeEI/image_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="982" /></a><br />
<br />
<br />
I re-tested and … VICTORY ! It solves my problem !<br />
<br />
Now, when I restart the server from the “producer” domain, the MDB disconnects & reconnects like it should have from the beginning !<br />
<br />
<blockquote>
<br />
<22/06/2012#06:20:20.593#+0000> <Debug> <EjbMdbConnection> <BEA-000000> <[JMSConnectionPoller] ** JMS Failure detected on destination:blog.foreign.queue. The exception was: weblogic.jms.common.LostServerException: java.lang.Exception: weblogic.rjvm.PeerGoneException: ; nested exception is: <br />
weblogic.utils.net.SocketResetException <br />
Nested exception: java.lang.Exception: weblogic.rjvm.PeerGoneException: ; nested exception is: <br />
weblogic.utils.net.SocketResetException <br />
Nested exception: weblogic.rjvm.PeerGoneException: ; nested exception is: <br />
weblogic.utils.net.SocketResetException <br />
Nested exception: weblogic.utils.net.SocketResetException> <br />
<22/06/2012#06:20:20.595#+0000> <Debug> <EjbMdbConnection> <BEA-000000> <[MDConnectionManager] MDB NewMessageBean: State = UNDEPLOYED> <br />
<22/06/2012#06:20:20.595#+0000> <Debug> <EjbMdbConnection> <BEA-000000> <[MDConnectionManager] MDB NewMessageBean: State = UNDEPLOYED> <br />
<22/06/2012#06:22:59.778#+0000> <Warning> <EJB> <BEA-010061> <The Message-Driven EJB: NewMessageBean is unable to connect to the JMS destination: blog.foreign.queue. The Error was: <br />
javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3://127.0.0.1:7001: Destination unreachable; nested exception is: <br />
java.net.ConnectException: Connection refused; No available router to destination] <br />
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:40) <br />
at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:788) <br />
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:366) <br />
at weblogic.jndi.Environment.getContext(Environment.java:315) <br />
at weblogic.jndi.Environment.getContext(Environment.java:285) <br />
at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117) <br />
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667) <br />
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288) <br />
at javax.naming.InitialContext.init(InitialContext.java:223) <br />
at javax.naming.InitialContext.<init>(InitialContext.java:197) <br />
at weblogic.deployment.jms.ForeignOpaqueReference.getReferent(ForeignOpaqueReference.java:182) <br />
at weblogic.jndi.internal.WLNamingManager.getObjectInstance(WLNamingManager.java:96) <br />
at weblogic.jndi.internal.ServerNamingNode.resolveObject(ServerNamingNode.java:377) <br />
at weblogic.jndi.internal.BasicNamingNode.resolveObject(BasicNamingNode.java:856) <br />
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:209) <br />
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:214) <br />
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:214) <br />
at weblogic.jndi.internal.WLEventContextImpl.lookup(WLEventContextImpl.java:254) <br />
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:411) <br />
at javax.naming.InitialContext.lookup(InitialContext.java:392) <br />
at weblogic.jms.common.CDS$2.run(CDS.java:486) <br />
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363) <br />
at weblogic.jms.common.CrossDomainSecurityManager.runAs(CrossDomainSecurityManager.java:131) <br />
at weblogic.jms.common.CDS.lookupDestination(CDS.java:480) <br />
at weblogic.jms.common.CDS.lookupDDAndCalloutListener(CDS.java:345) <br />
at weblogic.jms.common.CDS.ddLookup(CDS.java:1393) <br />
at weblogic.jms.common.CDS.access$600(CDS.java:41) <br />
at weblogic.jms.common.CDS$DDLookupTimerListener.timerExpired(CDS.java:1267) <br />
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273) <br />
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528) <br />
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207) <br />
at weblogic.work.ExecuteThread.run(ExecuteThread.java:176) <br />
Caused by: java.net.ConnectException: t3://127.0.0.1:7001: Destination unreachable; nested exception is: <br />
java.net.ConnectException: Connection refused; No available router to destination <br />
at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:216) <br />
at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:170) <br />
at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:153) <br />
at weblogic.jndi.WLInitialContextFactoryDelegate$1.run(WLInitialContextFactoryDelegate.java:345) <br />
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363) <br />
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146) <br />
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:340) <br />
... 29 more <br />
Caused by: java.rmi.ConnectException: Destination unreachable; nested exception is: <br />
java.net.ConnectException: Connection refused; No available router to destination <br />
at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:470) <br />
at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:321) <br />
at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:254) <br />
at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:197) <br />
at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:238) <br />
at weblogic.rjvm.RJVMFinder.findOrCreateRemoteCluster(RJVMFinder.java:316) <br />
at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:205) <br />
... 35 more <br />
<br />
[ ... ] And when the server is back online<br />
<br />
+++++++++++ Got JMSConnection ++++++++++ <br />
<br />
<22/06/2012#06:23:16.320#+0000> <Debug> <EjbMdbConnection> <BEA-000000> <[MDConnectionManager] MDB NewMessageBean: State = CONNECTED> <br />
<22/06/2012#06:23:16.320#+0000> <Debug> <EjbMdbConnection> <BEA-000000> <[JMSConnectionPoller]<br />
<br /></blockquote>
<h1>
Conclusion</h1>
<div>
<br /></div>
So as you can see, the problem is annoying, but the solution is not that hard to implement, as long as you know what to do. If you have any problem, I strongly suggest that you open a case to the Oracle Support. You could even find some traces of my case ;)<br />
<br />
But if you have the chance to run one of the latest versions (WLS 10.3.5 and up), everything should be fine for you ! (I successfully tested it myself)<br />
<br />
<div class="wlWriterEditableSmartContent" id="scid:0767317B-992E-4b12-91E0-4F059A8CECA8:60fc9486-c952-4f76-9c9d-b9026e0ee4fd" style="display: inline; float: none; margin: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">
Mots clés Technorati : <a href="http://technorati.com/tags/MDB" rel="tag">MDB</a>,<a href="http://technorati.com/tags/Reconnection" rel="tag">Reconnection</a>,<a href="http://technorati.com/tags/Foreign+server" rel="tag">Foreign server</a>,<a href="http://technorati.com/tags/SecurityException" rel="tag">SecurityException</a>,<a href="http://technorati.com/tags/Invalid+Subject" rel="tag">Invalid Subject</a></div>
Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com4tag:blogger.com,1999:blog-2592381182145606344.post-17590690845188216112012-09-09T17:13:00.002+02:002017-11-05T11:39:38.827+01:00BSU & OPatch commands<p><br></p><p>Just a little reminder, because I’m fed up with being forced to look for them everytime I have to work with patches …</p><h1>
BSU</h1><h2>
Listing all the current applied patches</h2><p dir="ltr"><br>[admin@machine bsu]$ <strong>./bsu.sh -view -status=applied -prod_dir=<em>yourBEAHome</em>/wlserver_10.3/</strong></p><p>ProductName: WebLogic Server <br>ProductVersion: 10.3 MP4 <br>Components: WebLogic Server/Core Application Server,WebLogic Server/Admi <br> nistration Console,WebLogic Server/Configuration Wizard and <br> Upgrade Framework,WebLogic Server/Web 2.0 HTTP Pub-Sub Serve <br> r,WebLogic Server/WebLogic SCA,WebLogic Server/WebLogic JDBC <br> Drivers,WebLogic Server/WebLogic Web Server Plugins,WebLogi <br> c Server/UDDI and Xquery Support,WebLogic Server/Workshop Co <br> de Completion Support <br>BEAHome: <em>yourBEAHome</em> <br>ProductHome: <em>yourBEAHome</em>/wlserver_10.3 <br>PatchSystemDir: <em>yourBEAHome</em>/utils/bsu <br>PatchDir: <em>yourBEAHome</em>/patch_wls1034 <br>Profile: Default <br>DownloadDir: <em>yourBEAHome</em>/utils/bsu/cache_dir <br>JavaVersion: 1.6.0_21 <br>JavaVendor: Sun</p><p>Patch ID: QWG8 (9923849) <br>Patch ID: Q8SV (11677325) <br>Patch ID: LCYX (BUG8268373) <br>Patch ID: PIEF (11733525) </p><h2>
Removing patch PIEF</h2><p>[admin@machine bsu]$ <strong>./bsu.sh -remove -patchlist=PIEF -prod_dir=<em>yourBEAHome</em>/wlserver_10.3/ </strong>Checking for conflicts.. <br>No conflict(s) detected</p><p>Removing Patch ID: PIEF. <br>Result: Success</p><h2>
Re-installing patch PIEF </h2><p>[admin@machine bsu]$ <strong>./bsu.sh -install -patchlist=PIEF -prod_dir=<em>yourBEAHome</em>/wlserver_10.3/ </strong>Checking for conflicts.. <br>No conflict(s) detected</p><p>Installing Patch ID: PIEF. <br>Result: Success</p><h2>
Get an old BEA Patch</h2><p>In My Oracle Support, choose the tab "Patches & Updates" and enter the patch ID. Then you should be proposed to download the patch.</p><h1>
OPatch</h1><h2>Pre-requisites</h2><p><a href="http://docs.oracle.com/cd/B28359_01/em.111/b31207/oui7_opatch.htm">http://docs.oracle.com/cd/B28359_01/em.111/b31207/oui7_opatch.htm</a></p><p>The OPatch utility requires the following environment:<br></p><ul>
<li> The Oracle home environment variable (<code>ORACLE_HOME</code>) must point to a valid Oracle home directory and match the value used during installation of the Oracle home directory.<br>
</li><li> JRE version 1.4 or higher, Java commands for Windows, and <code>ar</code>, <code>cp</code>, <code>fuser</code>, and <code>make</code> commands for UNIX must be made available.<br>
</li><li> The library path must be set correctly for Oracle Real Application Clusters environments. OPatch uses some APIs to detect if the system is a Real Application Clusters system. Ensure that the library path is set correctly as follows:<br>
<pre>For Solaris:
LD_LIBRARY_PATH = $ORACLE_HOME/lib32:$ORACLE_HOME/lib
For HP-UX:
SHLIB_PATH=$ORACLE_HOME/lib32:/usr/lib</pre><br> </li>
</ul><h2>Listing applied patches</h2><p><strong><em>yourBEAHome</em>/soa/OPatch/opatch lsinventory -jre <em>yourBEAHome</em>/jdk1.6_64</strong></p><p><br></p><h2>Apply a patch</h2><p><strong><em>yourBEAHome</em>/soa/OPatch/opatch apply -jre <em>yourBEAHome</em>/jdk1.6_64</strong><br></p><h2>
Uninstall a patch</h2><p><strong><em>yourBEAHome</em>/soa/OPatch/opatch rollback -id 13986504 -jre <em>yourBEAHome</em>/jdk1.6_64</strong></p><p><br></p><div class="custom-html-block"><div class="wlWriterEditableSmartContent" id="scid:0767317B-992E-4b12-91E0-4F059A8CECA8:a5f63d4c-c273-4e06-9b08-f4fcad97afd5" style="display: inline; float: none; margin: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">
Mots clés Technorati : <a href="http://technorati.com/tags/BSU" rel="tag">BSU</a>,<a href="http://technorati.com/tags/OPatch" rel="tag">OPatch</a>,<a href="http://technorati.com/tags/Oracle+Support" rel="tag">Oracle Support</a>,<a href="http://technorati.com/tags/Patches" rel="tag">Patches</a></div></div>Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com0tag:blogger.com,1999:blog-2592381182145606344.post-39898705513676614642011-12-12T15:11:00.001+01:002011-12-16T23:14:03.923+01:00WLS 12c : first steps<p> </p> <p>At last ! I’ve been waiting a very long time :)</p> <p>I wondered why WLS was so slow to come out … but ok, I know some huge work has been done on that version.</p> <h1>Waiting room </h1> <p>I was a bit surprised that Oracle did not announce it @Devoxx (since it would have been the perfect occasion) whereas some of my fellows already received an email, inviting them to attend the launch event on the 1st december. </p> <p>Speaking about that launch event, I was excited to see the WLS 12c new features and when I joined the event, the excitement quickly faded away : I was a bit bored by the first part … I am not saying it was not interesting, but honestly it felt the audience targeted was clearly people not like me. So I kept on doing stuff while listening with a distant ear to the speakers.</p> <p>Then came the second part, and that was one hundred time more interesting ! Even if the biggest feature announced was … Java EE 6 ! </p> <p>I’m not saying it’s not important, but as the Glassfish blog pointed it out recently, Java EE 6 is more than two year old now … So I guess everyone had already played with Java EE6 (with GF or another one, since WLS is the last one to be out !). I would have liked to see things more WebLogic specific.</p> <h1>Installing the product</h1> <p>I went several times on the Oracle download page to get the brand new WLS version, but everytime I had a “Check back soon” message.</p> <p>And then, finally, on Friday, it was here, ready to be downloaded : as promised, a zip version is available and its size is hard to believe : 168 Mb ! Looks amazing, compared to the Windows installer version (more than 1Gb).</p> <p>Following the instructions, I had to define my MW_HOME variable and then I could call the setWLSEnv script without any trouble. Once my environment set, I ran the script and I encountered the following exception :</p> <p><a href="http://lh6.ggpht.com/-ET2qk-e6MfM/TuvCf8fTt8I/AAAAAAAAEH8/UtnynKwhhJk/s1600-h/image%25255B12%25255D.png"><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/-mTApdrA16xs/TuvChHHK77I/AAAAAAAAEIE/oV46II64Z0w/image_thumb%25255B6%25255D.png?imgmax=800" width="1028" height="120" /></a></p> <p>Removing the JAVA_OPTION part (which I didn’t need) helped me to launch the script but I encountered another error :</p> <p><a href="http://lh4.ggpht.com/--8dLfVd3RYY/TuvCiAag2SI/AAAAAAAAEIM/iufFzZAYtd4/s1600-h/image%25255B13%25255D.png"><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="image" border="0" alt="image" src="http://lh5.ggpht.com/-8zMoillVvFM/TuvCjMO9c7I/AAAAAAAAEIU/yO4teocxVIw/image_thumb%25255B7%25255D.png?imgmax=800" width="1028" height="94" /></a></p> <p>Okay, my JAX-WS version is too old … yet, I’m using a recent JVM :</p> <blockquote> <p>java version "1.6.0_20-ea" <br />Java(TM) SE Runtime Environment (build 1.6.0_20-ea-b02) <br />Java HotSpot(TM) 64-Bit Server VM (build 17.0-b12, mixed mode)</p> </blockquote> <p>Ok, let’s update to the lattest version (update 25) … But it didn’t change a thing : I had to follow the instructions, that is to say copy the three libs into the jre/lib/endorsed directory)</p> <p><a href="http://lh6.ggpht.com/-dU-OojVc12Q/TuvCkP8y1KI/AAAAAAAAEIc/XgeOXIQUKOs/s1600-h/image%25255B14%25255D.png"><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="image" border="0" alt="image" src="http://lh5.ggpht.com/-6JSzIRVZX5E/TuvClNBpQII/AAAAAAAAEIk/HpN1yPWlOUI/image_thumb%25255B8%25255D.png?imgmax=800" width="1028" height="139" /></a></p> <p>It’s better but when I enter the username used to boot, the JVM shuts down …</p> <p>Thanks to Google, I added an option “-Dweblogic.management.allowPasswordEcho=true” and this got me unstuck</p> <p><a href="http://lh4.ggpht.com/-7PAOJ7weukU/TuvCm6jdZMI/AAAAAAAAEIs/fTbHHOpVgxY/s1600-h/image%25255B18%25255D.png"><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/-lR9v1h6-VtY/TuvCoskOzgI/AAAAAAAAEI0/LK8YWljzsU4/image_thumb%25255B10%25255D.png?imgmax=800" width="1028" height="532" /></a></p> <p>And finally, my domain is created !</p> <p><a href="http://lh4.ggpht.com/-fp33-oViQpQ/TuvCpdEHIyI/AAAAAAAAEI4/9jwN-EcteA8/s1600-h/image%25255B23%25255D.png"><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="image" border="0" alt="image" src="http://lh3.ggpht.com/-4h1NjUbQ7io/TuvCqTTabFI/AAAAAAAAEJE/xdQzcR3NVbE/image_thumb%25255B13%25255D.png?imgmax=800" width="644" height="418" /></a></p> <p>To sum up, I would say that the distro is really lighter than what I was used to but finally the installation is not as easy as I wanted it to be.</p> <p>I had to tweak the scripts, add some options (undocumented) : I guess an extra effort could have been made to simplify the initial installation.</p> <p>But now my domain has been set up, even if some things were a bit disappointing, I’m going to play with Java EE 6, on my favorite server and I’ll try to share some blogposts later on.</p> Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com5tag:blogger.com,1999:blog-2592381182145606344.post-26054754840254952462011-06-21T23:53:00.001+02:002011-06-21T23:53:54.195+02:00How to define a partnerLink endpoint from an externally defined configuration ?<p>(or how to have the same WLI XMLCache functionality in BPEL PM)</p> <h1>Part one : Get the data from the cache</h1> <p><strong><u>Source</u></strong> : <br />Excellent article from Simone : <a href="http://www.oracle.com/technetwork/topics/soa/dynamic-data-lookup-endpoints-geib-092046.html">http://www.oracle.com/technetwork/topics/soa/dynamic-data-lookup-endpoints-geib-092046.html</a> <br /></p> <p>When I read Simone’s article, I thought it was nice but I also thought that it did not go as far as I expected. In her example, she took a typical database example, with a set of data, identified by an ID. It’s a very interesting feature, but it does not meet exactly my requirement ... </p> <p>That’s why I’m going to write (with Simone’s permission :D) a little addendum which will help WLI users to transpose their good old WLI XMLCache to BPEL PM DVM. <br />The need for such a configuration is pretty obvious : I want to be able to deploy my project in several different environments without updating the code to reflect a changing server URL : typically, in a development environment, it would be an Apache and in a production environment, it would be an Alteon or a Big-Ip, for instance.</p> <p>In my case, I decided to divide my configuration into two different DVM. The first one is only composed of what’s going to change (URLs mostly …) and the other one will contain the “immutable” parts, that is to say the things that won’t change from an environment to another (context paths here).</p> <p>To know how to create a DVM, please refer to Simone’s article.</p> <p>Here is the first DVM :</p> <div align="center"> <table border="1" cellspacing="0" cellpadding="2" width="400" align="center"><tbody> <tr> <td valign="top" width="200"><strong><font color="#0000ff">Property</font></strong></td> <td valign="top" width="200"><strong><font color="#0000ff">Value</font></strong></td> </tr> <tr> <td valign="top" width="200">ServiceEntrypoint</td> <td valign="top" width="200"><a href="http://IPAddress:port">http://IPAddress:port</a></td> </tr> </tbody></table> </div> <p>To get the value of the “ServiceEntrypoint” property, I will have to use a BPEL function : <br /><strong><font color="#9b00d3">dvm:lookupValue1M("environment.dvm", "Property", 'ServiceEntrypoint', "Value")</font></strong> which will return “<a href="http://ipaddress:port/">http://ipaddress:port/</a>” <br /></p> <p>Here is the second one : </p> <div align="center"> <table border="1" cellspacing="0" cellpadding="2" width="400" align="center"><tbody> <tr> <td valign="top" width="200"><strong><font color="#0000ff">ServiceName</font></strong></td> <td valign="top" width="200"><strong><font color="#0000ff">ContextPath</font></strong></td> </tr> <tr> <td valign="top" width="200">TestWS</td> <td valign="top" width="200">/contextPath</td> </tr> </tbody></table> </div> <p>By the way, pay attention to to way you are filling data : columns are lines here :)</p> <p><a href="http://lh5.ggpht.com/-CW8bmHyrFWE/TgESueDGHVI/AAAAAAAADh8/eAAL7QwUdBs/s1600-h/image3.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; border-top-width: 0px; border-bottom-width: 0px; margin-left: auto; border-left-width: 0px; margin-right: auto; padding-top: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/-icKxB2KgQns/TgESu3Vp81I/AAAAAAAADiA/XUO--PFc138/image_thumb1.png?imgmax=800" width="502" height="406" /></a></p> <p>To get the context of my service, the function is the same as the one we used above :</p> <p><strong><font color="#9b00d3">dvm:lookupValue1M("servicepaths.dvm", "ServiceName", TestWS, "ContextPath")</font></strong> which will return “/contextPath” <br /></p> <p>Now we have the way to retrieve our two values, we can go in our BPEL Process.</p> <p>Here are the thing to perform.</p> <h2>1) Define variables </h2> <br /> <blockquote> <p>   <variable name="serviceEntrypoint" type="xsd:string"/> <br />   <variable name="contextPath" type="xsd:string"/> <br />   <variable name="endpoint" type="xsd:string"/> </p> </blockquote> <p>No magic here : a endpoint value which will receive the concatenation of the serviceEntrypoint and a contextPath.</p> <h2>2) Define a namespace in your BPEL process</h2> <blockquote> <p>xmlns:<font color="#000000"><strong>dvm</strong></font>="http://www.oracle.com/XSL/Transform/java/oracle.tip.dvm.LookupValue"</p> </blockquote> <p align="left">To be able to use DVMs. And if you are copying directly the code from the article, JDev won’t import for you the “xsd” namespace, then add it manually :</p> <blockquote> <p align="left"><a title="xmlns:xsd="http://www.w3.org/2001/XMLSchema"" href="xmlns:xsd="http://www.w3.org/2001/XMLSchema""><font color="#666666">xmlns:<strong><font color="#000000">xsd</font></strong>="http://www.w3.org/2001/XMLSchema"</font></a></p> </blockquote> <h2>3) Concatenate the information</h2> <p>In an Assign activity, concatenate the result of these two functions in a single string : </p> <blockquote> <p> <br />   <assign name="Endpoint"> <br />     <copy> <br />       <from expression='dvm:lookupValue1M("environment.dvm", "Property", "ServiceEntrypoint", "Value")'/> <br />       <to variable="serviceEntrypoint"/> <br />     </copy> <br />     <copy> <br />       <from expression='dvm:lookupValue1M("servicepaths.dvm", "ServiceName", "TestWS", "ContextPath")'/> <br />       <to variable="contextPath"/> <br />     </copy> <br />     <copy> <br />       <from expression="concat(bpws:getVariableData('serviceEntrypoint'), bpws:getVariableData('contextPath'))"/> <br />       <to variable="endpoint"/> <br />     </copy> <br />   </assign></p> </blockquote> <p>I guess the code is clear enough …</p> <h1>Part two : design a test webservice</h1> <p>I guess you know how to do it, so I won’t spend much time explaining it.</p> <p>Just design a small webservice “HelloWorld” or whatever to have something to call from our BPEL Process :)</p> <p>Once deployed, get the generated contextPath by going to the WLS console, in the section deployment, the name should be “<strong><font color="#000000">AppName-ProjectName-context-root</font></strong>”</p> <p align="center"><a href="http://lh3.ggpht.com/-xUEDAi0lQ7s/TgESvp9f-aI/AAAAAAAADiE/KZGVJ704xhc/s1600-h/image7.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px; padding-top: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/-OAOFUvYOWn8/TgESwUNTmGI/AAAAAAAADiI/6rd1C4eb0P0/image_thumb3.png?imgmax=800" width="828" height="417" /></a></p> <p align="left">Click on the “+” next to your deployment and locate your webservice and click on it.</p> <p align="left"><a href="http://lh4.ggpht.com/-ueLqh2EO874/TgESw6aHgyI/AAAAAAAADiM/RTFx9l-9e1I/s1600-h/image12.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px; padding-top: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/-Q2klQShqIXU/TgESxqU4T3I/AAAAAAAADiQ/6cVIvKBQpeI/image_thumb6.png?imgmax=800" width="1070" height="125" /></a></p> <p align="left">In the section “Testing”, click once again on the “+” just next to the webservice and you will have the choice to launch the TestClient or to display the WSDL.</p> <p align="left"><a href="http://lh5.ggpht.com/-Rii3_36AKKs/TgESyFPDb7I/AAAAAAAADiU/2ohMn7MkRow/s1600-h/image17.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; border-top-width: 0px; border-bottom-width: 0px; margin-left: auto; border-left-width: 0px; margin-right: auto; padding-top: 0px" title="image" border="0" alt="image" src="http://lh3.ggpht.com/-h6ZrPALYvq8/TgESyhO3P9I/AAAAAAAADiY/WPQpanrys08/image_thumb9.png?imgmax=800" width="853" height="322" /></a></p> <p align="left">In both cases, what’s important for us is the contextPath : <strong><font color="#000000">/Blog-XMLCache-context-root/HelloWorldPort</font></strong></p> <h1>Part three: assigning the URL to the partnerLink endpoint</h1> <p><u><strong>Sources</strong></u> :</p> <ul> <li>(thread forum Oracle) <a href="http://forums.oracle.com/forums/thread.jspa?messageID=859569">http://forums.oracle.com/forums/thread.jspa?messageID=859569</a> </li> <li>Blogpost <a href="http://blogs.oracle.com/dasoa/entry/11g_dynamic_partnerlink_example">http://blogs.oracle.com/dasoa/entry/11g_dynamic_partnerlink_example</a> </li> <li>Official doc : <a href="http://download.oracle.com/docs/cd/E19182-01/821-0539/cnfg_bpel-se-dynamic-ptnrlink_c/index.html">http://download.oracle.com/docs/cd/E19182-01/821-0539/cnfg_bpel-se-dynamic-ptnrlink_c/index.html</a> </li> </ul> <p>A big thanks to Deepak who provided the GoDynamic example which helped me go through that mist.</p> <h3>Create a simple partnerLink toward the Webservice</h3> <p>First thing of all, we need to define a partnerLink. To do so, drag a partnerLinkfrom the palette in the right swimlane. It will start an assistant : just fill the information and enter the WSDL location :</p> <p><a title="http://localhost:7001/Blog-XMLCache-context-root/HelloWorldPort?WSDL" href="http://localhost:7001/Blog-XMLCache-context-root/HelloWorldPort?WSDL">http://localhost:7001/Blog-XMLCache-context-root/HelloWorldPort?WSDL</a></p> <p>Click elsewhere and as soon as JDev detects the mouseOut event, it will display the WS port. Select the partnerRole and you’re done.</p> <p>Drag and drop an “Invoke” activity and wire it to the partnerLink :</p> <p><a href="http://lh6.ggpht.com/-qUNUOPIu6Bw/TgESzFOAW4I/AAAAAAAADic/K67UiFdeKBA/s1600-h/image25.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; border-top-width: 0px; border-bottom-width: 0px; margin-left: auto; border-left-width: 0px; margin-right: auto; padding-top: 0px" title="image" border="0" alt="image" src="http://lh5.ggpht.com/-q73ns4IEeAU/TgESzs5J4ZI/AAAAAAAADig/suiBAjlakkA/image_thumb13.png?imgmax=800" width="440" height="441" /></a></p> <p>Once done, JDev may complain since the input var has not been initialized. Add a copy operation in the Assign activty, for instance :</p> <p><a href="http://lh6.ggpht.com/-ahIGjKhIAQM/TgES0GaU3wI/AAAAAAAADik/jwQKk9BIXac/s1600-h/image32.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; border-top-width: 0px; border-bottom-width: 0px; margin-left: auto; border-left-width: 0px; margin-right: auto; padding-top: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/-30YAYXaCnGU/TgES0jjSTHI/AAAAAAAADio/utXEdkXz7nI/image_thumb16.png?imgmax=800" width="704" height="450" /></a></p> <p>But it doesn’t really matter, especially since I didn’t define any input for my webmethod. Anyway.</p> <h3>Update the partnerLink endpoint</h3> <p>To be able to change dynamically the endpoint of your partnerLink, we will need to use a XML element “EndpointReference”. That element is defined in a specific XSD which you need to import. <br /></p> <h2> </h2> <h2>1) Import the ws-addressing.xsd in your XSD directory</h2> <p>This file is located in your JDev home : <br /><strong><font color="#000000"><font color="#9b00d3">[JDEV_HOME]/jdeveloper/integration/seed/soa/shared/common/ws-addressing.xsd </font> <br /></font></strong></p> <p>For instance, if you’re using the VB appliance, this file will be there : <br /><strong><font color="#9b00d3">/oracle/jdevhome/jdeveloper/integration/seed/soa/shared/common/ws-addressing.xsd </font> <br /></strong></p> <h2> </h2> <h2>2) Define a namespace in your BPEL process</h2> <blockquote> <p>xmlns:<font color="#000000"><strong>dpl</strong></font>="http://schemas.xmlsoap.org/ws/2003/03/addressing"</p> </blockquote> <strong></strong><strong></strong> <h2> </h2> <h2>3) Define an import in the WSDL of your BPEL process</h2> <blockquote> <p><import namespace="<a href="http://schemas.xmlsoap.org/ws/2003/03/addressing">http://schemas.xmlsoap.org/ws/2003/03/addressing</a>" schemaLocation="<strong><font color="#000000">xsd/ws-addressing.xsd</font></strong>"/> </p> </blockquote> <h2>4) Declare a variable, matching your namespace prefix</h2> <blockquote> <p><variable name="endPointReference" element="<strong><font color="#000000">dpl:EndpointReference</font></strong>"/></p> </blockquote> <h2>5) Define an “Assign” activity</h2> <p>In a Assign activity, define those operations : </p> <blockquote> <p>   <assign name="PartnerLink"> <br />     <copy> <br />       <from> <br />         <EndpointReference xmlns="http://schemas.xmlsoap.org/ws/2003/03/addressing" xmlns:ns1="http://services.sap.com"> <br />           <Address/> <br />           <ReferenceProperties/> <br />           <ServiceName/> <br />         </EndpointReference> <br />       </from> <br />       <to variable="endPointReference"/> <br />     </copy> <br />     <copy> <br />       <from variable="endpoint"/> <br />       <to variable="endPointReference" query="/dpl:EndpointReference/dpl:Address"/> <br />     </copy> <br />     <copy> <br />       <from variable="endPointReference"/> <br />       <to partnerLink="HelloWorldService"/> <br />     </copy> <br />   </assign> <br /></p> </blockquote> <p>Where “endpoint” is the wanted URL (in my case, I constructed it from a double DVM lookup) </p> <h2>6) Call your partnerLink through an Invoke activity</h2> <p>Everything is in the title :)</p> <p>Now, you should have a BPEL process like that :</p> <p><a href="http://lh3.ggpht.com/-spSl0qQ2xoY/TgES1U8o-AI/AAAAAAAADis/K-Dtje3klzo/s1600-h/image%25255B5%25255D.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; border-top-width: 0px; border-bottom-width: 0px; margin-left: auto; border-left-width: 0px; margin-right: auto; padding-top: 0px" title="image" border="0" alt="image" src="http://lh5.ggpht.com/-Ok-j-reTBOY/TgES2IUD0EI/AAAAAAAADiw/TADOsh7rky8/image_thumb%25255B2%25255D.png?imgmax=800" width="1056" height="412" /></a></p> <p>And from the composite view :</p> <p align="center"><a href="http://lh6.ggpht.com/-BiwLfXTGudc/TgES2nrK1OI/AAAAAAAADi0/hmrLknvSP-o/s1600-h/image%25255B10%25255D.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px; padding-top: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/-EtNR9Bgu_R0/TgES3SjdQeI/AAAAAAAADi4/ruZ9VdldnIQ/image_thumb%25255B5%25255D.png?imgmax=800" width="734" height="439" /></a></p> <p> </p> <p>To sum up, we have a start node, then the first Assign activity, which gets the configuration information from the cache (DVM). Then a second assign activity that creates an empty “Endpoint” and copies the result of the previous assign (a concatenated string, representing the endpoint of our webservice) in the “address” property of the “Endpoint” variable.</p> <h1>Part four: testing our BPEL process</h1> <p>Okay, let’s roll ! Please note that I replaced “/contextPath” by the real context path of my service in the “ServicePaths.dvm” file.</p> <p>As my project is hybrid (composed of a webservice & of a BPEL process), you will have to deploy it twice. Once as a webservice project and the other one … ok, I guess you see the point.</p> <p><a href="http://lh3.ggpht.com/-YDXTS27-SRM/TgES3zgyomI/AAAAAAAADi8/00M84Swsq-4/s1600-h/image%25255B14%25255D.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; border-top-width: 0px; border-bottom-width: 0px; margin-left: auto; border-left-width: 0px; margin-right: auto; padding-top: 0px" title="image" border="0" alt="image" src="http://lh3.ggpht.com/-_Q5vRHC8hSM/TgES4lq7wAI/AAAAAAAADjA/QV6c9Lvu_AE/image_thumb%25255B7%25255D.png?imgmax=800" width="744" height="269" /></a></p> <p> </p> <p>Once done, go on the EM and test your BPEL process.</p> <p>Normally, you should have a successful test :</p> <p><a href="http://lh5.ggpht.com/-4pyGR--NQxU/TgES41DY3wI/AAAAAAAADjE/AifH3ZZNK0Q/s1600-h/image%25255B24%25255D.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; border-top-width: 0px; border-bottom-width: 0px; margin-left: auto; border-left-width: 0px; margin-right: auto; padding-top: 0px" title="image" border="0" alt="image" src="http://lh3.ggpht.com/-NXNwioSYSRw/TgES5iRrOqI/AAAAAAAADjI/FPvuYUNzYqM/image_thumb%25255B11%25255D.png?imgmax=800" width="853" height="145" /></a></p> <p>But now, if we change the server port to “8001”, in the “environment” DVM (either directly in the code, or through the SOA Composer), let’s see what’s happening :</p> <p><a href="http://lh4.ggpht.com/-utj33HokUyo/TgES6fmFPHI/AAAAAAAADjM/rtre6kH0O2k/s1600-h/image%25255B25%25255D.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; border-top-width: 0px; border-bottom-width: 0px; margin-left: auto; border-left-width: 0px; margin-right: auto; padding-top: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/-2C5XYxX5Zdw/TgES7A4SdLI/AAAAAAAADjQ/uj8LDAAOuMY/image_thumb%25255B12%25255D.png?imgmax=800" width="941" height="428" /></a></p> <p>Or if we take a closer look to the “DynamicPartnerLink” :</p> <p><a href="http://lh6.ggpht.com/-mPtB1krMRbs/TgES7ncKV6I/AAAAAAAADjU/gEG-ZJg7NlI/s1600-h/image%25255B26%25255D.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; border-top-width: 0px; border-bottom-width: 0px; margin-left: auto; border-left-width: 0px; margin-right: auto; padding-top: 0px" title="image" border="0" alt="image" src="http://lh5.ggpht.com/-uFLXSgUKBOk/TgES8WcyokI/AAAAAAAADjY/vZBh5PGJuQI/image_thumb%25255B13%25255D.png?imgmax=800" width="966" height="526" /></a></p> <p> </p> <p>So, WLI users, here is a “simple” way to replace your good old XMLCache ! Hope it will help some WLI fellows :)</p> <p> </p> <div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:0767317B-992E-4b12-91E0-4F059A8CECA8:dcdeca9a-e808-4610-8685-e96b48f38e05" class="wlWriterEditableSmartContent">Mots clés Technorati : <a href="http://technorati.com/tags/wli+xmlcache" rel="tag">wli xmlcache</a>,<a href="http://technorati.com/tags/domain+value+map" rel="tag">domain value map</a>,<a href="http://technorati.com/tags/DVM" rel="tag">DVM</a>,<a href="http://technorati.com/tags/cache" rel="tag">cache</a>,<a href="http://technorati.com/tags/dynamic+endpoint" rel="tag">dynamic endpoint</a>,<a href="http://technorati.com/tags/partnerLink" rel="tag">partnerLink</a></div> Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com1tag:blogger.com,1999:blog-2592381182145606344.post-46903538318505988392011-06-16T22:27:00.003+02:002011-06-16T22:47:17.415+02:00How to connect BPEL Process Manager on a JMS topic as a durable subscriber ?On my project, I use BPEL PM as the central BPM. It uses JMS messages to integrate external information. And, at a particular moment of the day, my process is launching some tasks and I want it to freeze the reception of external messages, to be sure my data is not changed during the computing process.<br />
When I was using WLI, I took a closer look at some of the BEA stuff to know how they were passivating / resuming EventGenerators through the WLI console. It gave me a <a href="http://m-button.blogspot.com/2009/09/how-to-suspend-resume-wli-81.html" target="_blank">working solution</a> (still in production), but it wasn’t official.<br />
<br />
Then, it hit me : isn’t the exact purpose of JMS topics & durable subscribers functionality ? Of course it is, but as the project was up & running in production, I didn’t get the authorization to change the existing system : we do not change something that’s working.<br />
Alright, I can understand. I then kept that idea in my mind … till we had to break up our architecture to replace WLI by BPEL PM. Then, I thought it was the perfect occasion. Then I decided to go for that solution I’ve been thinking about over six months !<br />
<br />
My stuff is now working and I thought it could help some people who need the same thing. Here’s what we are going to do in this article :<br />
<ol><li>Design a simple process that listens through a mediator to the publishing of a message on the EDN </li>
<li>Test that process with the Enterprise Manager </li>
<li>Configure the JCA Adapter in charge of JMS and plug our process to a JMS topic </li>
<li>Configure the JCA Adapter to make BPEL PM a durable subscriber </li>
<li>Test how to deconnect BPEL PM from the topic </li>
<li>Test that JMS messages are kept when BPEL PM is offline, and that they are correctly sent when BPEL PM comes back online. </li>
</ol>All the example I will be describing is going to be made thanks to the <a href="http://www.oracle.com/technetwork/middleware/soasuite/learnmore/vmsoa-172279.html" target="_blank">SOA Suite 11gR1PS2 Virtual Box appliance</a>.<br />
<h1>Design time</h1><h3>BPEL Process 1 & its Mediator : EDN message consumer</h3>That BPEL process is very simple : the message subscription is going to be performed by a mediator, which will, on the reception of a message, call the BPEL process.<br />
I will spare you the designing of such a process. <br />
Just know the message I’m using is following that schema definition (the same I used in that <a href="http://m-button.blogspot.com/2010/12/migrating-wli-process-toward-bpel.html" target="_blank">blogpost</a>) :<br />
<br />
<pre style="background-color: #fbfbfb; border-bottom: #cecece 1px solid; border-left: #cecece 1px solid; border-right: #cecece 1px solid; border-top: #cecece 1px solid; min-height: 40px; overflow: auto; padding-bottom: 5px; padding-left: 5px; padding-right: 5px; padding-top: 5px; width: 650px;"><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><?</span>xml version="1.0"<span style="color: blue;">?></span>
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">schema</span></pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: red;">xmlns</span>:<span style="color: red;">xs</span>=<span style="color: blue;"><a href="http://www.w3.org/2001/XMLSchema">http://www.w3.org/2001/XMLSchema</a></span></pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: red;">xmlns</span>:<span style="color: red;">tns</span>=<span style="color: blue;"><a href="http://temp.openuri.org/WLIMigrationToBPEL/sample.xsd">http://temp.openuri.org/WLIMigrationToBPEL/sample.xsd</a></span></pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: red;">targetNamespace</span>=<span style="color: blue;">"http://fr.mbutton.blog/WLIMigrationToBPEL"</span></pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: red;">elementFormDefault</span>=<span style="color: blue;">"qualified"</span></pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: red;">attributeFormDefault</span>=<span style="color: blue;">"unqualified"</span><span style="color: blue;">></span></pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">element</span> <span style="color: red;">name</span>=<span style="color: blue;">"message"</span><span style="color: blue;">></span></pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">complexType</span><span style="color: blue;">></span></pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">sequence</span><span style="color: blue;">></span></pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">element</span> <span style="color: red;">name</span>=<span style="color: blue;">"id"</span> <span style="color: red;">type</span>=<span style="color: blue;">"xs:string"</span><span style="color: blue;">/></span></pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">element</span> <span style="color: red;">name</span>=<span style="color: blue;">"label"</span> <span style="color: red;">type</span>=<span style="color: blue;">"xs:string"</span><span style="color: blue;">/></span></pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"></</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">sequence</span><span style="color: blue;">></span></pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"></</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">complexType</span><span style="color: blue;">></span></pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"></</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">element</span><span style="color: blue;">></span></pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"></</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">schema</span><span style="color: blue;">></span>
</pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"></pre></pre>I’ve updated the WSDL process to use the message as an input, declared a correlation set and defined a static routing rule to route any incoming message to my BPEL process.<br />
And in the end, the composite will look like that :<br />
<br />
<a href="http://lh3.ggpht.com/-qVsENEuNAf4/Tfpl6luSqMI/AAAAAAAADcY/LlJykXDQVb4/s1600-h/image3.png"><img alt="image" border="0" height="361" src="http://lh4.ggpht.com/-3id18OGzRAs/Tfpl7EZJE2I/AAAAAAAADcc/gzf-S2FZDpY/image_thumb1%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="379" /></a><br />
<h3></h3><h3>BPEL Process 2 : EDN message sender</h3>A simple BPEL Process whose role is to send (through an Invoke activity) a message over the EDN.<br />
The composite will look like that :<br />
<br />
<a href="http://lh3.ggpht.com/-5y4xCrRiKbE/Tfpl73iFcII/AAAAAAAADcg/XP5HLxq6K_4/s1600-h/image7.png"><img alt="image" border="0" height="570" src="http://lh3.ggpht.com/-fUT09xdrC6Y/Tfpl8R9nzJI/AAAAAAAADck/1ozTDNljHCA/image_thumb3.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="490" /></a><br />
<br />
Once everything’s ready, just deploy it to the WLS server.<br />
<h1>First test</h1>As our process is not yet wired to the real world (that is to say JMS), I’ll use the first BPEL process to push messages over the EDN. That way, it will allow me to see if my BPEL process is correctly triggered.<br />
Thanks to the Enterprise Manager, I’ll use the test interface on my EDNSender BPEL process to trigger message.<br />
<br />
<a href="http://lh5.ggpht.com/-lIqSEfXO5nc/Tfpl829zpVI/AAAAAAAADco/QcNqOgrw3aU/s1600-h/image4.png"><img alt="image" border="0" height="309" src="http://lh3.ggpht.com/-I_Fc5KDcQ70/Tfpl9nuORkI/AAAAAAAADcs/fMWWsZWeFaY/image_thumb11.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="900" /></a><br />
<br />
Note : I could have used the embedded message functionality (right click on “soa-infra (AdminServer)" ><br />
Business Events”, but I prefer my own tool :)<br />
<br />
<div align="center"><a href="http://lh3.ggpht.com/-jpLfa13fhqc/Tfpl-EF4OzI/AAAAAAAADcw/bxoMza91WhE/s1600-h/image12.png"><img alt="image" border="0" height="410" src="http://lh5.ggpht.com/-squJya9L4dg/Tfpl-zeLlvI/AAAAAAAADc0/abuzlcfZKnI/image_thumb5.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="787" /></a></div><br />
But okay, just send a message (one way or another) and see what happens :<br />
<br />
<a href="http://lh6.ggpht.com/-d2jvCn8iEHo/Tfpl_mtd-2I/AAAAAAAADc4/vqqujVvZxQs/s1600-h/image16.png"><img alt="image" border="0" height="191" src="http://lh4.ggpht.com/-bCiD5R5bCv8/TfpmAqa7c0I/AAAAAAAADc8/aHuOcV2MAHI/image_thumb7.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="974" /></a><br />
<br />
And when I click on the MessageEater instance, I see that the EDN message has been properly consumed :<br />
<br />
<a href="http://lh6.ggpht.com/-Sj1lyM397Lk/TfpmBbNmhEI/AAAAAAAADdA/pk6Lj1eWPUE/s1600-h/image20.png"><img alt="image" border="0" height="551" src="http://lh3.ggpht.com/-0M-7UCRcUGw/TfpmCMLM7ZI/AAAAAAAADdE/xhYT6ywQRyk/image_thumb9.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="670" /></a><br />
<h1>Wiring JMS </h1><h2> </h2><h3>Creating the JMS basis</h3>First thing of all, I will create a JMS server configured with a store (to persist messages, <span style="color: red;"><u>mandatory</u></span> for storing messages when a durable subscriber is offline). Then I define a module containing a topic whose name is <strong>BlogTopic</strong> and JNDI name is “<strong>fr/mbutton/blog/jms/topic</strong>”. and a connectionFactory, whose JNDI name is “<strong>fr/mbutton/blog/jms/connectionFactory</strong>” (pretty original, isn’t it ?).<br />
<br />
<table border="1" cellpadding="2" cellspacing="0" style="text-align: center; width: 499px;"><tbody>
<tr> <td valign="top" width="200"><div style="text-align: center;"><br />
</div><div style="text-align: center;"><strong><em><u>Name</u></em></strong></div></td> <td valign="top" width="297"><div style="text-align: center;"><br />
</div><div style="text-align: center;"><strong><em><u>JNDI Name</u></em></strong></div></td> </tr>
<tr> <td valign="top" width="200"><div style="text-align: center;"><br />
</div><div style="text-align: center;">BlogTopic</div></td> <td valign="top" width="297"><div style="text-align: center;"><br />
</div><div style="text-align: center;">fr/mbutton/blog/jms/topic</div></td> </tr>
<tr> <td valign="top" width="200"><div style="text-align: center;"><br />
</div><div style="text-align: center;">BlogConnectionFactory</div></td> <td valign="top" width="297"><div style="text-align: center;"><br />
</div><div style="text-align: center;">fr/mbutton/blog/jms/connectionFactory</div></td> </tr>
</tbody></table><br />
In the end, I’ve got something like that :<br />
<br />
<a href="http://lh4.ggpht.com/-r-lddtUZOr0/TfpmCstn6zI/AAAAAAAADdI/EnJVNuc46eo/s1600-h/image21%25255B1%25255D%25255B3%25255D.png"><img alt="image" border="0" height="132" src="http://lh4.ggpht.com/-xk_kfK6nXEI/TfpmDEbVnjI/AAAAAAAADdM/Nv7aKcE457o/image21%25255B1%25255D_thumb%25255B2%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="1078" /></a><br />
<br />
Just note that I chose to have a XA enabled connection factory.<br />
<h3>Durable subscriber configuration</h3>Now all the configuration has been made, let’s turn back into JDev and our BPEL processes. <br />
To make the link between the JMS part and my BPEL process, I also need to create a JMS Adapter, which will be in charge of receiving JMS messages and send it over the EDN. That adapter can’t exist by itself : it has to be used in a BPEL process. In my example, mine is called “JMSBridge”.<br />
In the component palette, click the section “BPEL Services” and drag and drop a “JMS Adapter” in the right swimlane.<br />
<br />
<a href="http://lh6.ggpht.com/-Nt7C1BsOEC8/TfpmDhlZntI/AAAAAAAADdQ/PXYrfXvM_gs/s1600-h/image4%25255B1%25255D.png"><img alt="image" border="0" height="323" src="http://lh4.ggpht.com/-kk1YFFAC74A/TfpmEYIIzBI/AAAAAAAADdU/tmhn5_XqlZk/image_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="259" /></a><br />
<br />
A wizard will start. First page is just a welcome screen. The 2nd will prompt for a service name, the 3rd will ask what kind of provider you want, chose “Oracle WebLogic JMS”<br />
<br />
<a href="http://lh3.ggpht.com/-sfbQgnFWoz4/TfpmE6HZ3qI/AAAAAAAADdY/-K43f1IoLEo/s1600-h/image13%25255B1%25255D.png"><img alt="image" border="0" height="217" src="http://lh4.ggpht.com/-qSuDiIAmoRQ/TfpmFhvDzsI/AAAAAAAADdc/3IpT-JQJvOM/image_thumb6.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="462" /></a><br />
<br />
On the 4th screen, chose a connection to a server (chose “localhost-wls”). On the 5th screen, leave it as it is (“Define from schema…”) and click next.<br />
<br />
<a href="http://lh4.ggpht.com/-xywhDChnJ8M/TfpmGKEV5wI/AAAAAAAADdg/V8KDOs1tsoM/s1600-h/image16%25255B1%25255D.png"><img alt="image" border="0" height="118" src="http://lh6.ggpht.com/-1IZT_o91Svw/TfpmG-MYUgI/AAAAAAAADdk/oDwQxSjDunA/image_thumb7%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="244" /></a><br />
<br />
On screen 6, chose the operation “Consume Message” and click “Next”. Screen 7 is where things become a bit trickier. <br />
<br />
<a href="http://lh6.ggpht.com/-p7Z4Hwty-zg/TfpmHjetpgI/AAAAAAAADdo/Kau9ZwVLUZw/s1600-h/image20%25255B1%25255D.png"><img alt="image" border="0" height="341" src="http://lh5.ggpht.com/-QU4E64aE13o/TfpmICCmw2I/AAAAAAAADds/Lc3tQ34YxJ8/image_thumb9%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="599" /></a><br />
<br />
You can see a field “JNDI Name” at the bottom. You would be tempted to enter the JNDI name you previously defined (for the topic). But that would make no sense since it seems that we can define the destination on which we want to consume messages. Then what can it be ? Let’s see …<br />
First, chose your destination (by clicking the “browse” button, you will scan through the previously chosen connection all the available destinations). <br />
<br />
<a href="http://lh3.ggpht.com/-9cZX6ouO39E/TfpmIk02riI/AAAAAAAADdw/ffj8_29UqKE/s1600-h/image64.png"><img alt="image" border="0" height="604" src="http://lh3.ggpht.com/-eCxekLE8JdA/TfpmJcTDioI/AAAAAAAADd0/4EQX8Ex58Kw/image_thumb43.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="275" /></a><br />
<br />
Once you’ve chosen the correct destination, the screen will display another option “Durable Subscriber ID”,<br />
due to the fact we are working with a topic, and a default value will be set for the field “JNDI Name”.<br />
<br />
<a href="http://lh3.ggpht.com/-z96SS4uKC-I/TfpmKGMMOxI/AAAAAAAADd4/s4Zk2IcCXTY/s1600-h/image68.png"><img alt="image" border="0" height="364" src="http://lh6.ggpht.com/-WqYvnUchB3w/TfpmK9rAcCI/AAAAAAAADd8/U-p2euB_lXc/image_thumb45.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="600" /></a><br />
<br />
Where does that JNDI name come from ?<br />
After googling a while, I found that this name is (and has to be) a JNDI name defined on the binding “Outbound Connection Pools” of the JCA Adapter : JMS Adapter. Ok, if you don’t know much about JCA, it sounds a bit complex, but it is not.<br />
Connect to the WLS console (<a href="http://localhost:7001/console">http://localhost:7001/console</a>)<br />
In the deployment section, look for the deployed “Resource Adapter” whose name is “JMS Adapter”, and click on it.<br />
<br />
<a href="http://lh6.ggpht.com/-PPqUtgkRXho/TfpmLb_iIUI/AAAAAAAADeA/Qs-mkf0P5o0/s1600-h/image23.png"><img alt="image" border="0" height="150" src="http://lh5.ggpht.com/-sOVNh0kwMf0/TfpmL3I_v0I/AAAAAAAADeE/HD67fvZaPwQ/image_thumb10.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="243" /></a><br />
<br />
Go in the section “<strong>Configuration</strong> > <strong>Outbound Connection Pools</strong>”<br />
<br />
<a href="http://lh6.ggpht.com/-bZBdbam4hMA/TfpmMWM6LhI/AAAAAAAADeI/BczYP_WtAto/s1600-h/image27.png"><img alt="image" border="0" height="320" src="http://lh6.ggpht.com/-y4xZpuMqk5w/TfpmM4k0QJI/AAAAAAAADeM/h2levMPRk2A/image_thumb12.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="606" /></a><br />
<br />
Click on the ‘<strong>+</strong>’ just before the “<strong>oracle.tip.adapter.jms.IJmsConnectionFactory</strong>” : it displays a set of configuration bound to JNDI names. You should start to guess where I’m about to go.<br />
<br />
<a href="http://lh4.ggpht.com/-_A4lozTtXps/TfpmNuRTquI/AAAAAAAADeQ/Cm7FdIF_FF8/s1600-h/image31.png"><img alt="image" border="0" height="493" src="http://lh5.ggpht.com/-U9_ezU9hJ_g/TfpmOEuKXJI/AAAAAAAADeU/PzqPr2fsjHI/image_thumb14.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="288" /></a><br />
<br />
Here’s the default JNDI name “<strong>eis/wls/Topic</strong>” that appeared in the assistant. Click on it and it will lead you<br />
to the configuration screen.<br />
<br />
<a href="http://lh6.ggpht.com/-nsCIdcOaQPA/TfpmO0qoj7I/AAAAAAAADeY/ltuKPEBt0-s/s1600-h/image60.png"><img alt="image" border="0" height="376" src="http://lh3.ggpht.com/-EFsUrAIxvOY/TfpmPovGd-I/AAAAAAAADec/QVDBCFtAEFs/image_thumb41.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="1054" /></a><br />
<br />
On that screen is some interesting information, such as the “<strong>ConnectionFactoryLocation</strong>” property, which indicates the JNDI name of the JMS connection factory you want to use.<br />
But going with that configuration will NOT work. It’s due to a simple fact : to properly define a durable subscriber, you have to define a durable subscriber ID <strong><u>AND</u></strong> a client ID.<br />
The durable subscriber ID was definable in the JDev assistant, but what about the client ID ? There’s unfortunately no dedicated field : we have to use the field “FactoryProperties”. To be more specific, you have to define your client ID such as : “ClientID=<em>yourUniqueClientID</em>”. As exposed here : <a href="http://forums.oracle.com/forums/thread.jspa?messageID=9520314">http://forums.oracle.com/forums/thread.jspa?messageID=9520314</a>, you have not only to define a unique subscriber ID but also a unique client ID. Then, as you can provide only a client ID for a “outbound connection”, you will have to define more than one.<br />
<u><em>Note</em></u> : no matter what your choice is (using the default conf or create yours), updating the default configuration will create a <a href="http://m-button.blogspot.com/2008/08/how-to-use-deployment-plan.html" target="_blank">deployment plan</a>.<br />
Personnally, I prefer to define my own stuff, that way, when I update a configuration, it’s easier to predict what changes it will imply. So click on the “New” button to define your own configuration. Select the existing group and click “Next”.<br />
<br />
<a href="http://lh5.ggpht.com/-qRWoNdjMrng/TfpmQJxuO_I/AAAAAAAADeg/HX6fpUC3mdE/s1600-h/image72.png"><img alt="image" border="0" height="332" src="http://lh5.ggpht.com/-8SQKl41K448/TfpmQyn_zEI/AAAAAAAADek/6uAHdAytukM/image_thumb47.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="395" /></a><br />
<br />
After that, enter a JNDI name and click “Finish”.<br />
<br />
<a href="http://lh5.ggpht.com/-oMoZIhvpbFg/TfpmRaCRZqI/AAAAAAAADeo/5A6NN0Eb-rY/s1600-h/image76.png"><img alt="image" border="0" height="328" src="http://lh5.ggpht.com/-1wJas0gPO2w/TfpmSMj21CI/AAAAAAAADes/W4rrvBUmP6E/image_thumb49.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="633" /></a><br />
<br />
And finally, chose a destination where to save your deployment plan.<br />
<br />
<a href="http://lh4.ggpht.com/-xsri68ge8fM/TfpmShAnWCI/AAAAAAAADew/xK_hRuORORo/s1600-h/image80.png"><img alt="image" border="0" height="435" src="http://lh3.ggpht.com/-OAuqM-u7RUE/TfpmTSiP5xI/AAAAAAAADe0/6vEjLy9dkLQ/image_thumb51.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="682" /></a><br />
Check that your deployment plan has been taken into account and you’re done.<br />
<a href="http://lh4.ggpht.com/-_wH4ZveUqa4/TfpmT6sSo_I/AAAAAAAADe4/Qs9zt0Tle5M/s1600-h/image84.png"><img alt="image" border="0" height="361" src="http://lh6.ggpht.com/-UEqS9BHwBdo/TfpmUhMhtaI/AAAAAAAADe8/zi9XdwisSgs/image_thumb53.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="700" /></a><br />
<br />
Last step, we now have to configure our freshly created “outbound connection pool”.<br />
<br />
<a href="http://lh4.ggpht.com/-XfdJyuh43bA/TfpmVNUVBwI/AAAAAAAADfA/J__lf1P1j7c/s1600-h/image122.png"><img alt="image" border="0" height="272" src="http://lh3.ggpht.com/-0r5W2c0pDis/TfpmViT-AkI/AAAAAAAADfE/qighNAiVsLE/image_thumb89.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="1064" /></a><br />
<br />
I simply defined the “ConnectionFactoryLocation” property to match the JNDI name I defined for my connection factory, I set a client ID and changed the fact my connectionFactory is transaction enabled. (of course, if your connectionFactory is not, leave that value to “false”).<br />
The configuration, from the WLS side, is now over, so let’s turn back to JDev.<br />
I can now set the last piece of missing information of the 7th screen …<br />
<br />
<a href="http://lh4.ggpht.com/-kBHIFLqakWo/TfpmYfKkiVI/AAAAAAAADfI/a6p8E7gWR_4/s1600-h/image133.png"><img alt="image" border="0" height="127" src="http://lh3.ggpht.com/-PFyRVACn5GE/TfpmY8VxhpI/AAAAAAAADfM/VrGOB99limI/image_thumb94.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="321" /></a><br />
<br />
… and go on with the wizard, straigth to the 8th screen. Here, we’ll define the schema used to qualify the message payload.<br />
<br />
<a href="http://lh3.ggpht.com/-gYQS8DgK74s/TfpmZnzSH4I/AAAAAAAADfQ/ZWOGetOlI7A/s1600-h/image137.png"><img alt="image" border="0" height="712" src="http://lh4.ggpht.com/-dGFhjlCWy44/TfpmaVR08BI/AAAAAAAADfU/UFkb2zXhiTk/image_thumb96.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="618" /></a><br />
<br />
And that’s it ! You’re done !<br />
<br />
<a href="http://lh5.ggpht.com/-IT4KhLt5HLM/TfpmbNbatTI/AAAAAAAADfY/AoxE98d5HBc/s1600-h/image141.png"><img alt="image" border="0" height="162" src="http://lh4.ggpht.com/-0xDD0ygzz-E/Tfpmb5UiReI/AAAAAAAADfc/1BVt9SpYQbM/image_thumb98.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="604" /></a><br />
Now on the BPEL Process “JMSBridge”, just remove the receive activity and replace it with another Receive activity, but this time which is linked to the JMS Adapter. Don’t forget to click on the checkbox “Create instance”. Remove also the original partnerLink as well as the variable used in the original Receive activity.<br />
Then, add a Invoke activity and chose “Event” instead of “PartnerLink” and add a variable. Add an assign activity between the Receive and the Invoke and define a copy operation to replicate the message from the incoming payload to the payload of the outgoing message.<br />
<br />
If, when you compile, you have an error message “Error(31,72): Service "jmsbridge_client_ep" does not<br />
exist as wire source”, edit the composite and remove what’s left of the previous partnerLink.<br />
<br />
<a href="http://lh6.ggpht.com/-uyGBW_JB9a4/TfpmcTAKRoI/AAAAAAAADfg/sLh5WEncbjo/s1600-h/image145.png"><img alt="image" border="0" height="188" src="http://lh6.ggpht.com/-6UbT4c1x-l4/TfpmdAD3B2I/AAAAAAAADfk/NLGVWlaCDJA/image_thumb100.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="646" /></a><br />
<br />
In the end, your process will look like that :<br />
<br />
<a href="http://lh3.ggpht.com/-j5mXI-wxD44/Tfpmdrx6BLI/AAAAAAAADfo/5_FhvdYfaiA/s1600-h/image153.png"><img alt="image" border="0" height="434" src="http://lh5.ggpht.com/-vnzfhUNSQE4/TfpmeIRSEGI/AAAAAAAADfs/QcdpdIpaKSQ/image_thumb104.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="765" /></a><br />
<h1>Testing the JMS binding</h1>First of all, restart your server. I’ve experienced some problem when I didn’t. Even if you redeploy your JMS Adapter, it won’t work and you will end in having NPE while trying to bind to the topic.<br />
During the server restart, look for that message :<br />
<blockquote>[2011-06-13T23:46:59.046+02:00] [AdminServer] [NOTIFICATION] [] [oracle.soa.adapter] [tid: weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@bf46a] [userId: <anonymous>] [ecid: 0000J2AyCeQFw000jzwkno1DxcGx00000I,0] [APP: soa-infra] [dcid: 11d1def534ea1be0:-66ae329:1308af61f0e:-7fd1-0000000000000011] JMSAdapter BPELDurableSubscriber JMSMessageConsumer_init: Successfully created MessageConsumer for destination fr/mbutton/blog/jms/topic (payload = 1, subscriber = JMSBridge) <br />
<br />
</blockquote>You can double check by verifying that your JMS Adapter is correctly bound to the topic. To do so, just go in the “<strong>Services > Messaging > JMS Modules</strong>” and click on your module.<br />
<br />
<br />
<a href="http://lh6.ggpht.com/-d0-Ho2vL-x0/Tfpme6n8FdI/AAAAAAAADfw/fwCkvMteBEk/s1600-h/image157.png"><img alt="image" border="0" height="205" src="http://lh4.ggpht.com/-tjgkCgGMUjQ/TfpmfSIAE3I/AAAAAAAADf0/wmpfF02wBhQ/image_thumb106.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="495" /></a><br />
<br />
Then in the monitoring section, click on the subtab “Durable Subscriber” : a line should be displayed.<br />
<br />
<a href="http://lh6.ggpht.com/-9e1fKBphyJE/TfpmfxK0LnI/AAAAAAAADf4/cr8QRdnUt1w/s1600-h/image161.png"><img alt="image" border="0" height="352" src="http://lh5.ggpht.com/-clBp2qpgv3g/TfpmgsT8GiI/AAAAAAAADf8/gSPvb_5dLnw/image_thumb108.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="682" /></a><br />
<br />
Now, we have to make sure that a BPEL process instance is created when a JMS message is sent / received.<br />
<br />
To send JMS messages, you can use a java class or use HermesJMS or do whatever you want to, it’s up to you ! :) But I chosed to write a custom class.<br />
Here’s the payload I sent : <br />
<br />
<a href="http://lh5.ggpht.com/-g8POK65jdFE/TfpmhFMWgMI/AAAAAAAADgA/UFaBc4dyTSg/s1600-h/image%25255B4%25255D.png"><img alt="image" border="0" height="126" src="http://lh3.ggpht.com/-sn6XlGKKiAo/Tfpmh-MB-7I/AAAAAAAADgE/ePBv2ng7_EY/image_thumb%25255B1%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="534" /></a><br />
<br />
And in the EnterpriseManager, here’s what I’ve got in return :<br />
<br />
<a href="http://lh6.ggpht.com/-zdn-1sVBi20/TfpmiarKVLI/AAAAAAAADgI/387yb9xvFbM/s1600-h/image%25255B11%25255D.png"><img alt="image" border="0" height="207" src="http://lh6.ggpht.com/--YJjy5_fSMY/TfpmixbkWII/AAAAAAAADgM/Yt9w6PTg35I/image_thumb%25255B4%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="867" /></a><br />
<br />
Here are the details of this BPEL process instance :<br />
<br />
<a href="http://lh3.ggpht.com/-kL4kZnNMgIE/Tfpmjgbe51I/AAAAAAAADgQ/OHaPapDH4DY/s1600-h/image%25255B12%25255D.png"><img alt="image" border="0" height="562" src="http://lh6.ggpht.com/-y3a8j5DhChU/TfpmkdVeooI/AAAAAAAADgU/ZFrgj6raTrw/image_thumb%25255B5%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="625" /></a><br />
<br />
So I guess I can say my bridge is working. Now the fun begins.<br />
<br />
<h1>Testing the durable subscription</h1>To test a durable subscription, there’s not many tests : the only one that makes sense is to de-activate the durable subscriber, send a JMS message, wait for a few seconds to see that the message is kept on the topic and that no BPEL process instance is created. Then, when the durable subscriber is reactivated, the JMS message must be consumed.<br />
But how to disconnect a BPEL process from a topic ? Well that’s an interesting question. During my tests, I found out that a shutdown was disconnecting completely the composite from the topic. Then when a message arrived, the topic did not persist it since there was no durable subscriber. Not what I needed.<br />
I then tried to “retire” the composite and there was my solution. The link was still there but was considered as “Inactive”, which is precisely what I was looking for.<br />
In my use case, I had to develop a class which allows me to retire the process programmatically. In that article, I will simply use the EM to do so.<br />
<h3>Retire the application</h3>Using the EM, click on the “Retire” button.<br />
<br />
<a href="http://lh5.ggpht.com/-LCGsXJIiT6k/Tfpmk0Wj6CI/AAAAAAAADgY/Naf1ukGohUE/s1600-h/image%25255B25%25255D.png"><img alt="image" border="0" height="237" src="http://lh3.ggpht.com/-k9jgUvPGwjw/TfpmloBSmoI/AAAAAAAADgc/C-cuFCFKV_s/image_thumb%25255B10%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="445" /></a><br />
<br />
Then if you click on your partition (the one on which you deployed your SAR), you should see the status “Retired”.<br />
<br />
<a href="http://lh4.ggpht.com/-0RYxESXZXe0/TfpmmUIg1DI/AAAAAAAADgg/Wq-9SIsmg1Y/s1600-h/image%25255B26%25255D.png"><img alt="image" border="0" height="95" src="http://lh4.ggpht.com/-9u8zqFyRNjw/Tfpmm2bWaiI/AAAAAAAADgk/pb-U-IA284M/image_thumb%25255B11%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="719" /></a><br />
<br />
But the most interesting part is the topic monitoring, especially, the durable subscriber tab.<br />
Before, it was like that :<br />
<br />
<a href="http://lh3.ggpht.com/-ciPzS5mtG3w/TfpmnSiXSII/AAAAAAAADgo/AIZFOPgs6Zs/s1600-h/image%25255B27%25255D.png"><img alt="image" border="0" height="160" src="http://lh4.ggpht.com/-No-Inp2AKHI/Tfpmn5sC1vI/AAAAAAAADgs/YuYlSgKf1yY/image_thumb%25255B12%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="681" /></a><br />
<br />
And now it’s more (at least it should be) like :<br />
<br />
<a href="http://lh5.ggpht.com/-9twTvp2MF4k/TfpmoV4BlWI/AAAAAAAADgw/MZ4Zgo64Tks/s1600-h/image%25255B28%25255D.png"><img alt="image" border="0" height="163" src="http://lh3.ggpht.com/-T4owN2KF5Yg/Tfpmo7I9YpI/AAAAAAAADg0/DBssDWrVFP8/image_thumb%25255B13%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="683" /></a><br />
<br />
Okay, so now, I have to send a message and normally, nothing should happen (fingers crossed)<br />
The message has been sent !<br />
<br />
<a href="http://lh4.ggpht.com/-S7NKGzz_bKg/TfpmpYoUfiI/AAAAAAAADg4/Ygl1W7sf-JY/s1600-h/image%25255B53%25255D.png"><img alt="image" border="0" height="84" src="http://lh5.ggpht.com/-TdH_ZmzQR_U/TfpmqfQ5dGI/AAAAAAAADg8/5Wbihk5sdL4/image_thumb%25255B26%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="644" /></a><br />
<br />
I’ll wait for about one minute. Meanwhile, let’s have a look at the topic monitoring :<br />
<br />
<a href="http://lh3.ggpht.com/-bIEcRBtVcAA/Tfpmq7gJdLI/AAAAAAAADhA/_BBloMGmuJw/s1600-h/image%25255B54%25255D.png"><img alt="image" border="0" height="161" src="http://lh5.ggpht.com/-3saEyMgUJDo/TfpmrePt06I/AAAAAAAADhE/aVfDAEnF-4Y/image_thumb%25255B27%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="905" /></a><br />
<br />
We can see that a message is currently on the topic. Means no one has eaten it yet (relief).<br />
On the EM dashboard, the two previous instances are still there but no new instance has been created :<br />
<br />
<a href="http://lh6.ggpht.com/-KSYGjCLzvvQ/Tfpmr7kM8_I/AAAAAAAADhI/8ZDNvt1DZ_s/s1600-h/image%25255B56%25255D.png"><img alt="image" border="0" height="93" src="http://lh3.ggpht.com/-WYArtsy_WVI/TfpmsnbisxI/AAAAAAAADhM/hGdq99pQ92s/image_thumb%25255B29%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="1076" /></a><br />
<br />
Now, it’s time to reactivate the composite (I won’t put a screenshot of the activation, I guess you know how it works :p)<br />
<br />
The topic monitoring shows the durablesubscriber has been reactivated (Active = true) and there’s no message left.<br />
<br />
<a href="http://lh5.ggpht.com/-JNpQbarvNro/TfpmtBVa_tI/AAAAAAAADhQ/akIzwoZkYLU/s1600-h/image%25255B50%25255D.png"><img alt="image" border="0" height="156" src="http://lh5.ggpht.com/-Xd5WKInX1vI/TfpmtnSk1zI/AAAAAAAADhU/AiwOemMmzFw/image_thumb%25255B23%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="910" /></a><br />
<br />
Finally, let’s have a look at the BPEL process instances : it works ! <br />
<br />
<a href="http://lh3.ggpht.com/-jmiVAoZIYEo/TfpmuTE7WFI/AAAAAAAADhY/D7KvXt1QURY/s1600-h/image%25255B58%25255D.png"><img alt="image" border="0" height="103" src="http://lh5.ggpht.com/-TCNed8tdmS8/Tfpmu8dvl-I/AAAAAAAADhc/OuILgRczFsc/image_thumb%25255B31%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="1065" /></a><br />
<br />
And in detail :<br />
<br />
<a href="http://lh3.ggpht.com/--quVrwKKro4/TfpmviN6L4I/AAAAAAAADhg/VOje5Ep2VIs/s1600-h/image%25255B47%25255D.png"><img alt="image" border="0" height="605" src="http://lh4.ggpht.com/-iZlChGpvpno/TfpnS4eirAI/AAAAAAAADh0/pmqzh24brBE/image_thumb%25255B20%25255D.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="image" width="656" /></a><br />
<br />
Hope it will help some fellows out there ! Thanks for reading ! :)<br />
<br />
<div class="wlWriterEditableSmartContent" id="scid:0767317B-992E-4b12-91E0-4F059A8CECA8:dd0f343c-a973-46a3-afd8-e822bc1103a9" style="display: inline; float: none; margin: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">Mots clés Technorati : <a href="http://technorati.com/tags/BPEL" rel="tag">BPEL</a>,<a href="http://technorati.com/tags/Oracle" rel="tag">Oracle</a>,<a href="http://technorati.com/tags/JMS" rel="tag">JMS</a>,<a href="http://technorati.com/tags/Topic" rel="tag">Topic</a>,<a href="http://technorati.com/tags/Durable+Subscriber" rel="tag">Durable Subscriber</a>,<a href="http://technorati.com/tags/ClientID" rel="tag">ClientID</a>,<a href="http://technorati.com/tags/BPEL+PM" rel="tag">BPEL PM</a></div>Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com4tag:blogger.com,1999:blog-2592381182145606344.post-36224031543459720002011-06-04T23:25:00.001+02:002011-06-04T23:25:19.638+02:00The easiest way to refactor a BPEL process with BPEL Process Manager<p> </p> <p>Lately, I’ve been pretty extensively using BPEL Process Manager to migrate my client WLI projects towards the new official Oracle preferred product.</p> <p>I could have used the official migration tool, which has been developed (I should say extended, since this feature is not exactly new), but as I explained it to several clients,</p> <p>I prefer to code my application from scratch, since it would allow me to really use the product.</p> <p>I heard that some projects which went the “migration tool” road, did an amazing job at automating the migration by tuning the tool and adding some post-processing scripts.</p> <p>But such a method is annoying about one point : you do not improve the knowledge of the new tool, which is important when you have to do some maintenance on projects relying on it.</p> <p>Then I started to work with BPEL PM and I quickly find myself with lots of files (BPEL file, Mediators, WSDL & _property WSDL for each and every new BPEL Process) and I felt like cleaning up my playground.</p> <p>I decided to create some directories and to put my BPEL processes (and their associated files) in them. Coming from Eclipse like lots of developers, I was expecting the refactor option to be in a contextual menu, but it was not that easy to find … indeed, it’s only accessible in a top menu :</p> <p> </p> <p><a href="http://lh3.ggpht.com/-6tYsEF7DjW0/Teqiuw7h0RI/AAAAAAAADcQ/iX4nEkerhwI/s1600-h/image3.png"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: block; float: none; border-top-width: 0px; border-bottom-width: 0px; margin-left: auto; border-left-width: 0px; margin-right: auto; padding-top: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/-vgizw4OwrVM/Teqivtif7ZI/AAAAAAAADcU/vqDgt71zS8o/image_thumb1.png?imgmax=800" width="416" height="445" /></a></p> <p> </p> <p>After a quick while, my process was moved. I tried to compile and I got an error message, saying the BPEL process was nowhere to be found, while validating the composite ….</p> <p>Huh ? I then looked into the composite and … here was the object of my compiler complaint : the path of the BPEL process has not been updated !</p> <p>I decided to file a case to the Oracle Support and I won’t make any comment about the answer I was given. If you have an Oracle Support account, see for yourself, it’s the following case number : <a href="http://support.oracle.com" target="_blank">SR 3-3082902481: JDev can't handle a proper BPEL refactor</a>.</p> <p>What I can tell you is that the answer was far from being satisfying.</p> <p>Then, I had to find a way to order my stuff, and to be more productive than editing all the files one by one, to make sure the references were correctly changed.</p> <p>I searched a bit and I started to find some Unix commands that sounded promising.</p> <p>And finally, I came to those lines who helped me A LOT ! </p> <blockquote> <p><em><strong>Those lines perform a rename ON the file itself. I was forced to use different cases because that’s the way BPEL PM does it.</strong></em></p> </blockquote> <blockquote> <p>find . -type f | egrep '.bpel$|.wsdl$|.mplan$|.componentType$|.xml$' | xargs rename Process1 Process2 *.* <br />find . -type f | egrep '.bpel$|.wsdl$|.mplan$|.componentType$|.xml$' | xargs rename process1 process2 *.* <br /></p> <p><em><strong>The following lines perform a rename IN the found file(s).</strong></em></p> <p>find . -type f | egrep '.bpel$|.wsdl$|.mplan$|.componentType$|.xml$' | xargs sed -i 's/Process1/Process2/g' <br />find . -type f | egrep '.bpel$|.wsdl$|.mplan$|.componentType$|.xml$' | xargs sed -i 's/process1/process2/g'</p> </blockquote> <p> </p> <p>Using those two commands, it will help you save up some time. At least, it did for me ! :)</p> <div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:0767317B-992E-4b12-91E0-4F059A8CECA8:1c80384b-ff9f-48c7-93bb-bc9f0a6995ca" class="wlWriterEditableSmartContent">Mots clés Technorati : <a href="http://technorati.com/tags/BPEL" rel="tag">BPEL</a>,<a href="http://technorati.com/tags/Process+Manager" rel="tag">Process Manager</a>,<a href="http://technorati.com/tags/refactor" rel="tag">refactor</a>,<a href="http://technorati.com/tags/rename" rel="tag">rename</a>,<a href="http://technorati.com/tags/JDeveloper" rel="tag">JDeveloper</a></div> Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com1tag:blogger.com,1999:blog-2592381182145606344.post-76721500081781850812010-12-19T23:37:00.002+01:002011-06-20T22:21:44.592+02:00WLI Message Broker versus BPEL PM EDNWLI is no more.<br />
Well, not exactly, not yet at least, but Oracle made it pretty clear : WLI will have a 9 year support but after that, clients will have to move toward BPEL PM. Knowing that WLI 8.1 is the most widely used version, that changes the whole situation since it will ceased to be supported … next year … (September 2011)<br />
Then, from there, you’ll have two (Oracle) choices : either to migrate your application to WLI 10, or migrate your application to BPEL PM. The choice seems easy to make : “let’s go for WLI 10 !”. But it’s not that simple, since whatever the choice you make, you’ll have to modify or recode from scratch your app and if you go for WLI 10, you still will have to do the same later on, to move towards BPEL PM. To me, migrating to WLI 10 is just a way to slow down what’s unevitable.<br />
With that in mind, what solution would you go for ? <br />
I personally think that going right now toward BPEL PM is a smarter move and that’s why I’m about to demonstrate how to migrate a simple WLI process toward BPEL PM.<br />
<blockquote>Note : I’ll start with a WLI process and then, we’ll see how to “translate” it into a BPEL fashion. I won’t spend much time talking about WLI, because I’m expecting readers to know a bit about it. This blogpost will mainly focus on the BPEL part.</blockquote><h2>1) Designing a simple WLI process</h2>No fancy stuff : our process will just wait for two messages to arrive, before going further.<br />
It’s a really common stuff WLI is excellent at ! Thanks to its message broker, it adds a very handy layer to handle your messages.<br />
Our messages will be defined according to the following XSD :<br />
<pre style="background-color: #fbfbfb; border-bottom: #cecece 1px solid; border-left: #cecece 1px solid; border-right: #cecece 1px solid; border-top: #cecece 1px solid; min-height: 40px; overflow: auto; padding-bottom: 5px; padding-left: 5px; padding-right: 5px; padding-top: 5px; width: 650px;"><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><?</span>xml version="1.0"<span style="color: blue;">?></span>
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">schema</span>
</pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: red;">xmlns</span>:<span style="color: red;">xs</span>=<span style="color: blue;">"http://www.w3.org/2001/XMLSchema"</span>
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: red;">xmlns</span>:<span style="color: red;">tns</span>=<span style="color: blue;">"http://temp.openuri.org/WLIMigrationToBPEL/sample.xsd"</span>
</pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: red;">targetNamespace</span>=<span style="color: blue;">"http://fr.mbutton.blog/WLIMigrationToBPEL"</span>
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: red;">elementFormDefault</span>=<span style="color: blue;">"qualified"</span>
</pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: red;">attributeFormDefault</span>=<span style="color: blue;">"unqualified"</span><span style="color: blue;">></span>
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">element</span> <span style="color: red;">name</span>=<span style="color: blue;">"message"</span><span style="color: blue;">></span>
</pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">complexType</span><span style="color: blue;">></span>
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">sequence</span><span style="color: blue;">></span>
</pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">element</span> <span style="color: red;">name</span>=<span style="color: blue;">"id"</span> <span style="color: red;">type</span>=<span style="color: blue;">"xs:string"</span><span style="color: blue;">/></span>
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"><</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">element</span> <span style="color: red;">name</span>=<span style="color: blue;">"label"</span> <span style="color: red;">type</span>=<span style="color: blue;">"xs:string"</span><span style="color: blue;">/></span>
</pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"></</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">sequence</span><span style="color: blue;">></span>
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"></</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">complexType</span><span style="color: blue;">></span>
</pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"></</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">element</span><span style="color: blue;">></span>
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"><span style="color: blue;"></</span><span style="color: mediumvioletred;">xs</span>:<span style="color: maroon;">schema</span><span style="color: blue;">></span>
</pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"></pre></pre><br />
First comes our receiver process :<br />
<br />
<br />
<br />
<br />
<br />
<a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZyXjcmgrI/AAAAAAAADJQ/8uc4A05ouTM/s1600-h/messageReceiverProcess6.gif"><img alt="messageReceiverProcess" border="0" height="823" src="http://lh6.ggpht.com/_bayCgqm3V20/TRZyYEhLPdI/AAAAAAAADJU/wWWkginOvsM/messageReceiverProcess_thumb4.gif?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="messageReceiverProcess" width="383" /></a> <br />
<br />
<br />
And then the “utility” process, we’ll use to send messages.<br />
<br />
Note that we could have done a more sophisticated system based on JMS messages & an EventGenerator, but that’s beyond the scope of this article.<br />
<br />
<br />
<a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZyYiRDylI/AAAAAAAADJY/0ARzEk0M18c/s1600-h/messageSenderProcess5.gif"><img alt="messageSenderProcess" border="0" height="389" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZyZZBl3bI/AAAAAAAADJc/PEv4Id5sRiU/messageSenderProcess_thumb3.gif?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="messageSenderProcess" width="255" /></a> <br />
<br />
<br />
<br />
<br />
<br />
Once deployed, as expected, no running process is displayed in the WLI Console :<br />
<br />
<br />
<br />
<br />
<br />
<a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZyaEat2vI/AAAAAAAADJg/xdkVWWAGe5w/s1600-h/no_running_process3.gif"><img alt="no_running_process" border="0" height="172" src="http://lh4.ggpht.com/_bayCgqm3V20/TRZya1GyWNI/AAAAAAAADJk/CK2GO4977AE/no_running_process_thumb1.gif?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="no_running_process" width="758" /></a> <br />
<br />
To do so, we just have to start the processes for testing. First one : the receiver.<br />
<br />
<br />
<a href="http://lh6.ggpht.com/_bayCgqm3V20/TRZybRmqtTI/AAAAAAAADJo/jmvHxhGWnwI/s1600-h/startMessageReceiver3.gif"><img alt="startMessageReceiver" border="0" height="262" src="http://lh6.ggpht.com/_bayCgqm3V20/TRZybz3YtpI/AAAAAAAADJs/Pzgm5dKXsdM/startMessageReceiver_thumb1.gif?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="startMessageReceiver" width="599" /></a> <br />
<br />
<br />
Now if we go back in the WLI console, we can now see that this process is running and waiting for some messages to arrive.<br />
<br />
<br />
<br />
<br />
<a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZycVFIO7I/AAAAAAAADJw/Zf2l6IrhRkw/s1600-h/processRunningAndWaiting4Msgs3.gif"><img alt="processRunningAndWaiting4Msgs" border="0" height="403" src="http://lh3.ggpht.com/_bayCgqm3V20/TRZydKDYvwI/AAAAAAAADJ0/8izGQTLm9AE/processRunningAndWaiting4Msgs_thumb1.gif?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="processRunningAndWaiting4Msgs" width="494" /></a> <br />
<br />
<br />
<br />
<br />
<br />
To complete it, we have to send two messages whose IDs are “<strong>ID_A</strong>” & “<strong>ID_B</strong>”.<br />
<br />
<br />
Just call the sender process to do so :<br />
<br />
<br />
<br />
<br />
<br />
<a href="http://lh6.ggpht.com/_bayCgqm3V20/TRZydqDNHII/AAAAAAAADJ4/EjcI1dS6XWk/s1600-h/sendMessageA3.gif"><img alt="sendMessageA" border="0" height="390" src="http://lh6.ggpht.com/_bayCgqm3V20/TRZyfSta1XI/AAAAAAAADJ8/HUn4OB3pHT4/sendMessageA_thumb1.gif?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="sendMessageA" width="929" /></a> <br />
<br />
<br />
In the console, we can now see that’s the process has received the event and is no longer waiting for it :<br />
<br />
<br />
<br />
<a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZyf_wuG-I/AAAAAAAADKA/nkcGfkGIlmA/s1600-h/msgAReceived3.gif"><img alt="msgAReceived" border="0" height="383" src="http://lh3.ggpht.com/_bayCgqm3V20/TRZygc6TuRI/AAAAAAAADKE/tMAXfO2Sc8E/msgAReceived_thumb1.gif?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="msgAReceived" width="545" /></a> <br />
<br />
<br />
<br />
<br />
<br />
If we repeat that step with message B, it will complete the process :<br />
<br />
<br />
<br />
<br />
<br />
<a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZyg9qecYI/AAAAAAAADKI/9qQQZWeTZCk/s1600-h/messageBReceivedProcessComplete3.gif"><img alt="messageBReceivedProcessComplete" border="0" height="295" src="http://lh3.ggpht.com/_bayCgqm3V20/TRZyh94JWUI/AAAAAAAADKM/-FTT2PCcbNY/messageBReceivedProcessComplete_thum.gif?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="messageBReceivedProcessComplete" width="447" /></a> <br />
<br />
<br />
Okay, our WLI process works fine, let’s do some more serious business.<br />
<br />
<h2>2) Converting to BPEL</h2><br />
<span class="Apple-style-span" style="font-size: 19px; font-weight: bold;">2.1) BPEL Export</span><br />
<br />
If you’re familiar with WLI, you should have noticed that an interesting option is proposed when you install the product : BPEL Export.<br />
<br />
<br />
Let’s see what’s that option doing :<br />
<br />
<br />
<a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZyiSCk-tI/AAAAAAAADKQ/z2MwMljCe1E/s1600-h/exportBPEL3.gif"><img alt="exportBPEL" border="0" height="361" src="http://lh4.ggpht.com/_bayCgqm3V20/TRZyi9CyZYI/AAAAAAAADKU/pB4bjVLqyZY/exportBPEL_thumb1.gif?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="exportBPEL" width="390" /></a> <br />
<br />
<br />
After the export has been done, you should see in the new BpelExport pane :<br />
<br />
<a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZyjX0DXKI/AAAAAAAADKY/R4SVlL0YziI/s1600-h/bpelConversionDone3.gif"><img alt="bpelConversionDone" border="0" height="128" src="http://lh3.ggpht.com/_bayCgqm3V20/TRZyjy6JbrI/AAAAAAAADKc/J675c_YHWyw/bpelConversionDone_thumb1.gif?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="bpelConversionDone" width="726" /></a><br />
<br />
<br />
<br />
And some new files in the Application pane : <br />
<br />
<br />
<a href="http://lh5.ggpht.com/_bayCgqm3V20/TRZykb8d2jI/AAAAAAAADKg/JEClevh7_X0/s1600-h/newBpelFiles3.gif"><img alt="newBpelFiles" border="0" height="169" src="http://lh6.ggpht.com/_bayCgqm3V20/TRZyk1lQRbI/AAAAAAAADKk/YtJPfFs3yD0/newBpelFiles_thumb1.gif?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="newBpelFiles" width="261" /></a> <br />
<br />
<br />
Let’s import the bpel files into JDev :<br />
<br />
<br />
<a href="http://lh6.ggpht.com/_bayCgqm3V20/TRZylTNW5iI/AAAAAAAADKo/raA9vchrydw/s1600-h/image6.png"><img alt="image" border="0" height="773" src="http://lh6.ggpht.com/_bayCgqm3V20/TRZymH2LcfI/AAAAAAAADKs/rb8_sGwH550/image_thumb2.png?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="image" width="936" /></a><br />
<br />
<br />
I have to say I’m impressed : I wasn’t expecting that kind of result. What I didn’t know, is that a BPEL process is not orchestrated like a WLI process. So, even if the graphical aspect is pretty much the same, that does not mean anything.<br />
<br />
<br />
It’s just an empty shell and with three errors stating that some partnerLinks are missing …<br />
<br />
<br />
<a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZymljreSI/AAAAAAAADKw/C1JmXUADFL0/s1600-h/image7.png"><img alt="image" border="0" height="189" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZynr00TII/AAAAAAAADK0/QS8dzmd6d2M/image_thumb3.png?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="image" width="404" /></a><br />
<br />
<br />
If you’re not familiar with BPEL, that must be confusing. (and it was for me, when I first worked with BPEL PM !)<br />
<br />
<h3>2.2) From scratch</h3><br />
<br />
The BPEL export is working but only converts the graphical view, which is not exactly what we wanted, especially since the product philosophies are different.<br />
<br />
<br />
Then, we will start again, from scratch. And the only thing that will remain the same is the XSD we used.<br />
<blockquote><br />
<strong><u>Note</u></strong> : To be able to develop quickly, I used (and am still using) the VM set up by Oracle which offers a full SOA Suite 11gR1PS2 environment (development & runtime) on RHEL 5. To get the (excellent) VirtualBox image, go to <a href="http://www.oracle.com/technetwork/middleware/soasuite/learnmore/vmsoa-172279.html" target="_blank">OTN</a>.<br />
<br />
</blockquote><span class="Apple-style-span" style="font-weight: bold;">2.2.1) Creating the composite</span><br />
<br />
First thing of all, we need to create a new SOA Project. You have to chose the composite template you want to start with.<br />
<br />
As we are interested in BPEL, we may choose “Composite With BPEL Process”.<br />
<br />
<br />
<a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZypL3cB_I/AAAAAAAADK4/d-yavqD8hTQ/s1600-h/ScreenshotCreateSOAProjectStep2of23.png"><img alt="Screenshot-Create SOA Project - Step 2 of 2" border="0" height="484" src="http://lh4.ggpht.com/_bayCgqm3V20/TRZyp_GGqKI/AAAAAAAADK8/HEJb99tMIaE/ScreenshotCreateSOAProjectStep2of2_t.png?imgmax=800" style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto;" title="Screenshot-Create SOA Project - Step 2 of 2" width="644" /></a><br />
<br />
<br />
<div align="left"><br />
</div><br />
<br />
<div align="left">As we chose to create a BPEL process, a popup shows up, asking some details about the way we want to create it.</div><br />
<br />
<div align="left">You can change the namespace, the name and the template (“Asynchronous”, “Synchronous” or “One Way”) or define it later. In our case, as there is no need for a return, just pick “One way”.</div><br />
<div align="center"><br />
</div><div align="center"><a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZyqerMQ5I/AAAAAAAADLA/24MW23RvcRw/s1600-h/02-Screenshot-Create-BPEL-Process3.png"><img alt="02-Screenshot-Create BPEL Process" border="0" height="441" src="http://lh6.ggpht.com/_bayCgqm3V20/TRZyq4dETTI/AAAAAAAADLE/tJIBEQ0SBj4/02-Screenshot-Create-BPEL-Process_th.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="02-Screenshot-Create BPEL Process" width="644" /></a></div><br />
<br />
<div align="left">Then, import the XSD in the XSD folder of your SOA project.</div><br />
<h4 align="left">2.2.2) Some explanations before the example</h4><br />
<div align="left">That’s where the fun begins. Coming from WLI, I designed something with the “WLI message broker” in mind : I thought my BPEL process would just listen to the desired messages (thanks to filters) and get them. But that’s not what happened.</div><br />
<br />
<div align="left">In fact, the “WLI message broker” & the EDN (Event Delivery Network) are pretty different. Just think as a JMS topic : when a producer sends a message, all the consumers receive it. That’s the principle of the “WLI message broker” (this has been voluntarily simplified). </div><br />
<br />
<div align="left">With EDN, even if a message, matching the filter you’ve defined in your BPEL process, arrives, you won’t receive it. Why ? Because you have to tell which instance is aimed. Lost ? We will see that in a few lines.</div><br />
<br />
<div align="left">In our simple example, in which two messages are expected before being able to go on, you need to first define a correlationSet for the messages to know where to go. It might sound pretty weird, especially when you’re used to work with classical MOMs but don’t worry, it’s not that bad :) </div><br />
<br />
<div align="left">Another thing you need to know is that for now, in the SOA Suite 11gR1PS2, there’s no way you can define two filters on the same message in a single BPEL process. I mean, you can do some tricks (such as defining several times the same event, with a different name) but I think this has been to be improved. Normally, it should be available next January, in the SOA Suite 11gR1PS3. For now, it’s easier to use a mediator to do the routing & filtering job.</div><br />
<br />
<div align="left">Again, we will see that in our example.</div><br />
<br />
<h4 align="left">2.2.3 ) Creating the Mediator</h4><br />
<br />
<div align="left">A mediator is a component whose role is to filter, transform and route messages within the SOA Suite (as opposed to the OSB, whose role is exactly the same, but outside the SOA Suite). In our example, the mediator is the component that will receive & filter EDN messages, and then route them to the good BPEL process route.</div><br />
<blockquote><br />
<div align="left">Note : I am sorry, but to limit the number of screenshots, I displayed all the cascading popups on a same screen. Just refer to the numbers to see in what order they’re displayed.</div><br />
</blockquote><br />
<ol><li><div align="left">Create a Mediator component by drag ‘n’ dropping a Mediator component (on the right) to the composite (in the middle), this will display window “1”</div><br />
</li>
<li><div align="left">Click on the “<span style="color: lime;"><strong>+</strong></span>”, this will bring the window “2”, in which you are asked to create an EDF (Event Definition File)</div><br />
</li>
<li><div align="left">Click once again on the “<span style="color: lime;"><strong>+</strong></span>”, to create your message. Windows “3” will show up. </div><br />
</li>
<li><div align="left">Click on the magnifying glass to chose your event (window “4”)</div><br />
</li>
<li><div align="left">Once again, click on the magnifying glass and select (in window '”5”) the type associated to the XML message</div><br />
</li>
</ol><a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZyr1cxKOI/AAAAAAAADLI/AcNxZvuXl-o/s1600-h/05-Mediator5.png"><img alt="05-Mediator" border="0" height="580" src="http://lh6.ggpht.com/_bayCgqm3V20/TRZytNgDs5I/AAAAAAAADLM/bvC3DRQOG2A/05-Mediator_thumb2.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="05-Mediator" width="1028" /></a><br />
<br />
<div align="center" style="text-align: -webkit-auto;"><br />
</div><div align="left">As the Mediator is not going to publish messages on the EDN, set the “Run as publisher” option to “No”.</div><br />
<br />
<div align="center"><a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZyuNalOgI/AAAAAAAADLQ/wU3mwWcE47U/s1600-h/06-runAsPublisherNo4.png"><img alt="06-runAsPublisherNo" border="0" height="580" src="http://lh3.ggpht.com/_bayCgqm3V20/TRZyvE88wKI/AAAAAAAADLU/8XOVs5F2PVM/06-runAsPublisherNo_thumb2.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="06-runAsPublisherNo" width="1028" /></a></div><br />
<br />
<div align="left">Here’s your Mediator (in purple) in the composite.</div><br />
<br />
<div align="center"><a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZywJ_MZXI/AAAAAAAADLY/_mmSGGKdh4g/s1600-h/07-compositeUpdated4.png"><img alt="07-compositeUpdated" border="0" height="580" src="http://lh4.ggpht.com/_bayCgqm3V20/TRZyxHFxoAI/AAAAAAAADLc/4sE9KTtTcFY/07-compositeUpdated_thumb2.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="07-compositeUpdated" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">For the mediator to be able to route the messages, we must have three “listening branches” in our BPEL process. One to start the process and the two others to receive the expected messages. To do so, we have to update the WSDL of the BPEL process :</div><div align="center" style="text-align: -webkit-auto;"><br />
</div><div align="center"><a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZyyEsJbuI/AAAAAAAADLg/1hAnMLH27KA/s1600-h/08-wsdlEdition4.png"><img alt="08-wsdlEdition" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZyzZE1PjI/AAAAAAAADLo/qwJF1AZ3kfM/08-wsdlEdition_thumb2.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="08-wsdlEdition" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="center">First, declare the message namespace (the one from your XSD) in the <em>wsdl:definition</em> tag. Then, import the namespace (<em>wsdl:types</em> tag).</div><br />
<div align="left">After that, just update the definition of the message you will receive, in the <em>wsdl:message > wsdl:part @element. </em>(according to the prefix you have defined for your imported namespace, it should be <em>yourNamespace</em>:message. And finally, create operations so as to have three different operations (“start'”, “receiveMessageA” & “receiveMessageB”). </div><br />
<br />
<div align="center"><a href="http://lh5.ggpht.com/_bayCgqm3V20/TRZy0klYyEI/AAAAAAAADLs/l0GgLx-lctw/s1600-h/09-wsdlEditionDone4.png"><img alt="09-wsdlEditionDone" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZy2YvRB1I/AAAAAAAADLw/hVLunWM8xp8/09-wsdlEditionDone_thumb2.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="09-wsdlEditionDone" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">Our BPEL process seems ready. Let’s go back to the mediator : that’s not mandatory, but as it is cleaner, I want you to be aware of it : if we do not define a filter rule in the Mediator, it wll listen for each and every message, even if the static routing rules you have defined are not designed to process that message. Don’t worry, I will explain what a static routing rule is.</div><br />
<br />
<div align="center"><a href="http://lh5.ggpht.com/_bayCgqm3V20/TRZy3QqVgBI/AAAAAAAADL0/rw00nmArNpI/s1600-h/10b-mediatorCompleteFilter4.png"><img alt="10b-mediatorCompleteFilter" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZy4dL5W7I/AAAAAAAADL4/QcuDy_ZUl6E/10b-mediatorCompleteFilter_thumb2.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="10b-mediatorCompleteFilter" width="1028" /></a></div><br />
<div align="center" style="text-align: -webkit-auto;"><br />
</div><div align="left">To define a filter, simply enter the correct expression knowing it allows logic expression (“or” & “and”), so you can do pretty much what you want.</div><br />
<div align="center"><br />
</div><div align="center"><a href="http://lh6.ggpht.com/_bayCgqm3V20/TRZy5C_TneI/AAAAAAAADL8/FsC8ZcBMhhM/s1600-h/10-mediatorFilter4.png"><img alt="10-mediatorFilter" border="0" height="580" src="http://lh4.ggpht.com/_bayCgqm3V20/TRZy65RuSRI/AAAAAAAADMA/ykCg-BCBW4M/10-mediatorFilter_thumb2.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="10-mediatorFilter" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">Now we’ve filtered the messages we want to process in our Mediator, we have to define a proper route for each one (out of three).</div><br />
<div align="center"><br />
</div><div align="center"><a href="http://lh6.ggpht.com/_bayCgqm3V20/TRZy7siVw4I/AAAAAAAADME/XHPDj4Dq55Q/s1600-h/11-addStaticRoutingRule4.png"><img alt="11-addStaticRoutingRule" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZy8sOgl5I/AAAAAAAADMI/3bSsxL4bG2c/11-addStaticRoutingRule_thumb2.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="11-addStaticRoutingRule" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">When you create a static routing rule, you have to define if it will invoke a service (partnerLink) or fire an event. In our case, it’s a partnerLink (= a call to a webservice operation, which is one of our BPEL process operation)</div><div align="center"><br />
</div><br />
<div align="center"><a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZy9XQf0BI/AAAAAAAADMM/268NSKJTKAo/s1600-h/12-serviceRoutingRule4.png"><img alt="12-serviceRoutingRule" border="0" height="580" src="http://lh3.ggpht.com/_bayCgqm3V20/TRZy-UuHkTI/AAAAAAAADMQ/3TPud53UcQs/12-serviceRoutingRule_thumb2.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="12-serviceRoutingRule" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">Then, we have to select the proper operation, matching the filter we are going to define.</div><br />
<div align="center"><a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZy_atTEjI/AAAAAAAADMU/gI4YAdCE3R8/s1600-h/13-chooseOperation4.png"><img alt="13-chooseOperation" border="0" height="580" src="http://lh3.ggpht.com/_bayCgqm3V20/TRZzAdy_NpI/AAAAAAAADMY/vEcBClq_Q_c/13-chooseOperation_thumb2.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="13-chooseOperation" width="1028" /></a></div><br />
<blockquote>Note : OK, I wanted to show that error because it seems to me that it is a real bug : if you want to update the filter rule, once you’ve already defined, JDev will try to add another filter, which is forbidden. If you take a look at the example below, it tries to add another rule, in addition to the previous version. You may fix that by going in the source and remove the troubling filter.<br />
<br />
</blockquote><div align="center"><a href="http://lh5.ggpht.com/_bayCgqm3V20/TRZzBUfxZnI/AAAAAAAADMc/6U0W7DGKo00/s1600-h/14-errorWhenUpdatingFilterRule4.png"><img alt="14-errorWhenUpdatingFilterRule" border="0" height="580" src="http://lh3.ggpht.com/_bayCgqm3V20/TRZzCp-m79I/AAAAAAAADMg/zgYfKNgsOvs/14-errorWhenUpdatingFilterRule_thumb.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="14-errorWhenUpdatingFilterRule" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="center">Anyway, what you have to remember is that once created, the filter needs to be updated directly in the source (composite.xml), if needed. Maybe it’s just the JDev version I worked with, but if you’re using the same Virtualbox appliance as me, you might encounter the same issue.</div><br />
<div align="center"><br />
</div><br />
<div align="center"><a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZzEfqgGUI/AAAAAAAADMk/6gbs4KNNjCA/s1600-h/15-compositeUpdateRule3.png"><img alt="15-compositeUpdateRule" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZzFqz0M1I/AAAAAAAADMo/e3oXA3j9l8g/15-compositeUpdateRule_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="15-compositeUpdateRule" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">Once you’ve defined what operation was to be called, we need to define what the trigger will be (= define a filter). In our example, I want the “start” branch of my BPEL process to be called when a message, whose ID is ‘START’, is received (pretty logical, huh ?). Just click on the filter icon and you will see a popup like that :</div><br />
<div align="center"><br />
</div><div align="center"><a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZzHHCw20I/AAAAAAAADMs/y8CMnEvUJFA/s1600-h/16A-START3.png"><img alt="16A-START" border="0" height="580" src="http://lh4.ggpht.com/_bayCgqm3V20/TRZzIsEPg4I/AAAAAAAADMw/4xoB55BY9Ho/16A-START_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="16A-START" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">Repeat the step for each message / BPEL branch to be called. Note that I used the mode “Parallel'” rather than “Sequential”. With three rules, it doesn’t really matter. But if you have more than twenty, it could lead to some performance issues : in “Parallel”, one thread is created to execute the checking (one rule = one thread), whereas in “Sequential”, only one thread will perform each rule, one after the other.</div><br />
<div align="center"><br />
</div><div align="center"><a href="http://lh6.ggpht.com/_bayCgqm3V20/TRZzJtclZFI/AAAAAAAADM0/NhMGPTx28Rg/s1600-h/16B-AllDone3.png"><img alt="16B-AllDone" border="0" height="580" src="http://lh4.ggpht.com/_bayCgqm3V20/TRZzKhNHHsI/AAAAAAAADM4/C0dKdBfMpi0/16B-AllDone_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="16B-AllDone" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">Once you are done, note that all the components are now wired in the composite graphical view.</div><br />
<div align="center"><br />
</div><div align="center"><a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZzLurKr6I/AAAAAAAADM8/yGBZyhzmo1s/s1600-h/17-compositeUpdated3.png"><img alt="17-compositeUpdated" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZzM69jyMI/AAAAAAAADNA/ag4bdSKRkE0/17-compositeUpdated_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="17-compositeUpdated" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">Now, we will add another process, used to push messages on the EDN.</div><br />
Define a new BPEL process. (“One-way” is the right choice for that process since we are not expecting any message in return).<br />
<br />
<div align="center"><br />
</div><div align="center"><a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZzN7NZdVI/AAAAAAAADNE/ajiyq4A9utA/s1600-h/18-messageSender3.png"><img alt="18-messageSender" border="0" height="580" src="http://lh4.ggpht.com/_bayCgqm3V20/TRZzO5hOOQI/AAAAAAAADNI/xhwl4PODiX8/18-messageSender_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="18-messageSender" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">Simply add an “Invoke” activity by drag ‘n’ dropping it just after the activity “receiveInput”. An “Invoke” activity means that the process is making a call, whereas a “Receive” activity means that the process is being called.</div><br />
<div align="left">In our example, the Invoke activity will publish a message (Event) on the EDN. Just switch “PartnerLink” to “Event” in the dropdown “Interaction Type”.</div><br />
<div align="left">Look for the event “myMessage” and then, define a variable associated to it (even if we will not use it). Be sure to update the WSDL in order to have the proper message used. (that means you’ll have to declare and import the correct namespace, like you did for the first BPEL process).</div><br />
<div align="center"><br />
</div><div align="center"><a href="http://lh6.ggpht.com/_bayCgqm3V20/TRZzP1N8TuI/AAAAAAAADNM/c-WVTH_kFC0/s1600-h/19-invoke3.png"><img alt="19-invoke" border="0" height="580" src="http://lh4.ggpht.com/_bayCgqm3V20/TRZzRm0zcKI/AAAAAAAADNU/upYW_dYSOC8/19-invoke_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="19-invoke" width="1028" /></a></div><div align="center"><br />
</div><br />
From a composite point-of-view, here is the the newly created process, which will help us to send messages over the EDN.<br />
<div align="center"><br />
</div><br />
<div align="center"><a href="http://lh5.ggpht.com/_bayCgqm3V20/TRZzSpX47iI/AAAAAAAADNY/xNDj34SvzSk/s1600-h/20-composite3.png"><img alt="20-composite" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZzTpSA4iI/AAAAAAAADNc/RY1dFEcXqWY/20-composite_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="20-composite" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">Now, we have the impression that everything is set up : a process which will receive messages (through its mediator) and another which will publish messages. But if you remember what I wrote earlier, you should know that this won’t be enough for this example to work : a correlationSet is missing …</div><br />
For your messages to be able to reach your instance, you have to define something that bind them together. That’s the role of the correlationSet : you define a common property, whose value is going to be the same.<br />
<br />
For instance, on my message, I’ll define a correlationSet on the property “label”. That way, all the messages whose “label” property is the same will be directed to the same BPEL instance(s).<br />
<br />
<blockquote><br />
<div align="left">Note : several BPEL instances may have the same correlationSet.</div><br />
</blockquote><ol><li><div align="left">On the “Receive” node, look for the “Correlations” tab, click on the starred white sheet to create a new correlationSet</div><br />
</li>
<li><div align="left">Chose a name, type & alias</div><br />
</li>
<li><div align="left">Chose on what message property your correlationSet will rely on</div><br />
</li>
</ol><br />
<a href="http://lh5.ggpht.com/_bayCgqm3V20/TRZzUtpYQfI/AAAAAAAADNg/5RhKf2xSBNs/s1600-h/21-CorrelationSet3.png"><img alt="21-CorrelationSet" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZzWSft2GI/AAAAAAAADNk/o-8vEz1F7M4/21-CorrelationSet_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="21-CorrelationSet" width="1028" /></a><br />
<div align="center"><br />
</div><br />
<div align="left">Add a flow element (means its branches will be processed in parallel) and define two branches. In each branch, add a receive activity. Here’s what our BPEL Process should look like.</div><div align="center"><br />
</div><br />
<div align="center"><a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZzXTDoCYI/AAAAAAAADNo/jE7qW77pCOc/s1600-h/22-Receive3.png"><img alt="22-Receive" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZzYdcRyxI/AAAAAAAADNs/Rolmqy34BdI/22-Receive_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="22-Receive" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">Now, just wire the newly created receive operations to the messageClient.</div><br />
<div align="center"><br />
</div><div align="center"><a href="http://lh6.ggpht.com/_bayCgqm3V20/TRZzZSsatbI/AAAAAAAADNw/uR78mFROkuw/s1600-h/23-LinkReceiveA4.png"><img alt="23-LinkReceiveA" border="0" height="580" src="http://lh6.ggpht.com/_bayCgqm3V20/TRZzaYAuPLI/AAAAAAAADN0/4gdhNeGkghM/23-LinkReceiveA_thumb2.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="23-LinkReceiveA" width="1028" /></a></div><div align="center"><br />
</div><br />
When the wire is drawn, a popup will ask some details about that link. Chose the webservice operation you want your receive activity to be bound to.<br />
<br />
<div align="center"><br />
</div><div align="center"><a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZzbtIvaeI/AAAAAAAADN4/C4MPMvPnEuY/s1600-h/24-ChooseOperationA4.png"><img alt="24-ChooseOperationA" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZzci8w2_I/AAAAAAAADN8/MQRPFgZ0Ai4/24-ChooseOperationA_thumb2.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="24-ChooseOperationA" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">Create a variable that will receive the payload coming from the webservice operation call.</div><br />
<div align="center"><br />
</div><div align="center"><a href="http://lh5.ggpht.com/_bayCgqm3V20/TRZzdzsD4yI/AAAAAAAADOA/VkALydp-Jpk/s1600-h/25-ReceiveAVariable3.png"><img alt="25-ReceiveAVariable" border="0" height="580" src="http://lh6.ggpht.com/_bayCgqm3V20/TRZzfeFqeLI/AAAAAAAADOE/Q1tsQm8VLw4/25-ReceiveAVariable_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="25-ReceiveAVariable" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">And then, chose the correlationSet you created before. That’s how you define a link between the three messages. Repeat that step for the second operation.</div><div align="center"><br />
</div><div align="center"><a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZzgfjvwgI/AAAAAAAADOI/3K1iBz0jriI/s1600-h/26-setCorrelationSet3.png"><img alt="26-setCorrelationSet" border="0" height="580" src="http://lh4.ggpht.com/_bayCgqm3V20/TRZzhtt3QJI/AAAAAAAADOQ/8PvoMasCXGY/26-setCorrelationSet_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="26-setCorrelationSet" width="1028" /></a></div><br />
<div align="center"><br />
</div><div align="left">Here’s what our final BPEL Process looks like :</div><div align="center"><br />
</div><div align="center"><a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZzih2DKCI/AAAAAAAADOU/rUSvJw6AwEU/s1600-h/27-BpelProcessComplete3.png"><img alt="27-BpelProcessComplete" border="0" height="580" src="http://lh4.ggpht.com/_bayCgqm3V20/TRZzkMnRX3I/AAAAAAAADOY/QzYCc33Bwks/27-BpelProcessComplete_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="27-BpelProcessComplete" width="1028" /></a></div><br />
<br />
Now we’re ready to have it running.<br />
<br />
Start your server and deploy your BPEL project.<br />
<br />
<br />
<a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZzlBIEcNI/AAAAAAAADOc/15m1uxTbtVw/s1600-h/28-Deploy7.png"><img alt="28-Deploy" border="0" height="580" src="http://lh6.ggpht.com/_bayCgqm3V20/TRZzmRcS7LI/AAAAAAAADOg/acoT3N2RLOc/28-Deploy_thumb5.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="28-Deploy" width="1028" /></a><br />
<br />
<br />
Don’t forget to define the version you want to use for your SOA Project<br />
<br />
<br />
<a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZznj0_wVI/AAAAAAAADOk/0Sg7CMww_Rc/s1600-h/29-DeployVersion6.png"><img alt="29-DeployVersion" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZzogTwSjI/AAAAAAAADOo/RW-BbL8-84w/29-DeployVersion_thumb2.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: inline; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="29-DeployVersion" width="1028" /></a><br />
<br />
<br />
In the SOA Pane, you should see seomething like <br />
<br />
<pre style="background-color: #fbfbfb; border-bottom: #cecece 1px solid; border-left: #cecece 1px solid; border-right: #cecece 1px solid; border-top: #cecece 1px solid; min-height: 40px; overflow: auto; padding-bottom: 5px; padding-left: 5px; padding-right: 5px; padding-top: 5px; width: 1000px;"><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;">Buildfile: /oracle/jdevhome/jdeveloper/bin/ant-sca-compile.xml
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"></pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;">scac:
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;">[scac] Validating composite "/home/oracle/jdeveloper/mywork/Migration/BPELFromWLI/composite.xml"
</pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;">[scac] Setting BPELC option 'classpath' to /oracle/jdevhome/jdeveloper/jdev/extensions/oracle.sca.modeler.jar:/oracle/jdevhome/jdeveloper/soa/modules/oracle.soa.fabric_11.1.1/fabric-runtime.jar:/oracle/jdevhome/jdeveloper/soa/modules/oracle.soa.mgmt_11.1.1/soa-infra-mgmt.jar:/oracle/jdevhome/oracle_common/modules/oracle.fabriccommon_11.1.1/fabric-common.jar:/oracle/jdevhome/jdeveloper/soa/modules/oracle.soa.bpel_11.1.1/orabpel.jar:/oracle/jdevhome/jdeveloper/soa/modules/oracle.soa.mediator_11.1.1/mediator_client.jar:/oracle/jdevhome/oracle_common/modules/oracle.mds_11.1.1/mdsrt.jar::/home/oracle/jdeveloper/mywork/Migration/BPELFromWLI/SCA-INF/classes
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;">[scac] Setting BPELC option 'classpath' to /oracle/jdevhome/jdeveloper/jdev/extensions/oracle.sca.modeler.jar:/oracle/jdevhome/jdeveloper/soa/modules/oracle.soa.fabric_11.1.1/fabric-runtime.jar:/oracle/jdevhome/jdeveloper/soa/modules/oracle.soa.mgmt_11.1.1/soa-infra-mgmt.jar:/oracle/jdevhome/oracle_common/modules/oracle.fabriccommon_11.1.1/fabric-common.jar:/oracle/jdevhome/jdeveloper/soa/modules/oracle.soa.bpel_11.1.1/orabpel.jar:/oracle/jdevhome/jdeveloper/soa/modules/oracle.soa.mediator_11.1.1/mediator_client.jar:/oracle/jdevhome/oracle_common/modules/oracle.mds_11.1.1/mdsrt.jar::/home/oracle/jdeveloper/mywork/Migration/BPELFromWLI/SCA-INF/classes
</pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;">[scac] WARNING: in Router.mplan: Case "MessageReceiver.messagereceiver_client.start" doesnt have any payload transformation Please make sure source and target message part name are same and of same type. Otherwise, target reference may fail to execute with error message like "Input sourcelike Null" or "Part not found"
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;">[scac] WARNING: in Router.mplan: Case "MessageReceiver.messagereceiver_client.receiveMessageB" doesnt have any payload transformation Please make sure source and target message part name are same and of same type. Otherwise, target reference may fail to execute with error message like "Input sourcelike Null" or "Part not found"
</pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;">[scac] WARNING: in Router.mplan: Case "MessageReceiver.messagereceiver_client.receiveMessageA" doesnt have any payload transformation Please make sure source and target message part name are same and of same type. Otherwise, target reference may fail to execute with error message like "Input sourcelike Null" or "Part not found"
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;"></pre><pre style="background-color: #fbfbfb; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;">BUILD SUCCESSFUL
</pre><pre style="background-color: white; font-family: consolas,'Courier New',courier,monospace; font-size: 12px; margin: 0em; width: 100%;">Total time: 6 seconds</pre></pre><br />
And in the Compiler Pane, no errors (but maybe a few warnings).<br />
<br />
<span class="Apple-style-span" style="font-size: 19px; font-weight: bold;">2.2.4) Running the example</span><br />
<br />
Now we’ve compiled & deployed our BPEL processes, it’s time to use them.<br />
<br />
<br />
<a href="http://lh6.ggpht.com/_bayCgqm3V20/TRZzpqL9rSI/AAAAAAAADOs/PC4INAes0zo/s1600-h/30-EM3.png"><img alt="30-EM" border="0" height="580" src="http://lh6.ggpht.com/_bayCgqm3V20/TRZzqSvSjqI/AAAAAAAADOw/5kSTZD25-lU/30-EM_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="30-EM" width="1028" /></a><br />
<br />
First thing of all, we will test the process that sends messages over the EDN and send a start message.<br />
<br />
To test a BPEL process, just click on the Test button, just like on the picture below.<br />
<br />
<a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZzrRfTDuI/AAAAAAAADO0/TIbuKvKJQRk/s1600-h/31-Test3.png"><img alt="31-Test" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZztfsFvYI/AAAAAAAADO4/C7SvWmJP9hc/31-Test_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="31-Test" width="1028" /></a><br />
<br />
<br />
Choose the process that sends messages and on the screen, chose to the XML view, that would normally display an empty message. Just complete the payload with the correct values, that is to say ‘START’ in the id field and whatever you want in the label field. Remember that, as we based our correlationSet on that label field, you will have to keep the same value if you want your messages to be routed to the correct instance.<br />
<br />
<br />
<a href="http://lh5.ggpht.com/_bayCgqm3V20/TRZzub5dbdI/AAAAAAAADPA/wPYfWJrjRqw/s1600-h/32-XMLView3.png"><img alt="32-XMLView" border="0" height="580" src="http://lh6.ggpht.com/_bayCgqm3V20/TRZzvKaLqlI/AAAAAAAADPE/fbMPofpfz1w/32-XMLView_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="32-XMLView" width="1028" /></a><br />
<br />
If everything went well, click on the link that allows you to see the details of the call. You should normally see the steps shown below, with the call of the “MessageSender”, the routing by the “Router” mediator and then finally, the process “MessageReceiver” which is running.<br />
<br />
<br />
<a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZzwNaY3xI/AAAAAAAADPI/XDblBXSVA2E/s1600-h/33-ProcessStarted3.png"><img alt="33-ProcessStarted" border="0" height="580" src="http://lh3.ggpht.com/_bayCgqm3V20/TRZzx_zhKKI/AAAAAAAADPM/WNHAH6ibBLw/33-ProcessStarted_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="33-ProcessStarted" width="1028" /></a><br />
<br />
<br />
If you click on that last step, you will see that the start branch has been called and that the process is now waiting for two branches (highlighted in yellow) to be called.<br />
<br />
<a href="http://lh5.ggpht.com/_bayCgqm3V20/TRZzy-sbxmI/AAAAAAAADPQ/UXV2-jpjfS4/s1600-h/34-MessagesPending3.png"><img alt="34-MessagesPending" border="0" height="580" src="http://lh4.ggpht.com/_bayCgqm3V20/TRZzz2iGRKI/AAAAAAAADPU/vjrq5xdJ90I/34-MessagesPending_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="34-MessagesPending" width="1028" /></a><br />
<br />
<br />
Just for some testing purposes, if you try to trigger the very same message, you’ll end up with a faulted process, whose failure reason is a conflicting receive. Which is absolutely normal, since we based our correlationSet on a field whose value is the same as the first we used to trigger the first process. If you want another process to start, change the value of the “label” field.<br />
<br />
<br />
<a href="http://lh4.ggpht.com/_bayCgqm3V20/TRZz0v4nYII/AAAAAAAADPY/cXFTHK3fxCU/s1600-h/35-SecondCall3.png"><img alt="35-SecondCall" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZz1h0gkyI/AAAAAAAADPc/BVrKjjs6Vqw/35-SecondCall_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="35-SecondCall" width="1028" /></a><br />
<br />
<br />
Now, if we send a message whose id is one the ids expected (and of course, with the same label), you should see something like the picture below, with a “mid process receive” activity.<br />
<br />
<br />
<a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZz2nuWieI/AAAAAAAADPg/O7c7k9UMY8o/s1600-h/36-MidProcessReceive3.png"><img alt="36-MidProcessReceive" border="0" height="580" src="http://lh5.ggpht.com/_bayCgqm3V20/TRZz3qVMChI/AAAAAAAADPk/YogOlrpg9qQ/36-MidProcessReceive_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="36-MidProcessReceive" width="1028" /></a><br />
<br />
<br />
If we take a closer look to that running instance, we now see that the first branch is no longer waiting to be called (= for a message to come).<br />
<br />
<br />
<a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZz4xddYPI/AAAAAAAADPo/5HzXAM8ml8U/s1600-h/37-IDA_Received3.png"><img alt="37-IDA_Received" border="0" height="580" src="http://lh3.ggpht.com/_bayCgqm3V20/TRZz55kloiI/AAAAAAAADPs/uOLjnAnvT5s/37-IDA_Received_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="37-IDA_Received" width="1028" /></a><br />
<br />
<br />
Now, we send the third and last message, whose id is “ID_B” and we can now see that a second “mid process receive” is visible and that all the states of the steps are at “Completed”.<br />
<br />
<br />
<a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZz7I2QVoI/AAAAAAAADPw/QG-P0g4ERMg/s1600-h/38-BothMsgReceived3.png"><img alt="38-BothMsgReceived" border="0" height="580" src="http://lh4.ggpht.com/_bayCgqm3V20/TRZz8WeIXOI/AAAAAAAADP0/2BFi5e_rIeo/38-BothMsgReceived_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="38-BothMsgReceived" width="1028" /></a><br />
<br />
<br />
Let’s have a look at the general BPEL instance, and now, we can see that everything has been received and that the instance is now completed.<br />
<br />
<br />
<a href="http://lh3.ggpht.com/_bayCgqm3V20/TRZz9-gBAbI/AAAAAAAADP4/UMSSHGGjywk/s1600-h/39-ProcessCompleted3.png"><img alt="39-ProcessCompleted" border="0" height="580" src="http://lh3.ggpht.com/_bayCgqm3V20/TRZz_Wcfa-I/AAAAAAAADP8/7BTiYpLZfM4/39-ProcessCompleted_thumb1.png?imgmax=800" style="background-image: none; border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-width: 0px; display: block; float: none; margin-left: auto; margin-right: auto; padding-left: 0px; padding-right: 0px; padding-top: 0px;" title="39-ProcessCompleted" width="1028" /></a><br />
<br />
<br />
Well, that’s it. I hope my explanation was clear enough. Else, feel free to ask and I’ll try to answer as accurately and as fast as possible.<br />
<br />
In the meantime, if you were to wonder about Oracle products, about the way to migrate WLI projects toward BPEL PM, I invite you to contact Simone Geib (SOA Suite Product Manager) and she’ll try to answer your questions. (Her email address is <a href="mailto:first_name.last_name@oracle.com">first_name.last_name@oracle.com</a>)<br />
<br />
Ok, to be sure to understand how everything works here, don’t hesitate to play with correlationSets, multiple instance and so on. Once again, you’ve got a perfect VirtualBox appliance to play with !<br />
<br />
Enjoy :)<br />
<br />
<br />
<br />
<div class="wlWriterEditableSmartContent" id="scid:0767317B-992E-4b12-91E0-4F059A8CECA8:b7443ea9-d418-41d5-86a6-30fd547c363d" style="display: inline; float: none; margin: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">Mots clés Technorati : <a href="http://technorati.com/tags/soa+suite+11gR1PS2" rel="tag">soa suite 11gR1PS2</a>,<a href="http://technorati.com/tags/wli" rel="tag">wli</a>,<a href="http://technorati.com/tags/migration" rel="tag">migration</a>,<a href="http://technorati.com/tags/correlationSet" rel="tag">correlationSet</a>,<a href="http://technorati.com/tags/bpel" rel="tag">bpel</a>,<a href="http://technorati.com/tags/mediator" rel="tag">mediator</a></div>Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com2tag:blogger.com,1999:blog-2592381182145606344.post-53451143801669784262010-11-21T22:43:00.001+01:002010-11-21T22:47:17.773+01:00Upcoming Packt reviews<p>It’s been quite some time I’ve been woking with <a href="http://www.packtpub.com/" target="_blank">Packt Publishing</a>. I started reviewing books in July 2008 and we are still in touch now.</p> <p>In a few weeks, I am about to review two books which are strongly related to what I am working on right now : the migration from WLI to BPEL Process Manager. It’s quite a hard task since those two products have two different philosophies. But, last week, I had the chance to work with <a href="http://www.linkedin.com/profile/view?id=4067843&authType=name&authToken=JSf3" target="_blank">Simone Geib</a>, <a href="http://goo.gl/qSj50" target="_blank">Deepak d’Arora</a>, <a href="http://www.linkedin.com/profile/view?id=3071138&authType=name&authToken=Neza" target="_blank">Laurence Habdas</a> & <a href="http://goo.gl/o6MG8" target="_blank">Eric Fischer</a> about the steps to follow to migrate successfully from BEA era to Oracle : it was really interesting.</p> <p>The two books are both been authored by guys from the same team. </p> <h3>Getting Started with Oracle BPM 11gR1</h3> <p><strong>Authors</strong> : <a href="https://www.packtpub.com/authors/profiles/heidi-buelow">Heidi Buelow</a>, <a href="https://www.packtpub.com/authors/profiles/manoj-das">Manoj Das</a>, <a href="https://www.packtpub.com/authors/profiles/manas-deb">Manas Deb</a>, <a href="https://www.packtpub.com/authors/profiles/prasen-palvankar">Prasen Palvankar</a>, <a href="https://www.packtpub.com/authors/profiles/meera-srinivasan">Meera Srinivasan</a></p> <p> </p> <blockquote> <p><a href="http://link.packtpub.com/YWvj2F" target="_blank"><img style="display: inline; margin-left: 0px; margin-right: 0px" alt="Getting Started with Oracle BPM Suite 11gR1 – A Hands-On Tutorial" src="https://www.packtpub.com/sites/default/files/imagecache/productview/1681EN_Getting%20started%20with%20oraclecov_LowRes.jpg" width="125" height="152" /></a> </p> </blockquote> <p>I intend to read carefully the parts dealing with BPMN and the BAM.</p> <p>Although I’m more BPEL than BPMN, I guess it will be interesting to develop my knowledge on that topic.</p> <p> </p> <h3>Getting Started with Oracle SOA Suite 11gR1 (PS2)</h3> <p><b>Authors : </b><a href="https://www.packtpub.com/authors/profiles/demed-lher">Demed L'Her</a>, <a href="https://www.packtpub.com/authors/profiles/heidi-buelow">Heidi Buelow</a>, <a href="https://www.packtpub.com/authors/profiles/jayaram-kasi">Jayaram Kasi</a>, <a href="https://www.packtpub.com/authors/profiles/manas-deb">Manas Deb</a>, <a href="https://www.packtpub.com/authors/profiles/prasen-palvankar">Prasen Palvankar</a></p> <blockquote> <p> </p> <p><a href="http://www.packtpub.com/getting-started-with-oracle-soa-suite-11g-r1/book" target="_blank"><img src="https://www.packtpub.com/sites/default/files/imagecache/productview/bookimages/9782_Getting%20Started%20With%20Oracle%20SOA%20Suite%2011gR.jpg" /></a></p> </blockquote> <p></p> <p>I started reading this one a few weeks (months ?) ago but I have not been able to finish it yet. Not that it was not interesting, but I only read some parts, that were useful for some of my tests. </p> <p>However, I can already tell you that it has a great coverage of all the underlying components (BPEL, Mediator, Rules, Workflow …) but I was a bit disappointed by the depth of some parts, especially the one dealing with EDN.</p> <p>But ok, I’ll say no more and keep some for the review.</p> <p> </p> <p>Stay tuned ! :)</p> Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com0tag:blogger.com,1999:blog-2592381182145606344.post-62420448846114708992010-10-12T21:02:00.002+02:002010-10-12T21:11:34.083+02:00Seeing the big picture !Lately, I've been pretty busy working on a major project which went live (successfully) a few hours ago.<br />
(that's also why I didn't blog much these last months)<br />
We worked hard on it, but thanks to Murphy's law, we had a hard time two weeks before the go-live.<br />
It started with the first user-testing week : everything was running nice & smooth but in the last minute (quite unbelievable),<br />
we ended up with a blocking exception, which forced us to use some workaround techniques.<br />
The exception was raised when a JMS message was sent from WLS to ALSB.<br />
 <br />
<br />
<br />
Here’s our architecture (sorry for the ugly stuff : at least, I know I will never be a web designer !)<br />
<br />
 <br />
<br />
<a href="http://lh4.ggpht.com/_bayCgqm3V20/TLSwzSveAyI/AAAAAAAACR8/hJAzY_4WSSE/s1600-h/EAI_ALSB%5B2%5D.jpg"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="EAI_ALSB" border="0" alt="EAI_ALSB" src="http://lh4.ggpht.com/_bayCgqm3V20/TLSw0C7vUdI/AAAAAAAACSA/eNPzyc5iHuo/EAI_ALSB_thumb%5B2%5D.jpg?imgmax=800" width="927" height="586" /></a> <br />
<br />
<br />
<br />
 <br />
<br />
<br />
<blockquote><br />
* In green is the “Send” mechanism : when WebMethods wants to send a message to our system, <br />
it first contacts the Alteon, which load-balances the request (with a server affinity) to a node, <br />
then looks up in the JNDI tree for a connection factory, <br />
which load-balances in its turn between the two nodes.<br />
* In blue is the other mechanism, used to receive messages from our system. <br />
As we use the same connection factory, <br />
we have almost no guarantee to know where the messages are going to land. <br />
We could have used the “message forward” system, embodied by the option “Forward Delay”,<br />
but our system is particular : consumers are only consuming messages based on a JMS header, <br />
so we could have a consumer connected but no one reading the messages.<br />
(that’s why “Forward Delay” is useless in our case)<br />
Then we had to have all the connectors doubled, to have both nodes served. <br />
<br />
<br />
And to ensure a proper connection on each node, we had to create a connection factory on each.<br />
</blockquote> <br />
Normally, when a message arrives, ALSB tries to persist the message into database (to be able to restore the message in case of crash).<br />
During that normally basic operation, we had the following stacktrace :<br />
<blockquote><br />
<pre>Caused by: weblogic.messaging.kernel.KernelException: Error persisting message
... 9 more
Caused by: weblogic.store.PersistentStoreFatalException: [Store:280065]java.sql.SQLException: ORA-01098: </pre><br />
<pre>erreur d'interface de programme lors d'une insertion longue
ORA-12155: TNS : type de données incorrect dans le paquet MSWMARKER reçu
(server="ALSB2a-01-COCESB" store="JDBCStore-0" table="jms_alsb1WLStore"):(Linked Cause, </pre><br />
<pre>"java.sql.SQLException: ORA-01098: erreur d'interface de programme lors d'une insertion longue
ORA-12155: TNS : type de données incorrect dans le paquet MSWMARKER reçu
")
at weblogic.store.io.jdbc.JDBCStoreIO.flush(JDBCStoreIO.java:1405)
at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:640)
at java.lang.Thread.run(Thread.java:595)</pre><br />
</blockquote><br />
And we were able to reproduce it everytime on the A-Node!<br />
As we could not really understand why it happened, we decided to cut that communication by switching to a filestore. <br />
It was clear that this solution was a poor workaround since we didn't understand the conditions in which that exception occurred.<br />
We spent three whole days trying to figure out why and we reached a partial conclusion : it happens only on the A-Node, with a JDBCStore. <br />
With the filestore configured, everything seemed to be ok and we started to think this problem was behind us ... we were unfortunately mistaken ...<br />
A week after that, while WebMethods was retrieving a JMS message from the bus, we noticed that it was reaaally slow. <br />
And finally, we had an exception :<br />
<br />
<pre> </pre><br />
<blockquote><pre>java.lang.Exception: weblogic.jms.common.JMSException: weblogic.messaging.dispatcher.DispatcherException: </pre><pre>weblogic.rjvm.PeerGoneException: ; nested exception is: java.io.IOException: </pre><pre>A complete message could not be read on socket: </pre><pre><a href="mailto:'weblogic.rjvm.t3.MuxableSocketT3@1f42269:Socket[addr=/10.203.3.143,port=9101,localport=57568]'">'weblogic.rjvm.t3.MuxableSocketT3@1f42269:Socket[addr=/10.203.3.143,port=9101,localport=57568]'</a>, </pre><pre>in the configured timeout period of '60' secs </pre><br />
</blockquote><br />
The JMS message was approximatively 3Mb but we already passed the test with a 15Mb message. Thus, the message size was not involved here.<br />
We first thought that it was a temporary network problem (latency) and then we tried first to increase a timeout value,<br />
embodied by the option "<strong>Complete Message Timeout</strong>" <br />
 <br />
<br />
<br />
<pre><a href="http://lh4.ggpht.com/_bayCgqm3V20/TLSw0jW18zI/AAAAAAAACSE/eHRCU9cTVdQ/s1600-h/CompleteMessageTimeout%5B6%5D.jpg"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="CompleteMessageTimeout" border="0" alt="CompleteMessageTimeout" src="http://lh6.ggpht.com/_bayCgqm3V20/TLSw1GxJOaI/AAAAAAAACSI/V3gre4brtMc/CompleteMessageTimeout_thumb%5B4%5D.jpg?imgmax=800" width="281" height="193" /></a> </pre><br />
 <br />
<br />
We shutted / re-started our servers and the first test passed. After several other tests, the exception showed up again, with the same value.<br />
I read that sometimes this option had to be set thanks to a JVM property. (<strong>-Dweblogic.CompleteMessageTimeout)</strong>.<br />
Then we tried that, and after a restart, and a few retries, the exception raised again. <br />
We also changed some options from the WebMethod side, but nothing changed. <br />
We then tried on another environment (prod mirror) and the message was consumed in less than five seconds.<br />
We switched the prod EAI toward the second environment : everything went fine too. <br />
Then we tried on the second node of the ALSB cluster in our production environment, B-Node and the attempt was ok as well (= fast consumed). <br />
 <br />
From those two exceptions (outgoing communications with the database and the EAI), we started to question the network configuration, <br />
since every outgoing communication presented a problem on that node, which is on machine A.<br />
We used the netstat command to get some information and we noticed something interesting : <br />
the number of collisions on machine A was abnormal because far too high.<br />
<br />
<blockquote><br />
<pre>Machine A
netstat -ni
<strong>Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue</strong>
lo0 8232 127.0.0.0 127.0.0.1 17943151 0 17943151 0 0 0
e1000g0 1500 10.203.3.0 10.203.3.141 880127761 0 385849067 80 <font color="#ff0000">6141032</font> 0
e1000g1 1500 10.203.72.0 10.203.72.28 60799620 0 199220059 0 0 0
e1000g4 1500 10.203.3.0 10.203.3.142 537705356 1651 8356636 0 0 0
Machine B
netstat -ni
<strong>Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue</strong>
lo0 8232 127.0.0.0 127.0.0.1 1457911 0 1457911 0 0 0
e1000g0 1500 10.203.3.0 10.203.3.144 857275091 0 380178757 0 0 0
e1000g1 1500 10.203.72.0 10.203.72.29 50600947 0 158826403 0 0 0
e1000g4 1500 10.203.3.0 10.203.3.145 537677014 2344 8336692 0 0 0</pre><br />
</blockquote> <br />
<br />
We didn't have the rights to execute the command "<a href="http://www.brandonhutchinson.com/Solaris_NIC_speed_and_duplex_settings.html" target="_blank">ndd</a>" so we asked our hosting provider to do so and then he came up with something :<br />
After some talks, a guy from the hosting provider came up with an interesting fact :<br />
"e1000g0 is configured in 100/HALF instead of 100/FULL"<br />
In a word, the network card is in half-duplex whereas it should have been in full-duplex. <br />
If you are unfamiliar with those terms, just think about a one-way road with a traffic light, <br />
where a way can only pass at one at a time : this is half duplex.<br />
With full-duplex, you have a two-way road, with no traffic light ... <br />
You can imagine which one is better in terms of fluidity, can't you ?<br />
 <br />
To confirm that our problems came from that configuration, <br />
we tried one last time to send a JMS message and have it consumed by the EAI and the result was pretty clear : <br />
the message was consumed as fast as on the other platforms ...<br />
We have not tested the JDBCStore yet, but we will in the next few weeks.<br />
 <br />
 <br />
As a conclusion, I would say that no matter how skilled you are in your field of expertise, it is not enough. <br />
You have to know more about the greater perimeter and be able to see the big picture to be more efficient.<br />
(work especially on the network aspect ! :p)Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com1tag:blogger.com,1999:blog-2592381182145606344.post-78237010741636311832010-05-14T16:38:00.001+02:002010-05-16T23:52:28.754+02:00Installing SOA Suite 11gR1 PS2 on Windows Se7en (64 bits)<p> </p> <p><a href="http://lh5.ggpht.com/_bayCgqm3V20/S-1f2Gv7jVI/AAAAAAAACJs/XoGTRpHO-8o/s1600-h/image%5B88%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/_bayCgqm3V20/S-1f2vEkPJI/AAAAAAAACJw/hzVFRPYZ0qY/image_thumb%5B40%5D.png?imgmax=800" width="244" height="67" /></a> <a href="http://lh3.ggpht.com/_bayCgqm3V20/S-1f3L3B8-I/AAAAAAAACJ0/WIPpcJT4woo/s1600-h/image%5B94%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_bayCgqm3V20/S-1f3lJmecI/AAAAAAAACJ4/h3UjA3rnExw/image_thumb%5B42%5D.png?imgmax=800" width="215" height="88" /></a> </p> <p> </p> <p>When I first started to configure the SOA Suite, it was a bit … messy (I tried to manually migrate 10g on WLS). </p> <p>Now the brand new 11gR1PS2  has been out for a couple of weeks, I wanted to install and test it on my W7 64 bits laptop.</p> <p>But even if I found some documentation (<a href="http://blogs.oracle.com/SOA/2009/08/installing_oracle_soa_suite_11.html" target="_blank">post by Demed L’Her</a> and the <a href="http://download.oracle.com/docs/cd/E14571_01/install.1111/b32474/start.htm#CIHBFJCF" target="_blank">official documentation</a>),</p> <p>it wasn’t that easy to find a proper modus operandi for a 64 bits Windows platform.</p> <p>On top of that, the latest patchset (PS) is not autonomous : you have to install the previous PS first.</p> <p> </p> <h2>1) Downloading</h2> <p> </p> <p>First step : get the required files ! </p> <p>Fortunately, you don’t need to download all the files composing the PS1 and the PS2.</p> <ol> <li>Download a 64 bits JDK from <a href="http://java.sun.com/javase/downloads/widget/jdk6.jsp" target="_blank">Sun</a> </li> <li>Go to <a href="http://www.oracle.com/technology/software/products/middleware/htdocs/fmw_11_download.html" target="_blank">OTN</a> and download the following products : <ul> <li><a href="http://download.oracle.com/otn/nt/middleware/11g/ofm_soa_generic_11.1.1.2.0_disk1_1of1.zip" target="_blank">SOA Suite PS1 (11.1.1.2.0)</a> </li> <li><a href="http://download.oracle.com/otn/nt/middleware/11g/ofm_soa_generic_11.1.1.3.0_disk1_1of1.zip" target="_blank">SOA Suite PS2 (11.1.1.3.0)</a> </li> <li><a href="http://download.oracle.com/otn/nt/middleware/11g/wls1033_generic.jar" target="_blank">WebLogic Server (10.3.3) generic</a> </li> <li><a href="http://download.oracle.com/otn/nt/middleware/11g/ofm_rcu_win_11.1.1.3.0_disk1_1of1.zip" target="_blank">Repository Creation Utility (11.1.1.3.0)</a> </li> <li><a href="http://www.oracle.com/technology/software/products/jdev/htdocs/soft11.html" target="_blank">Oracle JDeveloper 11g (11.1.1.3.0)</a> </li> <li><a href="http://download.oracle.com/otn/nt/oracle10g/xe/10201/OracleXEUniv.exe" target="_blank">Oracle XE (latest version)</a> </li> </ul> </li> <li>Optional but recommended : <a href="http://download.oracle.com/otn/java/sqldeveloper/sqldeveloper64-2.1.1.64.45-no-jre.zip" target="_blank">SQL Developer</a>. </li> </ol> <p> </p> <h2>2) Installing</h2> <p> </p> <h3>2.1) JDK 64 Bits</h3> <p> </p> <p>First of all, we have to install the JDK 64 bits.</p> <p>Wherever you want to install your JDK, it will install a part of your JDK in the directory “Program Files” (<strong>C:\Program Files\Java\jdk1.6.0_20</strong>)</p> <p><strong><em><u>Note</u></em></strong> : “Program Files (x86)” is for 32 bits software and “Program Files” is for 64 bits software.</p> <p> </p> <h3>2.2) JDeveloper & embedded WebLogic Server (64 bits)</h3> <p> </p> <p>To execute the installer in 64 bits, you have to specify the option “-D64”, such as : <strong>java –D64 –jar jdevstudio11113install.jar</strong></p> <p> </p> <p><a href="http://lh3.ggpht.com/_bayCgqm3V20/S-1f4LaxWHI/AAAAAAAACJ8/mYExlw45IPc/s1600-h/image%5B27%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_bayCgqm3V20/S-1f4oRuQzI/AAAAAAAACKA/EeSrIsOeJPQ/image_thumb%5B9%5D.png?imgmax=800" width="599" height="47" /></a> </p> <p> </p> <p>Choose a proper middleware home. I advise you to chose a directory structure like this one :</p> <p> </p> <p><a href="http://lh6.ggpht.com/_bayCgqm3V20/S-1f43NpyBI/AAAAAAAACKE/W4Q4CB4uhKc/s1600-h/image%5B129%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh3.ggpht.com/_bayCgqm3V20/S-1f5UgAmLI/AAAAAAAACKI/0gd9Yjoqa-U/image_thumb%5B59%5D.png?imgmax=800" width="214" height="82" /></a> </p> <p> </p> <p>As JDeveloper comes with its own version, it is important to prevent any conflict between both WLS versions (according to <a href="http://www.linkedin.com/in/demed" target="_blank">Demed L’Her</a>).</p> <p> </p> <p>As we downloaded the generic version of JDeveloper, there is no bundled JDK (which is pretty much what we wanted since we already installed one)</p> <p>Therefore, the installer will ask you for a JDK location : chose the freshly installed one (normally located in <strong>C:\Program Files\Java</strong>) : </p> <p> </p> <p><a href="http://lh3.ggpht.com/_bayCgqm3V20/S-1f52qecRI/AAAAAAAACKM/6LLl9DLpRDM/s1600-h/image%5B79%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_bayCgqm3V20/S-1f6aVzF2I/AAAAAAAACKQ/f4fXQX0sO5g/image_thumb%5B33%5D.png?imgmax=800" width="613" height="449" /></a> </p> <p> </p> <p>At the last screen, uncheck “Quickstart” and click “Done”.</p> <blockquote> <p><strong><u><em>EDIT Important note</em></u></strong> : I was a bit too optimistic to think it would allow JDev to run in a 64 bits mode. When I tried to start, it simply refuses to launch.</p> <p>Then, I’m really sorry, but you will still have to install a 32 bits JDK to be able to run it. WebLogic Server will run in a 64 bits mode though.</p> <p>To update the JVM location, edit the file <strong><em>JDEV_HOME</em>\jdeveloper\jdev\bin\jdev.conf</strong> and update the field <em><strong>SetJavaHome</strong></em></p> </blockquote> <p>Then update it to get the SOA module.</p> <p><a href="http://lh5.ggpht.com/_bayCgqm3V20/S_BojoXi_BI/AAAAAAAACNY/m9oRco9sIW8/s1600-h/image%5B3%5D.png"><img style="border-bottom: 0px; border-left: 0px; display: inline; border-top: 0px; border-right: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/_bayCgqm3V20/S_Bok-QUILI/AAAAAAAACNc/wIq_V0lWCxQ/image_thumb%5B1%5D.png?imgmax=800" width="397" height="419" /></a> </p> <p> </p> <p>Leave only “Oracle Fusion Middleware Products” checked.</p> <p> </p> <p><a href="http://lh3.ggpht.com/_bayCgqm3V20/S_Bol3NPsmI/AAAAAAAACNg/P5ndgpe-A10/s1600-h/image%5B13%5D.png"><img style="border-bottom: 0px; border-left: 0px; display: inline; border-top: 0px; border-right: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/_bayCgqm3V20/S_BomhgxgOI/AAAAAAAACNk/SDdBMzKFfcA/image_thumb%5B5%5D.png?imgmax=800" width="540" height="182" /></a> </p> <p> </p> <p>Type “<strong>soa</strong>” in the search field, and check the “<strong>Oracle SOA Composite Editor</strong>”</p> <p> </p> <p><a href="http://lh5.ggpht.com/_bayCgqm3V20/S_BooPJILiI/AAAAAAAACNo/P_Q7Y5vuk3w/s1600-h/image%5B14%5D.png"><img style="border-bottom: 0px; border-left: 0px; display: inline; border-top: 0px; border-right: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_bayCgqm3V20/S_BopOllqlI/AAAAAAAACNs/29HHtQrI_wI/image_thumb%5B6%5D.png?imgmax=800" width="632" height="148" /></a> </p> <p> </p> <p>Then the download starts.</p> <p> </p> <p><a href="http://lh4.ggpht.com/_bayCgqm3V20/S_BopYpi2II/AAAAAAAACNw/Z3eXqmMpjK0/s1600-h/image%5B15%5D.png"><img style="border-bottom: 0px; border-left: 0px; display: inline; border-top: 0px; border-right: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/_bayCgqm3V20/S_BoqUZSSMI/AAAAAAAACN0/XSJ7fzID-k0/image_thumb%5B7%5D.png?imgmax=800" width="425" height="148" /></a> </p> <p> </p> <p>Once the download is over, you are done with JDev.</p> <p> </p> <h3>2.3) WebLogic Server 64 bits </h3> <p> </p> <p>At that point, you may wonder why we are installing once again WebLogic Server, because as we saw it, it is bundled & normally installed with JDeveloper.</p> <p>This question is legitimate. The WLS shipped with JDev is not “strong” enough : I mean, it is only a light version, designed to work mainly with ADF.</p> <p>You will then not be able to use the SOA Suite on that version, that is why we must install the “real” one.</p> <p> </p> <p>Using the generic installer, just type : <strong>java –D64 –jar wls1033_generic.jar</strong></p> <p> </p> <p><a href="http://lh3.ggpht.com/_bayCgqm3V20/S-1f692m_OI/AAAAAAAACKU/shJlAohk_lI/s1600-h/image%5B110%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/_bayCgqm3V20/S-1f7Yk1XfI/AAAAAAAACKY/B7sd4DI-aqQ/image_thumb%5B50%5D.png?imgmax=800" width="497" height="91" /></a> </p> <p> </p> <p>As detailed in the JDev installation step, it is better if you do not mix up the WLS versions, so create another middleware home, such as :</p> <p> </p> <p><a href="http://lh3.ggpht.com/_bayCgqm3V20/S-1f77I2EDI/AAAAAAAACKc/P7QKnVNrhZo/s1600-h/image%5B132%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_bayCgqm3V20/S-1f8S_XqMI/AAAAAAAACKg/OKOiaZifbao/image_thumb%5B60%5D.png?imgmax=800" width="208" height="90" /></a> </p> <p> </p> <p>Then, just like for JDev, chose a proper JDK location.</p> <p> </p> <p><a href="http://lh4.ggpht.com/_bayCgqm3V20/S-1f86BbMlI/AAAAAAAACKk/_fjLCI9ySlw/s1600-h/image%5B122%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/_bayCgqm3V20/S-1f9QFh9RI/AAAAAAAACKo/eS4Lp152OTk/image_thumb%5B56%5D.png?imgmax=800" width="605" height="441" /></a> </p> <p> </p> <p>At the last screen, uncheck “Quickstart” and click “Done”.</p> <p> </p> <h3>2.4) Database : Oracle XE</h3> <p> </p> <p>Just install the product. Nothing hard here.</p> <p>Note that the principle of XE is to have only one instance (XE) but as many users (schemas) as you want.</p> <p>Chose a password for sys & system (it’s the same for both) and be sure to remember it. </p> <p>“manager” is a good development value as it is commonly used almost everywhere (even in production sometimes :p) : easy to remember.</p> <p>Once you’ve installed the database, you’ll have two services installed (OracleXE & Oracle TNS Listener),</p> <p>and an administration interface will be available on port 8080, at the address : <a href="http://localhost:8080/apex">http://localhost:8080/apex</a></p> <p> </p> <p><a href="http://lh3.ggpht.com/_bayCgqm3V20/S-1f9zZixMI/AAAAAAAACKs/NHGy1lEyyW4/s1600-h/image%5B77%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_bayCgqm3V20/S-1f-blvLwI/AAAAAAAACKw/5VMp6zLmBDQ/image_thumb%5B31%5D.png?imgmax=800" width="426" height="332" /></a> </p> <p> </p> <p>You can use it to administer some parts of your database (user creation and so on, but unfortunately, no tablespace management …)</p> <p> </p> <h3>Update the numbers of processes</h3> <p> </p> <p>If you run the RCU (described in next section), you will see that the test for user SOAINFRA fails.</p> <p>The reason comes from the number of processes which is, by default, not high enough.</p> <p>We then need to execute this SQL instruction to fix that :</p> <blockquote> <p><strong>alter system set processes=200 scope=spfile</strong></p> <p>(SCOPE = SPFILE / MEMORY / BOTH) (SPFile has to be binary) </p> </blockquote> <p>Oracle XE will not let you change the value of the parameter in memory, so you have to chose SPFILE and then restart your database.</p> <p>After the restart, to confirm that your change has been taken into account, execute this instruction :</p> <blockquote> <p><strong>select value from v$parameter where name = 'processes'</strong></p> </blockquote> <p>It should return the value you defined, that is to say, 200.</p> <p>Okay, now let’s create the schemas !</p> <p> </p> <h3>2.5) RCU (Repository Creation Utility)</h3> <p> </p> <p>There’s nothing to install from a software point of view : a wizard will just configure your database by creating the required users.</p> <p>Unzip the file “<strong>ofm_rcu_win_11.1.1.3.0_disk1_1of1.zip</strong>” in a folder I will now refer to as “UNZIP_FOLDER”.</p> <blockquote> <p><strong><u><em>Note</em></u></strong> : For the SOA Suite to work, you are not forced to keep the RCU after the job is done : it is a one-shot configuration</p> </blockquote> <p>Launch the file <strong>UNZIP_FOLDER\rcuHome\BIN\rcu.bat</strong></p> <p> </p> <p>Configure the access to your XE database :</p> <p> </p> <p><a href="http://lh6.ggpht.com/_bayCgqm3V20/S-1f_dSj8zI/AAAAAAAACK0/18qnPW7QYcE/s1600-h/image%5B80%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh3.ggpht.com/_bayCgqm3V20/S-1gAHhVqHI/AAAAAAAACK4/Y1NHumuIsFw/image_thumb%5B34%5D.png?imgmax=800" width="997" height="752" /></a> </p> <p> </p> <p>You may have some warning about the encoding : no big deal, click ok (maybe twice) and then you will have this screen </p> <p> </p> <p><a href="http://lh3.ggpht.com/_bayCgqm3V20/S-1gAnrriMI/AAAAAAAACK8/7blZRlv5fTQ/s1600-h/image%5B81%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_bayCgqm3V20/S-1gBf77LnI/AAAAAAAACLA/vJo8RuSya1A/image_thumb%5B35%5D.png?imgmax=800" width="539" height="223" /></a> </p> <p> </p> <p>Click “OK”. The RCU will now ask for a prefix. Usually, for a development environment, “DEV” fits :)</p> <p>So that is the prefix we are going to define here.</p> <p>Just select SOA & BPM Infrastructure and will check its dependency (MDS).</p> <p> </p> <p><a href="http://lh3.ggpht.com/_bayCgqm3V20/S-1gCb1c29I/AAAAAAAACLE/iUHAiKbr63s/s1600-h/image%5B82%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/_bayCgqm3V20/S-1gDr6EqqI/AAAAAAAACLI/g2Jx_nvteZs/image_thumb%5B36%5D.png?imgmax=800" width="675" height="356" /></a> </p> <p> </p> <p>Then go forward : the RCU will handle the tablespaces & users for you</p> <p> </p> <p><a href="http://lh5.ggpht.com/_bayCgqm3V20/S-1gEPZfnOI/AAAAAAAACLM/SxjsYIyXxpE/s1600-h/image%5B83%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_bayCgqm3V20/S-1gEjgmvZI/AAAAAAAACLQ/YzYnMk-wE5M/image_thumb%5B37%5D.png?imgmax=800" width="540" height="214" /></a> </p> <p> </p> <blockquote> <p><u><em><strong>Note</strong></em></u> : If you were ever forced to drop some schema, know that you may have to execute this procedure to be able to get rid of the SOAINFRA schema.</p> </blockquote> <blockquote> <p>BEGIN <br />  DBMS_AQADM.DROP_QUEUE_TABLE('EDN_OAOO_DELIVERY_TABLE', FORCE => TRUE); <br />  DBMS_AQADM.DROP_QUEUE_TABLE('EDN_EVENT_QUEUE_TABLE', FORCE => TRUE); <br />  DBMS_AQADM.DROP_QUEUE_TABLE('IP_QTAB', FORCE => TRUE); <br />END;</p> </blockquote> <p>A summary screen will show you the information you entered :</p> <p> </p> <p><a href="http://lh4.ggpht.com/_bayCgqm3V20/S-1gFr20gOI/AAAAAAAACLU/GNCcZZMF6h0/s1600-h/image%5B133%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/_bayCgqm3V20/S-1gGfVxd8I/AAAAAAAACLY/IgjMRmIKC4s/image_thumb%5B61%5D.png?imgmax=800" width="644" height="483" /></a> </p> <p> </p> <p>Click “Create”.</p> <p>Normally, everything should go painlessly and at the end, another summary screen will show up :</p> <p> </p> <p><a href="http://lh6.ggpht.com/_bayCgqm3V20/S-1gG5TtYII/AAAAAAAACLc/mY0Skur6_4w/s1600-h/image%5B85%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_bayCgqm3V20/S-1gHU8fGiI/AAAAAAAACLg/5BkpU4KnJ2A/image_thumb%5B39%5D.png?imgmax=800" width="710" height="109" /></a> </p> <p> </p> <p>Close the window and you’re done with the RCU.</p> <p> </p> <h3>2.6) SOA Suite PS1 (11.1.1.2)</h3> <p> </p> <p>So as I wrote it in the introduction, you’ll have to install the first patchset, then then second one.</p> <ol> <li>Unzip the file <strong>ofm_soa_generic_11.1.1.2.0_disk1_1of1</strong> </li> <li>Go in <strong>UNZIP_FOLDER\Disk1 and run setup.exe</strong> </li> <li>A DOS window will ask you to enter a JAVA_HOME </li> </ol> <p> </p> <p>Enter the path to the JDK in “Program Files”</p> <p> </p> <p><a href="http://lh4.ggpht.com/_bayCgqm3V20/S-1gH7ZbJfI/AAAAAAAACLk/2GxH8z-7cwM/s1600-h/image%5B62%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh3.ggpht.com/_bayCgqm3V20/S-1gIUawj7I/AAAAAAAACLo/P-R6Fbx88sg/image_thumb%5B22%5D.png?imgmax=800" width="1028" height="132" /></a> </p> <p> </p> <p>The DOS window will then vanish, and the Oracle Universal Installer will pop up :</p> <p> </p> <p><a href="http://lh4.ggpht.com/_bayCgqm3V20/S-1gJXeRJ6I/AAAAAAAACLs/1-hWK50ejzM/s1600-h/image%5B73%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_bayCgqm3V20/S-1gKIZ46cI/AAAAAAAACLw/wqajjcnFz2I/image_thumb%5B27%5D.png?imgmax=800" width="650" height="507" /></a> </p> <p> </p> <p>Chose the WebLogic Home in which you installed WebLogic Server 64 bits ([…]\<strong>SOASuite11gR1PS2\OFM</strong>)</p> <p>Make sure that the “Oracle Home” has the proper value.</p> <p> </p> <p><a href="http://lh3.ggpht.com/_bayCgqm3V20/S-1gKbp0MtI/AAAAAAAACL0/Der0_fP9Mgo/s1600-h/image%5B74%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_bayCgqm3V20/S-1gK8fl4hI/AAAAAAAACL4/z3QgsZz54SE/image_thumb%5B28%5D.png?imgmax=800" width="438" height="102" /></a> </p> <p> </p> <p>Then click install on the next screen.</p> <p> </p> <p><a href="http://lh5.ggpht.com/_bayCgqm3V20/S-1gL6XoiXI/AAAAAAAACL8/OBciCbp3nBY/s1600-h/image%5B75%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh5.ggpht.com/_bayCgqm3V20/S-1gMvdp4BI/AAAAAAAACMA/HLTNhftRfK4/image_thumb%5B29%5D.png?imgmax=800" width="653" height="509" /></a> </p> <p> </p> <p>You should normally have in your MIDDLEWARE_HOME the following directory structure :</p> <p> </p> <p><a href="http://lh6.ggpht.com/_bayCgqm3V20/S-1gNToH6FI/AAAAAAAACME/PIuk3plPCn0/s1600-h/image%5B145%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh5.ggpht.com/_bayCgqm3V20/S-1gN-1c7uI/AAAAAAAACMI/ezZ7RCnvvFg/image_thumb%5B67%5D.png?imgmax=800" width="187" height="349" /></a> </p> <p> </p> <h3>2.7) SOA Suite PS1 (11.1.1.3)</h3> <p> </p> <p>Now it’s done, we will upgrade our SOA Suite with the latest patchset.</p> <p>In a word, it’s just like the PS1 : you first have to unzip the file “<strong>ofm_soa_generic_11.1.1.3.0_disk1_1of1.zip</strong>” and then execute the setup.exe in the directory “<strong>Disk1</strong>”.</p> <p>You will be asked to enter the original directory in which you installed :</p> <ul> <li>your Oracle Middleware home (<strong>SOASuite11gR1PS2\OFM</strong>) </li> <li>your Oracle SOA Suite directory (<strong>Oracle_SOA</strong>) </li> </ul> <p> </p> <p><a href="http://lh5.ggpht.com/_bayCgqm3V20/S-1gOqw5H3I/AAAAAAAACMM/3ALoqqNdMDs/s1600-h/image%5B101%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh5.ggpht.com/_bayCgqm3V20/S-1gPWNS4gI/AAAAAAAACMQ/JL9wpfKSSuI/image_thumb%5B45%5D.png?imgmax=800" width="665" height="537" /></a> </p> <p> </p> <p>And here you go again :)</p> <p> </p> <p><a href="http://lh6.ggpht.com/_bayCgqm3V20/S-1gQv6aY2I/AAAAAAAACMU/kJgggLcnvPs/s1600-h/image%5B106%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh4.ggpht.com/_bayCgqm3V20/S-1gRaQlfFI/AAAAAAAACMY/Nf6rc_6YIjk/image_thumb%5B48%5D.png?imgmax=800" width="664" height="534" /></a> </p> <p> </p> <p>But this time note the versions that are displayed : they’re all in 11.1.1.3.0.</p> <p> </p> <p>Okay now’s everything is configured.</p> <p> </p> <h2>3) Checking</h2> <p> </p> <h3>3.1) 64 bits installation</h3> <p> </p> <p>This part is optional : just to check that WLS was correctly installed in 64 bits, you can create a sample domain, start your server and check the following :</p> <blockquote> <p>starting weblogic with Java version: <br />java version "1.6.0_20-ea" <br />Java(TM) SE Runtime Environment (build 1.6.0_20-ea-b02) <br />Java HotSpot(TM) <font color="#0000ff">64-Bit Server</font> VM (build 17.0-b12, mixed mode)</p> </blockquote> <p>And : </p> <blockquote> <p><14 mai 2010 01 h 20 CEST> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM)<font color="#0080ff"> </font><font color="#0000ff">64-Bit Server</font> VM Version 17.0-b12 from Sun Microsystems Inc.></p> </blockquote> <p>You should also see some references to the native 64 libraries in the PATH :</p> <blockquote> <p>[…] \<font color="#0000ff">server\native\win\x64</font> […]</p> </blockquote> <p> </p> <p></p> <h3>3.2) SOA Suite PS2</h3> <p> </p> <p>Check that all the templates are available.</p> <p> </p> <p><a href="http://lh3.ggpht.com/_bayCgqm3V20/S_Bg5Ajm2cI/AAAAAAAACOk/Mukqza54Csw/s1600-h/image146.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_bayCgqm3V20/S-1gTPXosOI/AAAAAAAACOo/bClECgNuD2U/image_thumb68.png?imgmax=800" width="793" height="568" /></a> </p> <p> </p> <p> </p> <h2>4) Conclusion</h2> <p> </p> <p>It is an official recommendation to install JDev in a directory and the SOA Suite in another, but I made a test in which I did override the WLS installed by JDev and it seemed to work.</p> <p>Some stuff I found weird : in the “<strong>SOASuite11gR1PS2\OFM\wlserver_10.3\common\bin</strong>” directory when I run <strong>config.exe</strong>, I don’t have the same result as if I had run <strong>config.cmd</strong> …</p> <p> </p> <p><u></u></p> <p><u><strong></strong></u></p> <p><u><strong>config.exe </strong>gives me :</u></p> <p><u></u></p> <p> </p> <p><u></u></p> <p><u></u></p> <p><u></u></p> <p><u></u></p> <p><a href="http://lh3.ggpht.com/_bayCgqm3V20/S_Bg5Ajm2cI/AAAAAAAACOs/jteUCKLjl1w/s1600-h/image141.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_bayCgqm3V20/S-1gTPXosOI/AAAAAAAACOw/lTLofWfwhNE/image_thumb65.png?imgmax=800" width="793" height="568" /></a> </p> <p> </p> <p>Whereas <strong>config.cmd</strong> gives me :</p> <p> </p> <p><a href="http://lh6.ggpht.com/_bayCgqm3V20/S-1gUhriiyI/AAAAAAAACMs/lzbUkqWrbrA/s1600-h/image%5B140%5D.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://lh5.ggpht.com/_bayCgqm3V20/S-1gVcReAiI/AAAAAAAACMw/16uu6j0pLDs/image_thumb%5B64%5D.png?imgmax=800" width="788" height="563" /></a> </p> <p></p> <p> </p> <p>Got to check whether this a common behavior or not ! :)</p> <p> </p> <div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:0767317B-992E-4b12-91E0-4F059A8CECA8:3e25b0eb-bf7d-4c2d-aac9-1c7cae33aa75" class="wlWriterEditableSmartContent">Mots clés Technorati : <a href="http://technorati.com/tags/windows+seven" rel="tag">windows seven</a>,<a href="http://technorati.com/tags/soa+suite+11gR1PS2" rel="tag">soa suite 11gR1PS2</a>,<a href="http://technorati.com/tags/rcu" rel="tag">rcu</a>,<a href="http://technorati.com/tags/weblogic+server" rel="tag">weblogic server</a>,<a href="http://technorati.com/tags/jdeveloper" rel="tag">jdeveloper</a>,<a href="http://technorati.com/tags/11.1.1.3" rel="tag">11.1.1.3</a></div> Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com1tag:blogger.com,1999:blog-2592381182145606344.post-80141130686358807152010-05-09T00:44:00.011+02:002010-05-09T10:47:56.106+02:00Shared Libraries versus Optional PackagesStarting with WebLogic Server 9, BEA introduced an interesting concept aiming at sharing more easily libraries between EARs. <br />
From a performance point of view and memory imprint, there’s absolutely no change : the library will still be loaded in each and every EAR classloader. <br />
But from a packaging point of view, it allows you to reduce significantly the size of your archives, by externalizing the common libraries. <br />
Moreover, when you have to upgrade one of your lib, it’s easier to do, thanks to centralization. <br />
<br />
In my case, I’m currently working on a key SOA project on which a huge work had been done on isolating services from a business point of view. <br />
But technically, all the services were packaged in the same EAR, which is a serious problem for the high availability we are supposed to provide. <br />
(in case of a new version, it would lead to an undeployment of the whole archive : <a href="http://download.oracle.com/docs/cd/E13222_01/wls/docs100/deployment/redeploy.html#wp1020276" target="_blank">production redeployment</a> is not an option since the archive is pretty bulky) <br />
So I had to explode the EAR in several WARs and adopt a strategy for the library sharing. <br />
<br />
<h2>Deploying libraries</h2><br />
First of all, whether you'll use the “shared library” or the “optional package” mechanism, your libraries (JARs in our case but can be WAR, RAR …) <br />
must respect a certain definition in their manifests, such as : <br />
<blockquote><pre>Extension-Name: fr.mbutton.blog
Specification-Vendor: Tengah Consulting
Specification-Version: 1.0
Implementation-Vendor-Id: fr.tengah-consulting
Implementation-Vendor: Tengah Consulting
Implementation-Version: 1.1</pre></blockquote>This example has been inspired by the <a href="http://java.sun.com/j2se/1.4.2/docs/guide/extensions/versioning.html" target="_blank">example found on sun.com</a>.<br />
Based on my experience, the only lines that really matter are the following :<br />
<blockquote><pre>Extension-Name: fr.mbutton.blog
Specification-Version: 1.0
Implementation-Version: 1.1</pre></blockquote>WebLogic Server will use the extension name to determine the name it will use for the library you’re deploying.<br />
If none, then the name of the file itself will be used to qualify deployed archive name, as explained on <a href="http://download.oracle.com/docs/cd/E12840_01/wls/docs103/programming/libraries.html#wp1067450" target="_blank">Oracle documentation</a>.<br />
<br />
In my case, I had to list all the libraries used in our application and see whether they were eligible for being deployed as a shared library.<br />
And I realized that lots of libraries you can find on the market are / were not thought to be deployed as optional package / shared libraries.<br />
<br />
So far, the only libraries that were clearly designed following this standard are the libs from Apache.<br />
Regarding the specification & implementation versions, you may specify those pieces of information <a href="http://download.oracle.com/docs/cd/E12840_01/wls/docs103/deployment/deploy.html#wp1023538" target="_blank">while deploying</a>,<br />
but it is a good practice to have them described in the library manifest.<br />
<br />
Moreover as described in the official Oracle documentation, you should be careful while using this modus operandi :<br />
<blockquote>“If both the manifest file and the <code>weblogic.Deployer</code> command line specify version information, and the values do not match, the deployment fails. <br />
If you initially registered a library without providing a specification or implementation version,<br />
you must undeploy the library before registering a newer version and specifying version string information.<br />
<br />
You can register multiple versions of a library or package, but each version must use the same version strings. <br />
For example, you cannot register a library having only a specification version,<br />
and then register a new version of the library having both a specification and an implementation version.”</blockquote><span class="Apple-style-span" style="font-size: 24px; font-weight: bold;">Referencing Shared Libraries</span><br />
<br />
Now our libraries are correctly defined and deployed the use as shared libraries is really simple. <br />
According to the documentation about the weblogic deployment descriptors (<a href="http://download.oracle.com/docs/cd/E12840_01/wls/docs103/webapp/weblogic_xml.html#wp1060332" target="_blank">weblogic.xml</a> & <a href="http://download.oracle.com/docs/cd/E11035_01/wls100/programming/app_xml.html#wp1076531" target="_blank">weblogic-application.xml</a>), the way to reference a library is the following :<br />
<blockquote><pre><library-ref>
<library-name>fr.mbutton.blog</library-name>
<specification-version>1.0</specification-version>
<implementation-version>1.1</implementation-version>
<exact-match>false</exact-match>
</library-ref></pre></blockquote>But one major problem is that you cannot reference all the types, just as <a href="http://download.oracle.com/docs/cd/E12840_01/wls/docs103/programming/libraries.html#wp1071062" target="_blank">it is clearly said in the official documentation</a> :<br />
<blockquote><em>“You cannot reference any other type of shared Java EE library (EJB, Enterprise application, or plain JAR file)</em> <br />
<em>from the <code>weblogic.xml</code> deployment descriptor file of a Web Application.”</em></blockquote>In other words, it is impossible to reference JAR files in a web application (WAR).<br />
Hummmm … annoying since it’s exactly what I’m trying to do …<br />
Then my friend & colleague <a href="http://fr.linkedin.com/pub/romain-buquet/2/5b2/702" target="_blank">Romain</a> told me about the optional packages.<br />
<span class="Apple-style-span" style="font-size: 24px; font-weight: bold;"><br />
</span><br />
<span class="Apple-style-span" style="font-size: 24px; font-weight: bold;">Referencing Optional Packages</span><br />
<br />
With optional packages, that annoying limitation is no more. <br />
If you followed the pieces of advice regarding the correct deployment of libraries for a use as shared libraries,<br />
that’s perfect because you won’t have anything to change to switch to optional packages.<br />
<br />
The only thing that changes is the way you will refer to them. <br />
Using optional packages, you simply have to update the manifest for your WAR such as :<br />
<blockquote><pre>Extension-List: blog
blog-Extension-Name: fr.mbutton.blog
blog-Specification-Version: 1.0
blog-Implementation-Version: 1.1
</pre></blockquote>And suppose we have another library :<br />
<blockquote><pre><library-ref>
<library-name>org.springframework</library-name>
<specification-version>3.0</specification-version>
<implementation-version>3.9beta</implementation-version>
<exact-match>false</exact-match>
</library-ref>
</pre></blockquote>This time, the definition in the manifest would be :<br />
<blockquote><pre>Extension-List: blog spring
blog-Extension-Name: fr.mbutton.blog
spring-Extension-Name: org.springframework
blog-Specification-Version: 1.0
spring-Implementation-Version: 3.9beta
</pre></blockquote>Once again, I’m not inventing this : you can read it <a href="http://download.oracle.com/docs/cd/E12840_01/wls/docs103/programming/libraries.html#wp1064645" target="_blank">here</a>.<br />
<br />
<span class="Apple-style-span" style="font-size: 24px; font-weight: bold;">Finally …</span><br />
<br />
Okay here we are, time for a conclusion !<br />
<br />
Shared libraries :<br />
<ul><li>are cool but do not allow all type of referencing (-) </li>
<li>are WebLogic Server specific (-) </li>
<li>are XML based (+?) </li>
</ul>Whereas optional packages :<br />
<ul><li>are standard … (+) </li>
<li>… but completely based on manifests : almost worst than Cobol ! (-) </li>
<li>allow a lot more possibilities (+) </li>
</ul>Hope this post helped to enlighten a bit your path.<br />
<br />
Resources :<br />
<ul><li>Oracle documentation <a href="http://download.oracle.com/docs/cd/E12840_01/wls/docs103/programming/libraries.html" target="_blank">here</a> & <a href="http://java.sun.com/j2se/1.4.2/docs/guide/extensions/versioning.html" target="_blank">here</a> (Sun) </li>
<li><a href="http://blogs.oracle.com/jamesbayer/2009/12/weblogic_server_shared_librari_1.html" target="_blank">James’ blog</a> </li>
</ul><br />
Mots clés Technorati : <a href="http://technorati.com/tags/weblogic+server" rel="tag">weblogic server</a>,<a href="http://technorati.com/tags/optional+packages" rel="tag">optional packages</a>,<a href="http://technorati.com/tags/shared+libraries" rel="tag">shared libraries</a>,<a href="http://technorati.com/tags/manifest" rel="tag">manifest</a>,<a href="http://technorati.com/tags/sharing+jar+across+wars" rel="tag">sharing jar across wars</a>Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com4tag:blogger.com,1999:blog-2592381182145606344.post-61841908338507944512010-05-06T22:56:00.001+02:002010-05-06T22:56:46.496+02:00About some of the tools I am using in my everyday life<p></p> <p></p> <p>As a consultant, I'm always on the run, switching from clients to others, resolving (or at least, trying to) problems.</p> <p>(even if it's not been the case lately, since I'm working for a very large client who needs a lot of attention :p)</p> <p></p> <p>Anyway, I could not do my job without my tools. And as they are very very useful to me, I wanted to share them with you !</p> <p>Sorry <a href="http://fr.linkedin.com/in/adudons" target="_blank">Aurélien</a> for taking your blogpost idea, but I’ve been waiting about a year for your blog to be created, and I’m still waiting ;)</p> <p> </p> <h2>Network (tools to grab some remote files, setup a FTP server ...)</h2> <p> </p> <p>First of all, <a href="http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html" target="_blank">Putty</a>. Very famous SSH client I find very useful BUT even more when used within <a href="http://puttycm.free.fr/cms/" target="_blank">Putty Connection Manager</a>.</p> <p>The latter is a layer above putty that allows you to centralize all your putty windows, enter some extra-commands, such as auto-login etc.</p> <p>I'm using it everyday !</p> <p> </p> <p>Then, when you want to browse a remote server windows-style, you may use a FTP client (of course, </p> <p>only if the server you want to reach is running a FTP server). </p> <p>In that case, the most famous FTP client is probably <a href="http://filezilla-project.org/" target="_blank">FileZilla</a>. If you want to serve your files through FTP, </p> <p>you may use <a href="http://filezilla-project.org/" target="_blank">FileZilla Server</a> or <a href="http://www.serv-u.com/" target="_blank">Serv-U FTP</a>, which is pretty good too (but not free).</p> <p>Else, I use <a href="http://winscp.net/eng/download.php" target="_blank">WinSCP</a>, SFTP / FTP but also SCP client.</p> <p> </p> <p>When you really want to know what’s going on& what’s exchanged between two servers, you may use <a href="http://www.fiddler2.com/fiddler2/" target="_blank">Fiddler</a> to watch HTTP traffic. </p> <p>And if you want to “go deep”, use <a href="http://www.wireshark.org/" target="_blank">WireShark</a>.</p> <p> </p> <p>If you’re looking for a cool & free LDAP client, then forget about <a href="http://www.ldapbrowser.com/" target="_blank">Softerra</a> & <a href="http://jxplorer.org/" target="_blank">JXplorer</a> ! </p> <p>Try <a href="http://directory.apache.org/studio/" target="_blank">Apache Directory Studio</a> and you’ll see what I mean …</p> <p> </p> <h2>Database</h2> <p> </p> <p>Well, all depends on the database manufacturer you're relying on. Personally, it's Oracle (no kidding). </p> <p>Then as my personal database, I'm using <a href="http://www.oracle.com/technology/products/database/xe/index.html" target="_blank">Oracle XE</a>, and as a client, I'm using Oracle <a href="http://www.oracle.com/technology/products/database/sql_developer/index.html" target="_blank">SQL Developer</a>, which is pretty good.</p> <p> </p> <h2>Java / Java EE</h2> <p> </p> <p></p> <p>I will not start a war by saying that my IDE is the best :)</p> <p>I’m using <a href="http://www.oracle.com/technology/products/workshop/index.html">Workshop</a> (Eclipse with a BEA flavor) but it’s more a pain in the … than everything else ! </p> <p>I remember enjoying <a href="http://www.jetbrains.com/idea/">IntelliJ</a> Idea but it was a long time ago. <a href="http://netbeans.org/">NetBeans</a> has a pretty good reputation !</p> <p>I have been playing with <a href="http://www.oracle.com/technology/products/jdev/index.html">JDeveloper</a> and even if it took me a while to get used to it, I can say it’s not bad, rather good in fact.</p> <p>Finally, I would go for Eclipse without the bunch of annoying plugins brought by Workshop.</p> <p></p> <p> </p> <p>When I’m stuck on a common error such as “NoClassDefFoundError”, my life saver is “<a href="http://sourceforge.net/projects/jarfinder/">Jar Finder</a>”. </p> <p>You may find some other ways to do so, such as websites, Eclipse plugins etc,</p> <p>but I can assure you that there is nothing better than this tool ! </p> <p>Quick to start (no workspace to load …, can use it offline) and pretty efficient : the must-have.</p> <p> </p> <p>And sometimes, when I want to take a closer to look to an implementation, I use <a href="http://java.decompiler.free.fr/">JD-*</a> (Java Decompiler) which is an excellent, </p> <p>I would even say the best tool to assist you in decompiling Java classes.</p> (and the fact it’s been written by a <a href="http://fr.linkedin.com/pub/emmanuel-dupuy/2/8a9/580">former colleague</a> has nothing to do with it ! :D) <p></p> <p> </p> <p>As everyone using webservices, I’m using <a href="http://www.soapui.org/">SOAPUi</a> (which allows you in its latest version to do some stress test too !)</p> <p> </p> <p>And when I have to perform some stress tests, I’m using <a href="http://jakarta.apache.org/jmeter/">JMeter</a> (I tried once <a href="http://www.opensta.org/">OpenSTA</a> but was more at ease with JMeter).</p> <p> </p> <p>Dealing with JMS, I’m also using <a href="http://www.hermesjms.com/confluence/display/HJMS/Home">HermesJMS</a> wich is good (not always providing the best stability but cool even though).</p> <p>My former colleague James blogged about it and its <a href="http://blogs.oracle.com/jamesbayer/2008/01/hermes_jms_open_source_jms_con.html">integration with WebLogic Server</a>.</p> <p> </p> <p>When I want to analyse some thread dumps, I’m using <a href="http://yusuke.homeip.net/samurai/en/index.html">Samurai</a>.</p> <p> <br /></p> <h2>Knowledge sharing (in other words blogging !)</h2> <p> </p> <p>To me, online writing is not a good solution. You may lose your content, you can't write anywhere and so on.</p> <p>As an offline writer, I'm using <a href="http://download.live.com/writer" target="_blank">Windows Live Writer</a>, which is pretty good. It supports a lots of blog providers : </p> <p>I started with it when I first blogged on Dev2Dev on WordPress, then I switched to blogspot and WLW switched without any trouble.</p> <p> </p> <p>To shoot and capture screens, I use <a href=" http://www.gadwin.com/printscreen/">Gadwin</a> which can be used in a free edition and it’s excellent ! </p> <p> </p> <h2>Other</h2> <p> </p> <p>Who hasn’t faced one day an annoying popup saying there was not enough space to copy a very important file ?</p> <p>When such a situation happens, you can spend a lot of time looking for some files & folders to delete or you can use <a href="http://windirstat.info/">WinDirStat</a>.</p> <p>It’s a freeware which will allow you to find in a snap <a href="http://img.clubic.com/photo/01892504.jpg" target="_blank">the biggest files on your harddrive</a>.</p> <p> </p> <p>I’m also currently playing with Sun’s (Oracle’s ?) <a href="http://www.virtualbox.org/">VirtualBox</a> to virtualize a whole development environment </p> <p>and it’s hard not be seduced by this wonderful soft. If you try it, you will adopt it :)</p> <p> </p> <p>To replace the original Windows file copying system, I’ve chosen <a href="http://supercopier.sfxteam.org/">SuperCopier</a>. Far better !</p> <p> </p> <p>And last but not least, the super text editor : Notepad++ ! </p> <p>If you’re working with XML : read that <a href="http://www.twu.ca/divisions/technology/sst/orion/blog/tidy-notepad-and-xml.html">useful post</a> about enabling XML indent in Notepad++.</p> <p> </p> <p>Well that’s all for now and if you have an extraordinary tool you like to share, don’t hesitate to post a comment ! :)</p> <p> </p> <div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:0767317B-992E-4b12-91E0-4F059A8CECA8:f332db54-629e-44f5-bb82-7a8e469b4e32" class="wlWriterEditableSmartContent">Mots clés Technorati : <a href="http://technorati.com/tags/j2EE+tools" rel="tag">j2EE tools</a>,<a href="http://technorati.com/tags/java+ee+tools" rel="tag">java ee tools</a>,<a href="http://technorati.com/tags/virtualbox+weblogic" rel="tag">virtualbox weblogic</a>,<a href="http://technorati.com/tags/hermesjms" rel="tag">hermesjms</a>,<a href="http://technorati.com/tags/notepad%2b%2b" rel="tag">notepad++</a>,<a href="http://technorati.com/tags/apache+virtual+directory+studio" rel="tag">apache virtual directory studio</a>,<a href="http://technorati.com/tags/wireshark" rel="tag">wireshark</a>,<a href="http://technorati.com/tags/fiddler" rel="tag">fiddler</a>,<a href="http://technorati.com/tags/windirstat" rel="tag">windirstat</a>,<a href="http://technorati.com/tags/putty" rel="tag">putty</a>,<a href="http://technorati.com/tags/putty+connection+manager" rel="tag">putty connection manager</a>,<a href="http://technorati.com/tags/sql+developer" rel="tag">sql developer</a>,<a href="http://technorati.com/tags/oracle+xe" rel="tag">oracle xe</a></div> Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com0tag:blogger.com,1999:blog-2592381182145606344.post-53457342736853294742009-10-24T00:18:00.001+02:002009-11-07T16:12:58.924+01:00Troubleshooting two common JMS exceptions<p>I've been working with a particular architecture lately.</p> <p>It's composed of an application server (WebLogic Server 10), hosting webservices, an ESB (ALSB 2.5) and an EAI / BPM (WLI 8.1 SP5).</p> <p> </p> <p><a href="http://lh3.ggpht.com/_bayCgqm3V20/SuIrruiaSUI/AAAAAAAACC8/1jatmswRjiE/s1600-h/image%5B3%5D.png"><img style="border-right-width: 0px; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" border="0" alt="image" src="http://lh4.ggpht.com/_bayCgqm3V20/SuIrsGXB2BI/AAAAAAAACDA/pnyHZqYdoJM/image_thumb%5B1%5D.png?imgmax=800" width="544" height="602"></a> </p> <p> </p> <p> </p> <p>The point we are going to focus on is the JMS communication between WLI 8.1 SP5 & WLS 10.</p> <p>WLI is hosting a JMS server and WLS 10, is hosting a foreign JMS Server, representing the WLI JMS Server.</p> <h2> </h2> <h2>First exception </h2> <p> </p> <p>From time to time, we happened to have this exception showing up in the logs :</p> <p> </p> <blockquote> <p><font color="#ff0000"><23 oct. 2009 21 h 50 CEST> <Error> <JTA> <BEA-114089> <User [<anonymous>] is not authorized to invoke ackPrepare on a Coordinator.><br><23 oct. 2009 21 h 54 CEST> <Error> <JTA> <BEA-110495> <User [<anonymous>] is not authorized to invoke AckRollback on a Coordinator.></font></p></blockquote> <p> <p>These particular errors were happening when WLS was sending a JMS message towards WLI <p>(which was <em>pending</em> since the transaction wasn't commited nor rollbacked yet) and then tried to commit the transaction. <p>Then we saw the first message (BEA-114089). WLS, seeing that it couldn't commit, tried to rollback but it still wasn't allowed to do such a thing. <p>If we take a look at the official documentation : <p> <blockquote> <p><strong>BEA-114089</strong> <p><strong><em>Error: </em></strong>User [<em>principal</em>] is not authorized to invoke ackPrepare on a Coordinator. <p><strong>Description</strong> <p><font color="#ff0000">The user with identity [<em>principal</em>] should not be attempting to directly invoke internal methods on the Coordinator object. This might subvert transactional integrity. </font> <p><font color="#ff0000">Permission was denied, and the attempt was ignored.</font> <p><strong>Cause</strong> <p>This might be a symptom of a potential security attack, or alternately, <font color="#0000ff">a problem with the "system" security identity of the server</font>. <p><strong>Action</strong> <p>Verify that the client software is not directly invoking an internal WebLogic system object.</p></blockquote> <p> <p>And <p> <blockquote> <p><strong>BEA-110495</strong> <p><strong><em>Error: </em></strong>User [<em>principal</em>] is not authorized to invoke AckRollback on a Coordinator. <p><strong>Description</strong> <p><font color="#ff0000">The user with identity [<em>principal</em>] should not be attempting to directly invoke internal methods on the Coordinator object. This might subvert transactional integrity. </font> <p><font color="#ff0000">Permission was denied, and the attempt was ignored.</font> <p><strong>Cause</strong> <p>This might be a symptom of a potential security attack, or alternately, <font color="#0000ff">a problem with the "system" security identity of the server</font>. <p><strong>Action</strong> <p>Verify that client software is not directly invoking an internal WebLogic system object.</p></blockquote> <p> <p>We see that at some point, these explanations are not very useful. <p>I then made a sandbox to reproduce this problem. <p>I played with cross-domain security and security interoperability mode. <p>Before playing with these two options, I strongly recommend you to read this official documentation : <ul> <li><a href="http://download.oracle.com/docs/cd/E12840_01/wls/docs103/jms/trans.html#wp1039006">http://download.oracle.com/docs/cd/E12840_01/wls/docs103/jms/trans.html#wp1039006</a> <li><a href="http://download.oracle.com/docs/cd/E12840_01/wls/docs103/jta/trxcon.html#interop">http://download.oracle.com/docs/cd/E12840_01/wls/docs103/jta/trxcon.html#interop</a></li></ul> <p> </p> <p><strong>Finally, I came to the conclusion that these errors only happened when I had cross-domain security enabled (with identical credentials on both domains)</strong></p> <p><strong>and that the security interoperabilty mode was set to "<em>Performance</em>" on WLS 10. (by default, this is the mode defined on WLS 8.1).</strong></p> <p>In spite of the fact that official documentation says "<em>Every participating server must set the <code>Security Interoperability Mode</code> parameter to the same value</em>",</p> <p>in my case, it worked with <em>Performance</em> on WLS 8.1 (WLI) and <em>Compatibility</em> on WLS 10 ...</p> <p> </p> <h2>Second Exception</h2> <p> </p> <p>During my tests I also encountered an annoying error :</p> <p> </p> <blockquote> <p><WSEE:13><br>Error invoking fr.edf.test.transaction.SampleWebService (POJO): weblogic.jms.common.JMSException<ComponentHandler.handleRequest:115><br><WSEE:13><font color="#ff0000">weblogic.jms.common.JMSException: Connection not found</font><ComponentHandler.handleRequest:115><br><font color="#ff0000">weblogic.jms.common.JMSException: Connection not found</font><br> at weblogic.jms.dispatcher.DispatcherAdapter.convertToJMSExceptionAndThrow(DispatcherAdapter.java:110)<br> at weblogic.jms.dispatcher.DispatcherAdapter.dispatchSync(DispatcherAdapter.java:45)<br> at weblogic.jms.client.JMSConnection.setupJMSSession(JMSConnection.java:499)<br> at weblogic.jms.client.JMSConnection.createSessionInternal(JMSConnection.java:467)<br> at weblogic.jms.client.JMSConnection.createQueueSession(JMSConnection.java:432)</p></blockquote> <p> </p> <p>I had to google some time to figure out that this error came from the fact that both my admin servers were named "admin".</p> <p>Seen in the official doc :</p> <p> </p> <blockquote> <p>Please note the following limitations for inter-domain transactions: <ul> <li><a name="wp1042693"></a>The domains and all participating resources must have unique names. That is, you cannot have a JDBC data source, a server, or a domain with the same name as an object in another domain or the domain itself.</li></ul></blockquote> <p> </p> <p>Well, that's it for now. I'm sure there are lots of potential other errors, but I hope that, at least, these two explanations may help some fellows out.</p> <p> </p> <div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:0767317B-992E-4b12-91E0-4F059A8CECA8:699dd2d3-986f-4b94-ab09-1d0e6ba4ea98" class="wlWriterSmartContent">Mots clés Technorati : <a href="http://technorati.com/tags/jms" rel="tag">jms</a>,<a href="http://technorati.com/tags/jta" rel="tag">jta</a>,<a href="http://technorati.com/tags/transaction" rel="tag">transaction</a>,<a href="http://technorati.com/tags/wls%208.1" rel="tag">wls 8.1</a>,<a href="http://technorati.com/tags/wls%2010" rel="tag">wls 10</a>,<a href="http://technorati.com/tags/BEA-110495" rel="tag">BEA-110495</a>,<a href="http://technorati.com/tags/BEA-114089" rel="tag">BEA-114089</a>,<a href="http://technorati.com/tags/JMSException" rel="tag">JMSException</a>,<a href="http://technorati.com/tags/Security%20Interoperability" rel="tag">Security Interoperability</a>,<a href="http://technorati.com/tags/cross-domain%20security" rel="tag">cross-domain security</a></div> Maxence Buttonhttp://www.blogger.com/profile/03432797928549149364noreply@blogger.com2