How to create a distributed WebLogic domain ?

Due to a lot of questions about that, I'll blog about the methodology to create

a simple domain, with a cluster distributed on several machines.

 

Architecture Schema

Usually, a best practice is to keep the admin server away from managed servers.

Because, if a strong load happens and overloads a managed server, it will consume all the resources available

on the machine, including those needed for the admin server to be safe & running.

 

Then consider we have three machines, one admin server and four managed servers :

 

 

Install the binaries

This part is the easiest one. On each machine, you'll have to install the binaries.

I strongly recommend that you use the silent mode, which is faster and requires no manual intervention.

http://e-docs.bea.com/common/docs100/install/silent.html 

Another advice : it may sound obvious, but you have to use the same WLS version on every machine ...

 

Create the domain

You've got two possibilities for a domain creation :

Using the Config Wizard

This tool allows you to simply create a domain, by answering the asked questions, through a GUI or a command line.

It's based on the default WebLogic template, which is almost empty :)

(you may later base your domain on your own template, but that's another topic).

 

 

 

Using WLST

Using WLST, the concept is the same, you will base your domain on the default WLS template and

then, you'll script the instructions to create your domain.

 

Note : you may also use WLST with WLS 7 & 8

Unfortunately, this tool was hosted on projects Dev2Dev, which is not available anymore.

If you're interested in getting it, I may send it to you through email (about 2 539k).

 

Populate the domain

When you create a domain, as explained above, it will automatically create an administration server but

nothing else, unless you specify it. You can do so, either during the creation process, by chosing to

customize instead of selecting default, or by creating a light domain, launching  the admin server,

and then creating all the wanted resources.

No matter what is the solution you've taken, you'll need to configure at least two kind of resources :

1. Machines (which only are a logical representation of your physical machines)
2. Managed Servers, which represent the JVM hosting your business logic.

It is mandatory that you define machines if you're willing to use the NodeManager. Else, even if you

don't want to use it, it's a good practice to define your machines. Will help you, when administrating

your domain, to see where your managed servers are hosted.

Then, once you've done those two steps, you may proceed to optional next steps (configuring datasources ...)

 

Distribute your domain

Now your domain is full and complete, we need to duplicate it on other machines.

Pack & Unpack commands

These commands were designed to package a domain and unpackage it on another machine.

You could do the same by using a zip or a tar.gz, but using these commands will ensure you that

you have taken only the necessary files for your domain to work (no log files and unnecessary stuff).

http://download.oracle.com/docs/cd/E12840_01/common/docs103/pack/commands.html

 

Pack / Unpack (WLST version)

You can also do that task by using WLST.

Here is the pack script (taken from the WLST project page on dev2dev) :

 

# pack.py: convert from pack command to wlst script

# This script shows how to convert form the pack command to a wlst script.
# Note that the domain and template settings must be single-quoted.

# Read in the domain that you want to templatize
readDomain(<domain>)

# Create a template from the domain
writeTemplate(<template>) 

closeDomain()

 

And here comes the unpack script : 

 

# unpack.py: convert from unpack command to wlst script

# This script shows how to convert from the unpack command to a wlst script.
# Note that the domain and template values, and the options to setOption, must be single-quoted

# Specify the template that you want to use
readTemplate('c:\wls9\user_templates\wlst_wls_template.jar')

# If you specified the -username and -password option in the unpack command,
# Specify them here. Otherwise, delete these lines`
# Note that the domain_name field here is just the name of the domain, not the full path as specified in writeDomain below
cd ('/Security/<domain-name>')
create (<user_name>,'User')
cd ('User/<user_name>')
set ('Password',<password>)

# analogous to unpack -java_home
setOption('JavaHome',<java_home>)

# analogous to unpack -server_start_mode
setOption('ServerStartMode',<server_start_mode>)

# analogous to unpack -app_dir
setOption('AppDir',<app_dir>)

# write the domain
writeDomain(<domain>)

closeTemplate() 

 

By the way, I was talking about the fact you can zip your domain and install it elsewhere : that possibility should be considered as an

excellent way of saving snapshots of your domain (for instance, before installing a new resource ...).

If something should go wrong, you would simply have to delete your domain and unzip your backup to start as it was before.

 

NodeManager (optional)

The NodeManager is a tool that allows you several things, but the most important that come to my mind are :

• => to start servers from the console
• => to automatically restart a failed managed server
• => to migrate pinned services (JTA / JMS)

But some cilents don't want to use it because they prefer to use scripts. If this is your case, well, you're done.

Just start your managed servers on each machine and your cluster is up and running.

Else, read carefully the following lines !

For a domain to be managed by the NodeManager, you will have to enroll it first.

That means that you've got to help the NodeManager to be aware that a new domain was created.

(this step is NOT necessary when you're using the config wizard)

To see if the different nodemanagers are reachable, check the machines.

In case of a problem, you may see :

 

 

Of course, ideally, the status should be "Running".

 

 

Start your servers

 

If you chose to use the NodeManager, you only have to go on the administration console, and start your cluster.

 

 

Or you may start your managed servers using the start scripts. But be aware that with this solution, since your domain

has been duplicated on each machine, you could start all the servers on each machine. It's up to you to start only the relevant

servers on each machine.

 

 

Mots clés Technorati : weblogic domain,cluster,distributed,machine,nodemanager,nmenroll,pack,unpack,wlst

SOA Suite 11g : first quick look

Working @ Oracle, it gives me some interesting advantages, like the ability to play with the 11th version of

the Oracle SOA Suite, before its release to the market.

Here's a little summary of what I've had the chance to play with.

But, be aware that what I'm exposing right here is NOT representating the final version.

It's only a beta but as far as I can tell, our guys have done a good job !

 

What's new ?

Well, of course OC4J is gone and WebLogic Server has replaced it.

Now, we've got a new WLS template, reflecting the SOA Suite products :

 

 

The installer has been updated (Xtreme Makeover :p) :

 

 

And some webpages have been changed as well :

 

 

 

 

I'm quite a fan of that new graphical aspect.

I unfortunately didn't have enough time to play more with it, so stay tuned for the next episode :)

 

However, what I think could be improved :

 

=> JDeveloper 11g still needs another WLS install to be able to launch (ideally, I would have expected a automated

installation with all the products and only one WLS install, but I'm sure it's on the list)

=> Enterprise Manager wasn't happy since the French resource bundle was missing. I would have prefered the English

one to be loaded by default if not found.

=> For now, loading times (when server starts up) are quite long.

=> I used to use my own JVM memory arguments by using the USER_MEM_ARGS variable. Now, it's impossible because,

it's already used as an internal variable. I would have liked another variable like SOA_MEM_ARGS and the possibility

to still have the USER_MEM_ARGS var overriding the other values.

 

 

Mots clés Technorati : SOA Suite 11g,first look,jdeveloper11g,weblogic server 11g

Building a high available and scalable architecture with WebLogic Server

For once, I'd like to talk about general points. As fas as I know, I still haven't seen any process

that has been respecting all these simple rules.

 

Determine the target

Define your capacity planning.

What kind of business are you dealing with ?

Is it fault tolerant ?

Is there money involved ?

 

Your application

First of all, make sure you know what EAI, ETL, SOA and so on refer to.

Because before you build your house, you'd better know what tools exist & could help you.

 

Designing your application

Separation of concern (a layer per function)

Use design patterns where applicable

Make your developpers write the documentation when they're coding. Else, you'll get some unmaintainable code.

Be sure that each class has its unit test !

No custom database access, use a connection pool.

Think beyond the scope of your application : are there some business services which could be reused later ?

Choose the technology that fits your needs (webservices are not THE ULTIMATE solution)

For instance, don't use Hibernate if you can't use it or tune it properly. Prefer a custom DAO layer if you feel more comfortable with it.

Generally, don't use a tool / framework just because it's cool or trendy. Take it if you really need it and

understand what it can bring to you.

Dependency injection : why not if you master the framework you're using.

What kind of data is going to be exchanged between your application and 3rd party app ? (size, protocol, frequence)

Do you need to manipulate huge loads of data ?

Make some POCs !!! And compare performance measures !

 

Tuning your application for production use

Prefer the request scope instead of session : it will help keeping your session light.

Make all the session objects serializable and don't forget to use "transient" objets where applicable.

Your session should not exceed 35kb (for a good replication performance)

Think about packaging. Don't forget that some people are going to manage your application ! Eventually, use deployment plans.

 

Defining WorkManagers

Even if you don't want to use them for now, define one for each subcomponent (webapp / EJB).

When starting, if WLS doesn't find it, it will use the default : you'll still be able to use them later.

 

Continuous Integration

Think about Maven !

Be sure that your code can be compiled EVERY DAY ! Use reports to detect as early as possible

the cyclomatic complexity, the lack of documentation, unit tests coverage and so on.

When integrating a new version, start from scratch : no iterative delivery.

You can even scratch your install and reinstall it from the beginning : a good way to know if you're at ease with WLST :)

 

Your architecture

 

Which frontend ?

Choose a configuration with a minimum intelligence (WebServer + WLS Plugin)

If you plan using a hardware load-balancer, make sure it handles passive cookies and that you have someone in your team

who knows how it works and how to configure it.

 

How many managed servers ?

Depends on the capacity planning.

Should involve several physical machines, to have better SLA and/or handle a hardware crash.

You may think about setting up a MAN or WAN if needed (site disaster).

 

Security configuration (SSL, Authentication, Connection Filters ...) ?

Set some connection filters to avoid man-in-the-middle attacks.

Use SSL if your data is important and must be encrypted but change the DemoCertificates & Keystores.

Remember that SSL has an impact on performance.

SSL might not be sufficient : use a real authentication if needed.

 

Creating your WebLogic domain

 

Admin Server

Chose a (free) default port and determine if you wish to dedicate a port for administration traffic.

 

Template and WLST

Don't forget to industrialize your building. If you have to setup a new configuration at once,

it will spare you a lot of time.

 

Managed Servers

Pack / Unpack commands to export / import the domain on different machines. (it's like a zip but without the unnecessary files).

 

Setting up a cluster

Use multicast only for backward compatibility.

Else Oracle recommends strongly the use of Unicast

Don't forget about the cluster address (even if WLS is completing it for you, it's good to know which servers are

defined in your cluster).

 

Creating your own certificates

Using keytool, manage your own certificates and keystores (trust & identity).

Once you've got your CSR, use a CA or make your own.

 

Setting up the NodeManager (machines)

I'm going to quote official documentation as it's very detailed.

http://edocs.beasys.com/wls/docs103/nodemgr/nodemgr_config.html

 

Tuning your domain performance

 

Defining relevant log levels

No debug on a production server. By cutting out the unnecessary I/O, you'll gain performance and limit the risks of a "disk "quota exceeded".

Keep the logs on a dedicated file system, so that when they dwell, they do not crash the whole system.

 

Enabling Native I/O

Sounds obvious, but a native socketmuxer can give you a large performance gain.

 

Configuring session replication

Depends if you have to, but if it's the case, be sure your session isn't too big, and that all the objects you want to replicate are serializable.

Chose wisely your replication group.

 

Tuning the thread pool

The thread pool has to be defined to a sufficient initial value (default is one and is obviously not enough).

Adjust this parameter according to the test results you got from your load tests.

 

Tuning the connection pool

One good way to size your connection pool is once again linked to the capacity planning. In a word, you have to put a quite high number

of connections. You launch a stress test and then you take a look at the max number of connections that were created.

It will give you the right number.

 

Configuring Panic & Overload modes

By default, no action is performed on an OOME and on stuck threads. So far, you can't do a lot of things, but it's better than nothing.

If your server faces a OOME (Out Of Memory Exception), you may request it to shut down. That way, if a NodeManager is up,

it can start your server up again. The main con is that you may not be aware of a server failure as it is fully automated.

Same thing for stuck threads : you can either chose to shut the instance down or switch it to an admin mode, that is to say,

serving only administration requests, for the time threads get unstuck.

 

Monitoring (optional but recommended)

 

JConsole

Starting with JVM 1.5, Sun has shipped with its JVM a very nice tool to monitor and supervize a JVM : the JConsole.

http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html

 

JRockit Mission Control

BEA and now Oracle has its own JVM : JRockit. And the team (hi swedish fellows !) has developped a very cool tool as well.

It allows to monitor what's happening on your JVM, but can also record any activity for a later analysis.

 

WLDF

Built-in (and then free) tool to replace the "8.1 Performance Tab".

You can choose to monitor all the MBeans exposed by WebLogic Server, that is to say, almost everything !

http://e-docs.bea.com/wls/docs103/wldf_configuring/understand_wldf_config.html

And you may instrument your code as well, as explained in one of my previous posts.

 

SNMP

The use of external tools such as Nagios, Cacto, Tivoli or Mercury BAC is possible with the activation of SNMP.

http://edocs.beasys.com/wls/docs103/snmpman/index.html

 

Oracle Enterprise Manager

http://www.oracle.com/enterprise_manager/index.html

 

Summary : Application Delivery Process

It sounds obvious but it's not often the case : your application must follow a lifecycle such as :

• => Development / Unit Testing
• => Integration (continuous) : build from scratch the application and make sure it deploys correctly and works.
• => FuncTest : Play all the scenarii that are going to be played by users.
• => Benchmark : Once the application is approved : determine the capacity planning and perform
• stress tests (very high charge on a short timeline) & load tests (110% of your estimated charge on a large timeline)
• => PreProduction : Deployment on this environment as if it were the final production environment.
• => GO-LIVE : Two options.
  • Either you apply the same procedure you did for the PreProduction (with a nightly shutdown for instance)
  • Or you switch the roles and make the PreProduction become the Production and vice-versa.
  • Or best solution, you have two clusters with a load-balancer which is going to perform a "graceful migration".

 

Mots clés Technorati : best practices,weblogic,J2EE,Java EE,lifecycle,production,developpement,continuous integration