2009-02-03

Tuning the default WorkManager on WebLogic Server

 

If you're not yet familiar with WorkManagers, well, it's time ! :)

With WLS 8, we had the concept of ExecuteQueues, replaced by workmanagers, starting with WLS 9. What's changed ?

Well, it's got self-tuning, allows a large number of SLA combinations and ... well, go read the official documentation about it :)

(Gush, I wanted to point towards E-docs, but it has been finally killed ... RIP ! I'll miss you)

So have a look at this good article instead.

 

Playing with the default workmanager

 

The default workmanager, as its name tells, is the workmanager defined by default.Thus, all applications deployed on WLS will use it.

One good thing is to first understand how they work, and then define some custom, matching the business rules or SLA you need.

But sometimes, when your application is already in production, it's obvious you can't take your EAR / WAR, update the deployment descriptor(s) and redeploy it.

(well, you could if your app isn't too big, thanks to side-by-side deployment, but that's another story)

But if you're interested, here's a live demo by my colleague Steve Button)

 

The default workmanager belongs to a threadpool.

As the initial threadpool comes with only five threads, that's not much.

 

image

 

If your application has to face a large number of hits, you may want to start with more than that.

Well, that's quite easy. You've got two ways to do so.

 

1) Modifying the config.xml

Just add the following line(s) in your server definition :

 

<server>

<name>AdminServer</name>

<self-tuning-thread-pool-size-min>100</self-tuning-thread-pool-size-min>

<self-tuning-thread-pool-size-max>200</self-tuning-thread-pool-size-max>

[...]

</server>

2) Adding some JVM parameters

 

Personally, that's my favorite option since I don't like when people go in a generated config file

and mess up with tags. If you're sure of what you're doing, that's ok, but it's safer the following way :

add the following option in your command line : -Dweblogic.threadpool.MinPoolSize=100

 

Observing the result

 

Personally, I decided to apply the change by adding the JVM parameter.

To do so, I just updated the startWeblogic script with the USER_MEM_ARGS line.

 

@ECHO OFF

@REM WARNING: This file is created by the Configuration Wizard.
@REM Any changes to this script may be lost when adding extensions to this configuration.

SETLOCAL

set DOMAIN_HOME=D:\BEA_ROOT\user_projects\domains\essex\ora_bpel
set USER_MEM_ARGS=-Dweblogic.threadpool.MinPoolSize=100
call "%DOMAIN_HOME%\bin\startWebLogic.cmd" %*

ENDLOCAL

 

Reboot WLS and see the option has been taken into account :

 

image

 

And finally :

 

image

 

Wasn't so hard, was it ? :)

 

27 comments:

Pete said...

Nice article - does this mean that I no longer have to worry about individual thread pools for different processes?

I currently have one setup for the web-app, with say 50 threads in it....another for report generation with say 25, one for SOAP and all the SOAP traffic goes to that one.

As I read it, you set it at a sensible size and let WebLogic sort it all out for you??

Is that how you would do it in a production environment?

This is one of the mysterious things about WLS10 as we're still running 8.1.

Pete

Maxence Button said...

Hi Pete !

I will change my article a bit since I guess some terms are misleading.
You have to make the difference between the general thread pool and the work managers. The thread pool represents the number of threads allocated for the server. Workmanagers are dedicated to applications.
If the threadpool is set to a max of 200 threads, then the sum of the threads defined in the workmanagers must not exceed 200.

To answer your question, in production you would set a higher than default "minimal value" for the threadpool but no maximal value (WLS will sort it out by itself).
And a workmanager per application is a good thing. That way, you will be able to handle your SLA with a better accuracy. (Fine-grained, to use a trendy expression in the SOA world)

I guess it's clearer. Else, don't hesitate to post !

Anonymous said...

Its really a very useful article.Thank you very much for the details about work manager which is not easily found in the net.

can you please help me how to associate an application with the work manager(newly created)? - Is it done through console or through deployment descriptor.

Hoping for a lucid explanation.

Thank you,
Ranjani

Maxence Button said...

Hi Ranjani,

Thank you for the kind word.
Well, what you've got to know is that you may declare your workmanager either in your application (description descriptor) or through the console.

Once created, the way to link your workmanager to your app is once again in the deployment descriptor.
http://edocs.bea.com/wls/docs100/config_wls/self_tuned.html#wp1059038

Look for "dispatch-policy". You may reference a workmanager that doesn't exist. It will then use the default workmanager.

That's a good practice to declare a workmanager for each web application or EJB even if it doesn't exist. You will have a warning but the day when you want to use a dedicated workmanager, you won't have to modify your app. The simple fact to declare it through the console will automatically assign it to your app at the next restart.

HTH

Anonymous said...

Hi Maxence Button,

Thanks a lot for the immediate response. It was very useful.

I would like to have an important piece of advice from you on tuning our production WebLogic server - currently we are faced with the following scenario: We are performing load test on our server/application. Our application deployed on a managed server is not able to handle more than 650-700 sessions (cumulative count). Thereafter the server goes into 'Warning' state.

Exception thrown on the console is:
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

Reason: 'ThreadPool has stuck threads'

We understand that the error has got something to do with tuning the server. It would very nice if you could throw some light towards resolving this seemingly simple but severe problem we are faced with.

Thank you,
Ranjani

Anonymous said...

Hi Maxence Button,

Thanks a lot for the immediate response. It was very useful.

I would like to have an important piece of advice from you on tuning our production WebLogic server - currently we are faced with the following scenario: We are performing load test on our server/application. Our application deployed on a managed server is not able to handle more than 650-700 sessions (cumulative count). Thereafter the server goes into 'Warning' state.

Exception thrown on the console is:
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

Reason: 'ThreadPool has stuck threads'

We understand that the error has got something to do with tuning the server. It would very nice if you could throw some light towards resolving this seemingly simple but severe problem we are faced with.

Thank you,
Ranjani

Maxence Button said...

Hi again Ranjani,

Well, if you're starting to see stuck threads, the only tuning you'll be able to do is to configure the panic and/or overload mode.
http://edocs.bea.com/wls/docs103/config_wls/overload.html#wp1063453
Combined with the NodeManager and a cluster, it will not fix the problem, but help to reach a better QoS.

Anyway, my advice would be to perform regular thread dumps when your server starts to scream. (BTW, you can find a WLST script in the blog archives that will help you to do so).
Then take a closer look to the stacktraces and locate the code responsible for the lock. The major reason to this behavior may be because of an external part (file system, database ...).
To be sure => thread dumps !

Best regards.

Anonymous said...

We are facing a similar issue with stuck threads. I am unclear why 1 or 2 stuck threads seem to "cripple" an application. It seems like there should be enough threads left to handle any new requests. Can you shed some light on the reason for this? Also, where was the WLST script located to generate the thread dumps?

Maxence Button said...

You're right : only one or two stuck thread won't prevent the application server from serving requests.

But the fact is : when you're starting to get stuck threads, the number is likely to keep growing till your server die.

To know what's responsible for that, you will have to perform threaddumps.
One way to do so, is to use WLST(
http://m-button.blogspot.com/2008/07/using-wlst-to-perform-regular.html)

Then using a tool like Samourai or TDA (https://tda.dev.java.net/) will help you find the source of your problem.

Hope this helps.

ewdunkle said...

Maxence, I wish I could find details like this in WebLogic docs. How do I go about setting other tunings for the default Work Manager? Is there a timeout setting somewhere? I am getting timeouts when deleting content via the portal console.

Thanks.

Anonymous said...

Hi,
I'm working with global workmanager and I'm facing a trouble about tuning issues. I've been defined a MinThreadConstraints on my WM to 20. When I monitoring SERVER's workload, I found OutOfOrder Execution Count and MaxWaitTime whith non-zero values. Weblogic documentation about this MBEan attributes is not so clear. What is your opinion about theese values? Best regards. Nat.

Maxence Button said...

@edwunkle : Unfortunately, the official doc (even if I shouldn't say that) is quite poor on the subject. It could get even worse when the BEA E-docs site will be decommissioned by the end of the month. Some migration process proved that some data was lost during the process. Well, if you're concerned about that, I advise you to contact the Oracle Support and ask for links, pointers and all the documentation they can provide.
In the meantime, I'll try to complete this post with my pointers.

@Nat : Sorry, I can't answer that specific question with my knowledge. You should ask the Oracle Support for that matter. I'll take a look but I can't promise anything :)
(I've been pretty busy these last weeks as you may have noticed)

Syed said...

Hi Button,

Nice arcticle. But my doubt is how can we increase no of threads to our wish? it also needs adequate memory to support that if i am not wrong. As when i was doing the changes i got Our of memory error while hitting the application console.

Please clarify

Thanks,
Syed.

Anonymous said...

Hi,
We are using commonj Work manager. WE understand that this needs to be mapped to WL Work manager. Could you pls give some inputs on how this mapping details can be propogated to different environments. Do we need to write some script or the definitions/declartaions in deployment descriptor will take care of it. Expecting an early reply..


thanks,
MS

Maxence Button said...

@Syed : yes you're definitely right. You can't just set a high number of threads and hope it's gonna fit your needs. You'll probably have to perform workload tests and adjust the number of threads (as well as the number of database connections) which will give you the best performance level. It entirely depends on your application : no magic settings.

@MS : Well, I must admit I'm not an expert of CommonJ, but what I understand is that you have to call for a WorkManager name.
Prior to deploying your app, you have to ensure that this workmanager exists, by executing WLST scripts or be sure that your DD declare it.
WLST might be a more elegant solution since it's more "container-oriented".

Hope it was clear.

Anonymous said...

Hi maxence,
I have a cluster environment running on 9.2 weblogic. The issue we are facing if we do not define any work manager at all, which means weblogic uses its own workmanager. I see that we are running out of the Jdbc connections, in turn weblogic is spawning upto 200 threads which is making the memory resources to be used up fully. After reading the doc bea provides(which is not good regarding this work manager thing), I see that we can create some workmanagers specifically for the application. Below is the one I tried to do.

1.Create a new work manager named default.
2.Within this work manager create a MaxThreadsConstraint leaving the count to default -1 and enter the DataSource name of the Pool in the DataSource field. The max capacity of the data source is used as the constraint. This means that the number of threads that will get used at the maximum will be the limit the Jdbc connection pool set. This ensures that if at all at any point of time all the Jdbc connection pools get used, weblogic does not spawn more threads and starts rejecting the requests.
3.Then in the available targets section select the cluster to deploy this work manager for.


Over here do you see any problem, or any more tuning that I can use for higher performance? Also the name of the work manager I gave is default. What I learned from one BEA engineer is that if I do not give it default, then weblogic in addition to the WM that I define also uses the usual default WM on its own. Is this true? So should I define the WM name to be default to make sure the weblogic should and must use only this WM that I create and not any others on its own?

Appreciate you help me over here.

Ven.

Maxence Button said...

Hi Ven,

I agree with you : as I wrote before, the official documentation could be a bit more detailed about Workmanagers.
But nothing's perfect !

Well, the advice you got from the support is relevant. If you don't name your workmanager "default", WebLogic will consider it as a new one and won't then affect your "main" workmanager.

As for the configuration you've got, there's nothing I could say about it, if it fits your application needs.

But I would advise you to elaborate a capacity planning, then to shoot your test and see how many JDBC connections you really need. That way, it will prevent you from uselessly enqueuing requests. (don't forget that in a cluster, the number of JDBC connections in the pool is multiplied by the number of managed servers you have).

Indeed, if you underestimated the charge you'll have to face, it could lead to a server crash.

Hope it helps.

Anonymous said...

Question: That way, it will prevent you from uselessly enqueuing requests. (don't forget that in a cluster, the number of JDBC connections in the pool is multiplied by the number of managed servers you have).

Does this rule still apply if the managed servers are not in a cluster? Example. If I have a single connection pool setup for 25 connections and the connection is targeted to four managed servers. Do I have 25 connections for the four servers or 100 connections?

Maxence Button said...

Hi,

Yes, this rule is still true in your case.

If you have a connection pool defining 5 connections and you target it to 4 managed servers, you'll have 20 (5x4) connections open on your database.

The best proof you could have is to do the test yourself.

Regards.

Richard said...

Hi Maxence,
thanks for documenting these flags. It was a good starting point.

Just a note, the command-line property

-Dweblogic.threadpool.MinPoolSize=nnn

and the xml tag self-tuning-thread-pool-size-min exhibit different behaviour. The former silently limits the initial size to 400, the latter doesn't show this limit.

I am currently trying to get this documented by oracle via a support case.

Jim said...

This is a great Blog!!!

I'm looking for a way to override the default work manager that weblogic 10.3 has. I created a New global work manager and attached a Max thread-10 constarint to it, when i heavily load the server the exceute thread count is going above 10. I didn't add the work manager to the deplyment descriptor.

I created a work manager with the name "default" and tried it as well, still it not restricting the number of threads. Is there anyway i cant get a MAX thread constraint to work???

Is there a way to make this work from the admin console alone?

Srinivas said...

Hey Max , i really became your fan !

Anonymous said...

Hi,

We are running WebLogic 10g3 over RHEL 5. And its taking forever to come up. Is there any way to check where it might be taking time

Avinah durge said...

Replace

securerandom.source=file:/dev/urandom (in /bea/jdk160_05/jre/lib/security/java.security file)

with

securerandom.source=file:/dev/./urandom

it improves the starup performance.

Naveen said...

hi,

We have a problem with number of open sockets in our application.
This is the result of increase in number of stuck threads.

But what we observe is these open sockets are getting deleted automatically after a certain count or after a certain time, which we could not figure out exactly.

Can you please let me know which feature in weblogic allows this, i.e., deletion of open sockets automatically and how can we configure it.

Thanks in Advance,

Naveen

Naveen said...

hi,

We have a problem with number of open sockets in our application.
This is the result of increase in number of stuck threads.

But what we observe is these open sockets are getting deleted automatically after a certain count or after a certain time, which we could not figure out exactly.

Can you please let me know which feature in weblogic allows this, i.e., deletion of open sockets automatically and how can we configure it.

Thanks in Advance,

Naveen

appi said...

HI Max ,

We have created a workmanager with Min and Max Thread constraint and associated it with the MQSeriesAdpater application deployed in OracleSOASuite installation.

However, under load, we do not see the requests being processed by the work manager in the console. Please let us know if any additional configuration is required or not for the work manager settings.

BR ,
Appi