2008-08-14

How does WebLogic handle socket muxers ?

Muxer ? What's that ?

Taken from the documentation (http://edocs.bea.com/wls/docs100/perform/WLSTuning.html#wp1152246) :

WebLogic Server uses software modules called muxers to read incoming requests on the server and incoming responses on the client.

These muxers are of two primary types: the Java muxer or native muxer. A Java muxer has the following characteristics:

    * Uses pure Java to read data from sockets.

    * It is also the only muxer available for RMI clients.

    * Blocks on reads until there is data to be read from a socket.

    (This behavior does not scale well when there are a large number of sockets and/or when data arrives infrequently at sockets.

    This is typically not an issue for clients, but it can create a huge bottleneck for a server. )

      Native muxers use platform-specific native binaries to read data from sockets. The majority of all platforms provide some mechanism to poll a socket for data.

       

      For example, Unix systems use the poll system and the Windows architecture uses completion ports.

      Native provide superior scalability because they implement a non-blocking thread model.

      When a native muxer is used, the server creates a fixed number of threads dedicated to reading incoming requests.

       

      BEA recommends using the default setting of selected for the Enable Native IO parameter which allows the server automatically selects the appropriate muxer for the server to use.

      If the Enable Native IO parameter is not selected, the server instance exclusively uses the Java muxer.

      This maybe acceptable if there are a small number of clients and the rate at which requests arrive at the server is fairly high.

      Under these conditions, the Java muxer performs as well as a native muxer and eliminate Java Native Interface (JNI) overhead. Unlike native muxers,

      the number of threads used to read requests is not fixed and is tunable for Java muxers by configuring the Percent Socket Readers parameter setting in the Administration Console.

       

      See Changing the Number of Available Socket Readers.

      Ideally, you should configure this parameter so the number of threads roughly equals the number of remote concurrently connected clients up to 50% of the total thread pool size.

      Each thread waits for a fixed amount of time for data to become available at a socket.

      If no data arrives, the thread moves to the next socket.

      Then, for those reasons, it is obviously better to use native muxers.

       

      How do you know that your muxers are native and not java ?

      Well, you've got two ways to know that.

      Log File

      First one, when you start your server, if the log level is high enough, you should see a line like this one :

      <Info> <Socket> <BEA-000446> <Native IO Enabled.>

      If you don't have the opportunity to check for that line in the logs, it is possible to see that information, thanks to thread dumps.
      To perform a thread dump, you've got to send a precise signal to the JVM (http://en.wikipedia.org/wiki/SIGQUIT).

      Thread dump

      Perform a thread dump

      If you're in a Windows environment and your server has been launched with a startup script, switch to the command line window and press CTRL + Break.

      That will display the thread dump in the console.

      If you're in a Windows environment, but in a service mode, use the beasvc tool to perform the dump. (beasvc -dump)

      If Unix is your playground, then you should use the following command "kill -3 PID" (PID is the process id of your JVM)

      Another way is to use WLST to perform the dump (quite useful in a AIX environment, trust me !). Take a look at one of my previous post to know more about it.

      Once you have your thread dump, you'll have to analyze it.

      Analyze a thread dump

      In the dump, look for a section in which you can find queue: 'weblogic.socket.Muxer'"

      Like that :

      "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x5c tid=2492 prio=5 alive, in native, daemon
          at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)
          at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)
          at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
          at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
          at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
          at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
          at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
          -- end of trace

      Here you can see (in blue) that the muxer used is the NTSoketMuxer. As long as it is NOT the class weblogic.socket.SocketMuxer, you're using a native muxer.

      Else it's a java muxer.

       

      List of muxers

      Java Muxer

      As said above, a java muxer is embodied by the class weblogic.socket.SocketMuxer

      Native muxers

      Windows : weblogic.socket.NTSocketMuxer

      UNIX weblogic.socket.PosixSocketMuxer & weblogic.socket.DevPollSocketMuxer

      Note : On UNIX environments (such as Solaris and HP-UX), WebLogic Server uses DevPollSocketMuxer by default,

      which prints out unnecessary SocketExceptions errors to the server log.

      As a workaround, you can enable PosixSocketMuxer via the WebLogic Scripting Tool as follows:

      edit()

      cd ('Servers/'MyServerName')

      startEdit()

      set('MuxerClass','weblogic.socket.PosixSocketMuxer')

      activate()

       

      Mots clés Technorati : ,,,,

      5 comments:

      Anonymous said...

      Bonjour,

      Merci pour ce poste très instructif. Je voudrais simplement partager une petite expérience personnelle qui me laisse un peu perplexe. J'ai récemment modifié la configuration d'un cluster Weblogic 9 (sous Linux) pour utiliser le fameux muxer posix natif.

      Dans un premier temps, j'ai constaté que les native I/O étaient désactivées. Cela était du au fait que nous utilisions une distribution Weblogic 32 bits sur une plate-forme Red Hat 64 bits, et la JVM cherchait les librairies dans un répertoire x86_64 qui n'existait pas. Après installation d'une distribution 64 bits, plus aucun problème, retour des native I/O en enabled.

      Ce qui me laisse perplexe donc, c'est qu'avant ce changement de configuration, les native I/O étaient bien en enabled, et que le serveur devait donc utiliser un muxer natif. Je ne comprends pas quelles étaient les librairies natives utilisées à ce moment là, vu que le mécanisme de recherche est probablement le même (variable arch du ${WL_HOME}/common/bin/commEnv.sh initialisée à l'aide de la commande uname -m) pour tous les muxers natifs.

      Y a-t-il un moyen de tracer le chargement de ces librairies ?

      En tout cas, excellent blog :)

      Yann

      Anonymous said...

      Apparemment, la librairie libmuxer.so n'est pas utilisée avec le muxer natif par défaut (EPollSocketMuxer). Elle est par contre bien chargée au boot si on utilise le PosixSocketMuxer. Cela répond à mon commentaire précédent.

      Maxence Button said...

      Bonjour Yann !

      Eh bien, ça c'est le type de question que j'affectionne particulièrement car vous y avez répondu seul ;)

      Il faut que je retrouve cette commande Unix qui trace les librairies chargées par un process.
      Cela enrichira ce post.

      Et merci pour le gentil mot.

      A bientôt !

      Yannick said...

      Cher Max,

      I have an issue for you!

      WLS 8.1 sp6. I use nativeI/O and I can see PosixSocketMuxer is used in the 'weblogic.socket.Muxer' queue. Great so far!

      Except it doesn't appear in 'weblogic.kernel.Default' queue, threads use SocketMuxer. Is-it normal?

      I ask this question because all my threads are blocked when my website is stressed with intensive load. They are all stucked on ReplicationManager.updateSecondary(). As you can guess I have session replication enabled.


      99 threads are blocked on weblogic.rjvm.ResponseImpl.waitForData and 1 thread is blocked on java.net.SocketOutputStream.socketWrite0 (the one using SocketMuxer).

      Unfortunately I can share the thread dump on this post.

      Bise a ta puce,
      Take care,

      Yannick (Robin)

      Thread "ExecuteThread: '66' for queue: 'weblogic.kernel.Default'": I/O wait (0:03:45.774)

      Address: 0x00000800201ffa00, Priority: 5, Object blocked: 15,041 ms, Object wait: 6,098,248 ms, CPU wait: 1,532 ms, I/O wait: 150,425 ms, CPU: 231,783 ms

      * java.net.SocketOutputStream.socketWrite0 (native method)
      * java.net.SocketOutputStream.socketWrite (SocketOutputStream.java:92, bci=58, server compiler)
      * java.net.SocketOutputStream.write (SocketOutputStream.java:136, bci=4, server compiler)
      * com.wily.introscope.agent.probe.net.ManagedSocketOutputStream.write (ManagedSocketOutputStream.java:78, bci=41, server compiler)
      * weblogic.socket.SocketMuxer.write (SocketMuxer.java:758, bci=34, server compiler)
      * weblogic.rjvm.t3.T3JVMConnection.sendMsg (T3JVMConnection.java:936, bci=31, server compiler)
      * weblogic.rjvm.MsgAbbrevJVMConnection.sendOutMsg (MsgAbbrevJVMConnection.java:269, bci=112, server compiler)
      * weblogic.rjvm.MsgAbbrevJVMConnection.sendMsg (MsgAbbrevJVMConnection.java:167, bci=75, server compiler)
      * weblogic.rjvm.ConnectionManager.sendMsg (ConnectionManager.java:552, bci=318, server compiler)
      * weblogic.rjvm.RJVMImpl.send (RJVMImpl.java:910, bci=209, server compiler)
      * weblogic.rjvm.MsgAbbrevOutputStream.flushAndSendRaw (MsgAbbrevOutputStream.java:292, bci=13, server compiler)
      * weblogic.rjvm.MsgAbbrevOutputStream.flushAndSend (MsgAbbrevOutputStream.java:300, bci=6, server compiler)
      * weblogic.rjvm.MsgAbbrevOutputStream.sendRecv (MsgAbbrevOutputStream.java:322, bci=27, server compiler)
      * weblogic.rjvm.BasicOutboundRequest.sendReceive (BasicOutboundRequest.java:103, bci=13, server compiler)
      * weblogic.rmi.internal.BasicRemoteRef.invoke (BasicRemoteRef.java:164, bci=80, client compiler)
      * weblogic.cluster.replication.ReplicationManager_816_WLStub.update (unknown source, bci=57, server compiler)
      * weblogic.cluster.replication.ReplicationManager.updateSecondary (ReplicationManager.java:863, bci=53, server compiler)
      * weblogic.servlet.internal.session.ReplicatedSessionData.syncSession (ReplicatedSessionData.java:500, bci=138, server compiler)

      Amarendra Konda said...

      Hi,

      That's good explanation about the socket threads.
      While running some tests, i have observed that most of the socket threads are BLOCKED. and the stacktrace is as follows

      "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" - Thread t@24
      java.lang.Thread.State: BLOCKED on java.lang.String@1f121f7 owned by: ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'
      at weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:93)
      at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
      at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
      at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
      at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

      we are using WebLogic 10, on Solaris 10 ( SPARC T 2000).

      Why these threads are getting blocked? Thanks in advance for your valuable time.

      Amar