Jahia Install Forum > Jahia 4 Websphere...

0 (0 Good)
0 (0 Bad)

Jahia 4 Websphere 6.1

by  mariocauwe »  2011/06/07 20:54

Hi,

I have a Jahia 4 site running on Websphere 5. After a migration to Websphere 6.1, the site is no longer running stable.
After a few days the server is overloaded due to too many open threads. When I investigate the threads, I see many waiting threads with the following stacktrace:
[list:vbsogcc9]
Thread Name= WebContainer : 20591
State = Waiting on condition
Java Stack=
at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:231(Compiled Code)) [b:vbsogcc9]at EDU/oswego/cs/dl/util/concurrent/WaiterPreferenceSemaphore.attempt(Bytecode PC:86(Compiled Code)) at org/jahia/operations/OperationManager.handleOperations(OperationManager.java:161[/b:vbsogcc9](Compiled Code)) at org/jahia/bin/JahiaAction.execute(JahiaAction.java:50(Compiled Code)) at org/apache/struts/action/RequestProcessor.processActionPerform(RequestProcessor.java:421(Compiled Code)) at org/apache/struts/action/RequestProcessor.process(RequestProcessor.java:226(Compiled Code)) at org/apache/struts/action/ActionServlet.process(ActionServlet.java:1164(Compiled Code)) at org/jahia/bin/Jahia.process(Jahia.java:1573(Compiled Code)) at org/jahia/bin/Jahia.service(Jahia.java:870(Compiled Code)) at javax/servlet/http/HttpServlet.service(HttpServlet.java:856(Compiled Code)) at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1146(Compiled Code)) at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:592(Compiled Code)) at com/ibm/ws/wswebcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:524(Compiled Code)) at com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:90(Compiled Code)) at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:751(Compiled Code)) at com/ibm/ws/wswebcontainer/WebContainer.handleRequest(WebContainer.java:1478(Compiled Code)) at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:125(Compiled Code)) at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458(Compiled Code)) at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewInformation(HttpInboundLink.java:387(Compiled Code)) at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.ready(HttpInboundLink.java:267(Compiled Code)) at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214(Compiled Code)) at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113(Compiled Code)) at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:64(Compiled Code)) at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code)) at com/ibm/io/async/AsyncChannelFuture$1.run(AsyncChannelFuture.java:205(Compiled Code)) at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1497(Compiled Code))
Native Stack= No Native stack trace available[/list:u:vbsogcc9]

Does anybody have an idea what is causing this waiting-state or how to fix this?
  • Jahia 4 Websphere 6.1
    2011/06/07 20:54

    mariocauwe Hi,<br/><br/>I have a Jahia 4 site running on Websphere 5. After a migration to Websphere 6.1, the site is no longer running stable.<br/>After a few days the server is overloaded due to too many open threads. When I investigate the threads, I see many waiting threads with the following stacktrace:<br/>[list:vbsogcc9]<br/>Thread Name= WebContainer : 20591<br/>State = Waiting on condition<br/>Java Stack=<br/>at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:231(Compiled Code)) [b:vbsogcc9]at EDU/oswego/cs/dl/util/concurrent/WaiterPreferenceSemaphore.attempt(Bytecode PC:86(Compiled Code)) at org/jahia/operations/OperationManager.handleOperations(OperationManager.java:161[/b:vbsogcc9](Compiled Code)) at org/jahia/bin/JahiaAction.execute(JahiaAction.java:50(Compiled Code)) at org/apache/struts/action/RequestProcessor.processActionPerform(RequestProcessor.java:421(Compiled Code)) at org/apache/struts/action/RequestProcessor.process(RequestProcessor.java:226(Compiled Code)) at org/apache/struts/action/ActionServlet.process(ActionServlet.java:1164(Compiled Code)) at org/jahia/bin/Jahia.process(Jahia.java:1573(Compiled Code)) at org/jahia/bin/Jahia.service(Jahia.java:870(Compiled Code)) at javax/servlet/http/HttpServlet.service(HttpServlet.java:856(Compiled Code)) at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1146(Compiled Code)) at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:592(Compiled Code)) at com/ibm/ws/wswebcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:524(Compiled Code)) at com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:90(Compiled Code)) at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:751(Compiled Code)) at com/ibm/ws/wswebcontainer/WebContainer.handleRequest(WebContainer.java:1478(Compiled Code)) at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:125(Compiled Code)) at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458(Compiled Code)) at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewInformation(HttpInboundLink.java:387(Compiled Code)) at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.ready(HttpInboundLink.java:267(Compiled Code)) at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214(Compiled Code)) at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113(Compiled Code)) at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:64(Compiled Code)) at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code)) at com/ibm/io/async/AsyncChannelFuture$1.run(AsyncChannelFuture.java:205(Compiled Code)) at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1497(Compiled Code))<br/>Native Stack= No Native stack trace available[/list:u:vbsogcc9]<br/><br/>Does anybody have an idea what is causing this waiting-state or how to fix this?

  • Number of messages  20
    Registration date Jun 7, 2011
    0 (0 Good)
    0 (0 Bad)

    Re: Jahia 4 Websphere 6.1

    by  shyrkov »  2011/06/07 20:54

    Hello Mario,

    could you, please, provide a bit more details about the problem?

    1) Which exact version and build number of Jahia 4 are you running now?
    2) If present in your Jahia build, please, post here the values for "maxParallelProcessings" and "pageGenerationWaitTime" parameters in your WEB-INF/etc/config/jahia.properties file
    3) If possible, create and provide a thread dump (on *nix systems can be done using kill -QUIT <process-id> command) that we can see, which threads are blocked

    Kind regards
    Sergiy

  • Re: Jahia 4 Websphere 6.1
    2011/07/04 15:57

    shyrkov <p> Hello Mario,<br /> <br /> could you, please, provide a bit more details about the problem?<br /> <br /> 1) Which exact version and build number of Jahia 4 are you running now?<br /> 2) If present in your Jahia build, please, post here the values for &quot;maxParallelProcessings&quot; and &quot;pageGenerationWaitTime&quot; parameters in your WEB-INF/etc/config/jahia.properties file<br /> 3) If possible, create and provide a thread dump (on *nix systems can be done using kill -QUIT &lt;process-id&gt; command) that we can see, which threads are blocked<br /> <br /> Kind regards<br /> Sergiy</p>

  • Number of messages  407
    Registration date Jun 7, 2011
    0 (0 Good)
    0 (0 Bad)

    Re: Jahia 4 Websphere 6.1

    by  shyrkov »  2011/06/07 20:54

    Other issue came to mind mind that should be easier to check.
    Please, confirm that you have migrated from JDK 1.4 (WebSphere 5.1) to JDK 1.5 (WebSphere 6.1)?
    And check if you have mappings for compiled JSPs in your web.xml file, similar to:



       <servlet>
    
          <servlet-name>views.engines.engines_default_layout_jsp</servlet-name>
    
          <servlet-class>views.engines.engines_default_layout_jsp</servlet-class>
    
       </servlet>

     

    Thanks!

     

  • Re: Jahia 4 Websphere 6.1
    2011/07/04 15:58

    shyrkov <p> Other issue came to mind mind that should be easier to check.<br /> Please, confirm that you have migrated from JDK 1.4 (WebSphere 5.1) to JDK 1.5 (WebSphere 6.1)?<br /> And check if you have mappings for compiled JSPs in your web.xml file, similar to:</p> <br /> <br /> <p> <title></title> </p> <pre> &lt;servlet&gt; &lt;servlet-name&gt;views.engines.engines_default_layout_jsp&lt;/servlet-name&gt; &lt;servlet-class&gt;views.engines.engines_default_layout_jsp&lt;/servlet-class&gt; &lt;/servlet&gt;</pre> <p> &nbsp;</p> <p> Thanks!</p> <p> &nbsp;</p>

  • Number of messages  407
    Registration date Jun 7, 2011
    0 (0 Good)
    0 (0 Bad)

    Re: Jahia 4 Websphere 6.1

    by  mariocauwe »  2011/06/07 20:54

    Thanks for the fast reply!

    maxParallelProcessings = 10
    pageGenerationWaitTime = 1800000

    I did not rebuild the jahia project, so it's still compiled for Java 1.4 instead of Java 5. I assumed it would be adequate because it runs fine on the testserver.

    The Jahia build is 4.2.1 build 19618

    The servlet mapping you mention there, I don't have. Actually, it doens't sound familiar either.
  • Re: Jahia 4 Websphere 6.1
    2011/06/07 20:54

    mariocauwe Thanks for the fast reply!<br/><br/>maxParallelProcessings = 10<br/>pageGenerationWaitTime = 1800000<br/><br/>I did not rebuild the jahia project, so it's still compiled for Java 1.4 instead of Java 5. I assumed it would be adequate because it runs fine on the testserver.<br/><br/>The Jahia build is 4.2.1 build 19618<br/><br/>The servlet mapping you mention there, I don't have. Actually, it doens't sound familiar either.

  • Number of messages  20
    Registration date Jun 7, 2011
    0 (0 Good)
    0 (0 Bad)

    Re: Jahia 4 Websphere 6.1

    by  shyrkov »  2011/06/07 20:54

    Hello,

    yes, there should be no problem with the Jahia Java code for JDK upgrade.
    What I meant are the compiled JSPs and especially the corresponding mappings in the web.xml file (we provide Jahia with the pre-compiled JSPs). We had multiple customers experiencing problems, when upgrading server (especially Tomcat, due to JSP compiler changes). That is why I have asked about the pre-compiled JSPs.

    Have you tried to tune the values for maxParallelProcessings and pageGenerationWaitTime?
    The default value for maxParallelProcessings is 40 threads and for the pageGenerationWaitTime - 2 minutes. And for production we recommend higher maxParallelProcessings and lower pageGenerationWaitTime values.

    Do you see any 503 (server overloaded) errors in the server log/console?
    For troubleshooting such cases it would be really helpful to have a thread dump, when such problem occurs (see Performing Stack trace dumps)

    Kind regards
    Sergiy

  • Re: Jahia 4 Websphere 6.1
    2011/07/04 15:59

    shyrkov <p> Hello,<br /> <br /> yes, there should be no problem with the Jahia Java code for JDK upgrade.<br /> What I meant are the compiled JSPs and especially the corresponding mappings in the web.xml file (we provide Jahia with the pre-compiled JSPs). We had multiple customers experiencing problems, when upgrading server (especially Tomcat, due to JSP compiler changes). That is why I have asked about the pre-compiled JSPs.<br /> <br /> Have you tried to tune the values for maxParallelProcessings and pageGenerationWaitTime?<br /> The default value for maxParallelProcessings is 40 threads and for the pageGenerationWaitTime - 2 minutes. And for production we recommend higher maxParallelProcessings and lower pageGenerationWaitTime values.<br /> <br /> Do you see any 503 (server overloaded) errors in the server log/console?<br /> For troubleshooting such cases it would be really helpful to have a thread dump, when such problem occurs (see&nbsp;<a href="http://extranet.jahia.com/downloads/jahia/jahia5.0.5/readme/howto_monitoring.html#stacktracedumps">Performing Stack trace dumps</a>)<br /> <br /> Kind regards<br /> Sergiy</p>

  • Number of messages  407
    Registration date Jun 7, 2011
    0 (0 Good)
    0 (0 Bad)

    Re: Jahia 4 Websphere 6.1

    by  mariocauwe »  2011/06/07 20:54

    Hi,
    I didn't recently tune these parameter values. They must have been like this for a long time, certainly on Websphere 5.1 deploy as well. Maybe I should adapt them anyway.

    You can find the javacore file attached to this posting.
    You'll notice many threads in the 'running' state as well, but that's due to a non-responding webservice call. I don't think it's related. I'm currently more concerned about the 'waiting' threads.

    Due to the threads, we do indeed have 503 responses in the log:
     

     

    [2/9/10 12:32:18:951 CET] 0000504c ThreadMonitor W   WSVR0606W: Thread "WebContainer : 20470" (0000504c) was previously reported to be hung but has completed.  It was active for approximately 1800317 milliseconds.  There is/are 166 thread(s) in total in the server that still may be hung.
    
    [2/9/10 12:32:23:543 CET] 0000504d WebApp        A   SRVE0180I: [ear5.1#O001P001_Jahia4Web421.war] [/jahia] [Servlet.LOG]: Your Jahia Server has generated an error. Please review the details below for additional information:
    
    Severity : ERROR
    
    Error : 503 - server overloaded, 503 - Server Overloaded

    About the precompiled jsp's, I don't have anything like you mentioned, just this one that seems more or less relevant:

    <servlet>

    <servlet-name>JspPrecompileServlet</servlet-name>

    <servlet-class>org.jahia.tools.precompile.JspPrecompileServlet</servlet-class>

    </servlet>

  • Re: Jahia 4 Websphere 6.1
    2011/07/04 16:01

    mariocauwe <p> Hi,<br /> I didn&#39;t recently tune these parameter values. They must have been like this for a long time, certainly on Websphere 5.1 deploy as well. Maybe I should adapt them anyway.<br /> <br /> You can find the javacore file attached to this posting.<br /> You&#39;ll notice many threads in the &#39;running&#39; state as well, but that&#39;s due to a non-responding webservice call. I don&#39;t think it&#39;s related. I&#39;m currently more concerned about the &#39;waiting&#39; threads.<br /> <br /> Due to the threads, we do indeed have 503 responses in the log:<br /> &nbsp;</p> <p> &nbsp;</p> <p> <title></title> </p> <pre> [2/9/10 12:32:18:951 CET] 0000504c ThreadMonitor W WSVR0606W: Thread &quot;WebContainer : 20470&quot; (0000504c) was previously reported to be hung but has completed. It was active for approximately 1800317 milliseconds. There is/are 166 thread(s) in total in the server that still may be hung. [2/9/10 12:32:23:543 CET] 0000504d WebApp A SRVE0180I: [ear5.1#O001P001_Jahia4Web421.war] [/jahia] [Servlet.LOG]: Your Jahia Server has generated an error. Please review the details below for additional information: Severity : ERROR Error : 503 - server overloaded, 503 - Server Overloaded</pre> <p> About the precompiled jsp&#39;s, I don&#39;t have anything like you mentioned, just this one that seems more or less relevant:</p> <p> &lt;servlet&gt;</p> <p> &lt;servlet-name&gt;JspPrecompileServlet&lt;/servlet-name&gt;</p> <p> &lt;servlet-class&gt;org.jahia.tools.precompile.JspPrecompileServlet&lt;/servlet-class&gt;</p> <p> &lt;/servlet&gt;</p>

  • Number of messages  20
    Registration date Jun 7, 2011
    0 (0 Good)
    0 (0 Bad)

    Re: Jahia 4 Websphere 6.1

    by  mariocauwe »  2011/06/07 20:54

    It seems the hanging threads (due to the none-responsive webservices) were actually causing the other threads to hang as well. By fixing the webservice, the site runs stable now. Thanks for your input!
  • Re: Jahia 4 Websphere 6.1
    2011/06/07 20:54

    mariocauwe It seems the hanging threads (due to the none-responsive webservices) were actually causing the other threads to hang as well. By fixing the webservice, the site runs stable now. Thanks for your input!

  • Number of messages  20
    Registration date Jun 7, 2011
    Contact
    Share
    Feedback

    Get in touch

    Whether you are a current user or if you are just evaluating Jahia, we are here to help.

    Contact us

    Share this page