We recently migrated Cisco CVP 9 to CVP 11 and after every 8 days we are facing out of memory exception java.lang.OutOfMemoryError: Java heap space.
We never faced this issue in Version 9 CVP application code but only after upgrade we are getting these issues. We are rebooting servers week to reset the memory errors.
We are noticing following two issues,
Issue 1
- IVR web service does not provide any response till 30 seconds during the issue. We notice a lot of bad fetch errors and ultimately have to reboot servers and everything becomes normal.
Issue 2
- We also see activity logger not generating log files and the attached error log throws a java error,
*********************************************************************************************************************************************************
08/21/2017 08:02:43.668, SERVER ERROR: An error occurred while trying to log a different error encountered while attempting to log an event. A listing of the original error follows this one in the error log.
java.lang.IllegalStateException: getAttribute: Session already invalidated
at org.apache.catalina.session.StandardSession.getAttribute(StandardSession.java:1175)
at org.apache.catalina.session.StandardSessionFacade.getAttribute(StandardSessionFacade.java:122)
at com.audium.server.session.ControllerData.getSessionLoggerScratchData(ControllerData.java:692)
at com.audium.server.session.ControllerData.getLoggerScratchData(ControllerData.java:4921)
at com.audium.server.session.LoggerAPI.getLoggerScratchData(LoggerAPI.java:72)
at com.audium.logger.application.activity.ActivityLogger.doError(ActivityLogger.java:695)
at com.audium.logger.application.activity.ActivityLogger.log(ActivityLogger.java:174)
at com.audium.server.logger.ApplicationLoggerBase.runLogger(ApplicationLoggerBase.java:178)
at com.audium.server.logger.EventFactory.handleLoggerErrorEvent(EventFactory.java:1822)
at com.audium.server.logger.LoggerManager$LoggerRunner.run(LoggerManager.java:77)
at com.audium.server.util.ThreadPool$Worker.run(ThreadPool.java:775)
at java.lang.Thread.run(Thread.java:744)
*********************************************************************************************************************************************************
Current JVM Config
*********************************************************************************************************************************************************
Current JVM config
-Djre.home=C:\Cisco\CVP\jre
-Dcatalina.base=C:\Cisco\CVP\VXMLServer\Tomcat
-Dcatalina.home=C:\Cisco\CVP\VXMLServer\Tomcat
-Djava.endorsed.dirs=C:\Cisco\CVP\VXMLServer\Tomcat\common\endorsed
-Djava.io.tmpdir=C:\Cisco\CVP\VXMLServer\Tomcat\temp
-DCVP_HOME=C:\Cisco\CVP
-Djava.util.logging.config.file=C:\Cisco\CVP\VXMLServer\Tomcat\conf\logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.rmi.server.hostname=10.3.3.81
-XX:NewSize=512m
-XX:MaxNewSize=512m
-Xms1024m
-Xmx1024m
-XX:+ForceTimeHighResolution
-XX:SurvivorRatio=5
-XX:TargetSurvivorRatio=90
-XX:MaxTenuringThreshold=12
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode
-XX:+CMSIncrementalPacing
-XX:+CMSParallelRemarkEnabled
-XX:+UseParNewGC
-XX:+UseTLAB
-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=C:\Cisco\CVP\VXMLServer\logs
-DAudium.Home=C:\Cisco\CVP\VXMLServer
-Dccbu.logging.config.file=log4j_vxml.xml
-Dcom.sun.management.config.file=C:\Cisco\CVP\conf\jmx_vxml.conf
-Djavax.net.ssl.keyStore=C:\Cisco\CVP\conf\security\.ormKeyStore
-Djavax.net.ssl.keyStorePassword=LkJ6CBiN[@06uDw8C@wmM4,F6RIzBtOdM7s@R.7u{0K.Yg5{
-Djavax.net.ssl.keyStoreType=JCEKS
*********************************************************************************************************************************************************
All the forums are pointing to JVM analyzer and after running MAT analyzer we find issue mostly with the following classes. Is this is a CISCO server error or do we have to look at our application?
*********************************************************************************************************************************************************
Problem Suspect 1
26,354 instances of "org.apache.catalina.session.StandardSession", loaded by "org.apache.catalina.loader.StandardClassLoader @ 0x241fc548" occupy 585,754,240 (58.97%) bytes. These instances are referenced from one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]", loaded by "<system class loader>" Keywords org.apache.catalina.session.StandardSession java.util.concurrent.ConcurrentHashMap$Segment[] org.apache.catalina.loader.StandardClassLoader @ 0x241fc548
Problem Suspect 2
27,722 instances of "org.apache.catalina.session.StandardSession", loaded by "org.apache.catalina.loader.StandardClassLoader @ 0x24215978" occupy 612,736,064 (61.69%) bytes. These instances are referenced from one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]", loaded by "<system class loader>" Keywords org.apache.catalina.session.StandardSession org.apache.catalina.loader.StandardClassLoader @ 0x24215978 java.util.concurrent.ConcurrentHashMap$Segment[]
Problem Suspect 3
The class "com.audium.server.controller.AudiumServerConfiguration", loaded by "org.apache.catalina.loader.WebappClassLoader @ 0x2461f010", occupies 357,469,984 (35.99%) bytes. The memory is accumulated in one instance of "java.util.HashMap$Entry[]" loaded by "<system class loader>". Keywords com.audium.server.controller.AudiumServerConfiguration java.util.HashMap$Entry[] org.apache.catalina.loader.WebappClassLoader @ 0x2461f010
26,354 instances of "com.audium.server.session.ControllerData", loaded by "org.apache.catalina.loader.WebappClassLoader @ 0x241f9ae0" occupy 114,140,416 (11.49%) bytes. These instances are referenced from one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]", loaded by "<system class loader>" Keywords java.util.concurrent.ConcurrentHashMap$Segment[] com.audium.server.session.ControllerData org.apache.catalina.loader.WebappClassLoader @ 0x241f9ae0
******************************************************************************************************************************************************
