SAP: Warning – JAVA Code Cache is Full

Recently in Java – dev_serverx log file,we have seen multiple warning occurred as below –

J : [140083218769664] 01:32:31 ***Warning: Code cache is full: Try increasing the code cache size using -XX:ReservedCodeCacheSize=…
J : [140083218769664] 01:32:31 ***Warning: JIT stopped
J : [140083218769664] 01:32:31 ***Warning: JIT restarted

There is an Standard SAP note which is describing this situation & providing the multiple solutions –

2579836 – AS Java system has performance problem – FCAException – Best practices and tuning recommendations

Point No. 4 – Increase the code cache size to 1024 MB.

As per SAP, we need to increase the parameter ReservedCodeCacheSize to 1024M value but currently this parameter is not set in our system. it’s using the default 250M (Code Cache Size) value.

For Verification of the present code cache size & usage follow the below note –

2632653 – How to check present Code cache usage on SAP JVM

In my case, I am not able see Code Cache size via graphical tool then we use jvmmon tool to dump the code cache as below –

The SAP JVM Monitoring Tool jvmmon can be used to obtain the present value of the code cache. Steps are:
> Start the jvmmon
> Connect to the PID
> dump jit info
The output is written into the std_server file in the work directory. Click here for more information on the SAP JVM Monitoring Tool: jvmmon.

we did set this parameter in config tool as defined in SAP note & restart the whole cluster including DB & observe the performance of the system. it’s since more then a week, we didn’t see this Warning any more in dev_serverX log file.

Will wait for another month & see if this situation reoccurs & will update with next resolution.

SAP: JAVA System Performance Issue

Hello All,

Recently we have faced issue in our SAP Process Orchestration system where all DB connections were getting occupied after starting in few minutes.

We have identified the cause of this issue where 1 of our channel is sending thousands of messages to third party & due to delay in acknowledgment, it’s consumed all the DB process & system get hung.

We found that after even restart of application servers, DB connections were not getting destroyed automatically even though there enough DB connections are free at DB side. The issue is lying under application layer & below error is occurring Default Trace –

Exception of type com.sap.sql.log.OpenSQLException caught: No connection to data source SAPDB available. All 100 pooled connections are in use and 40 connection requests are currently waiting. The connection pool size might need to be adjusted..

Default DB Pool size = 100

There is an standard SAP note where SAP is suggesting to increase the value from 100 to 250 of respective parameter in Config Tool & restart the system. we tried the same but after restart 250 connections completely occupied & same issue was occurring.

https://launchpad.support.sap.com/#/notes/1600085

PO team identified the channel & stop the channel temporarily to avoid the issue & system back to Normal. They have cleared the stuck message & restart the channel after some time.

Now from BASIS side our responsibility is to Monitor the DB connections to avoid the hanging situation. we found below URL to monitor the DB connections & also destroy the same if requires.

http:// hostname:port number/OpenSQLMonitors

The above URL is just like ST04 of any ABAP system & it’s very useful to monitor your critical SAP JAVA system regularly to avoid any unscheduled downtime.

SAP: Digitally Signed SAP Note Support

As per SAP Blog, Jan 2020 onward download & upload process of SAP note from SNOTE transaction will stop until unless SNOTE is enabled in ABAP systems to with Digitally signed SAP Notes.

This is the secure method to download the SAP notes & all SAP note will get download only as SAR files.

Below are the SAP note which is helpful for Configuring the same.

Enable Note Assistant (SNOTE) for downloading and uploading digitally signed SAP Notes,
please implement SAP Notes 2408073, 2546220 and 2508268.

2576306 – Transport-Based Correction Instruction (TCI), it contains the SAP Notes 2408073, 2546220 and 2508268. If the SNOTE in your ABAP system is already enabled for TCI, It is recommended to implement only TCI SAP Note 2576306 instead of applying the above individual SAP Notes.

You can download the instructions from Support Portal with Subject – SAP Note Transport-Based Correction Instructions.

SAP: What is SAPCPE & Usage of SAPCPE

SAPCPE is an program which exists in your Kernel folder. it helps to copy kernel binaries from one location to another exe location.

Whenever we run the “STARTSAP” command, SAPCPE copy or verify the kernel binaries from “CDEXE” location to Instance/exe. it depends whether how much instances you have including ASCS/SCS, ERS & Gateway.

When we perform Kernel Upgrade activity we update the kernel patch level on “CDEXE” location & rest other location are having the old version of kernel. if we want to update the kernel patch level in Instance EXE before starting the system then we can perform the same via SAPCPE manually.

For doing this, we need to find the respective instance profile & run the below command.

Example –

sapcpe pf=/usr/sap//SYS/profile/Instance_Profile

Once you trigger the above command it copies the required binaries from CDEXE location to respective profile instance directory/exe & update the kernel.

How it Works Internally

There are some LST files exists in the kernel directory(CDEXE) where it read the LST file to copy the same. we have separate LST file for Instance, ERS, ASCS/SCS & Gateways. These LST files are mentioned in the respective profiles.

For Example –

sapcpe pf=/usr/sap//SYS/profile/Instance_Profile

When you run the above command SAPCPE read the Instance_Profile & find out the LST file mentioned in profile & based on this LST file, it transfer the respective binaries only from CDEXE to Instance Directory/exe.

The same procedure also happen when you trigger the STARTSAP.

I hope this clarifies the SAPCPE usage.

SAP – IGS failing with RFC External Failure

Recently we have faced issue where IGS stopped working after BW system restart. As our BW system is connected to Portal & it’s throwing error while generating reports.

We have checked that IGS RFC is working fine in SM59 – TCP/IP but when we run the report in SA38 – GRAPHICS_IGS_ADMIN. it’s failing with error “RFC_EXTERNAL_FAILURE”.

By going through the multiple log files, we identify that it’s failing as IGS is not able to bind the DEFAULT Port.

Port 4NN01 and 4NN02 where NN is the instance number

Due to this it’s not connecting to IGS server. We followed SAP note – 2719952 to resolve the issue.

We change the IGS default port to customize & restart the system.

Example as below –

igs/listener/http = 40085
igs/mux/port = 40005
igs/pw/1 = 40006,7
igs/pw/2 = 40008,9

By changing above now IGS is listening to different Port & RFC & Reports are working fine.
Hope this Helps.

SAP : JAVA Transport failing with Internal Server Error HTTP Code 500

Recently we are facing a issue where all BPM transport import is failing in PO JAVA system with below error –

WS Execution Error in CDTDEPLOY_DI (cx_ai_system_fault)
SOAP:1007 SRT: Unsupported xstream found: (HTTP Code 500 : Internal Server Error)
Error when processing Web Service Call

Connectivity for PO Java system is as below –

Solution Manager ——–> NWDS ———> SAP PO (JAVA)

Transport import from Solution Manager (CTS+)

The above transport is failing on Pre Prod system but it’s successfully completing in QA system so with same configuration & same NWDS system.

We followed SAP note – 1139406 but it didn’t help us there was no configuration change in the system because it’s deploying successfully in QA. We did restart PO multiple times but that also not help us.

we again trigger the transport & saw the below error in Developer Trace while transport is running.

ClientServerException exception:Write object <> failed! Caused by: com.sap.engine.frame.core.locking.LockException:Cannot lock 14051311214586500000pwdf5674……………….18228850,J2EE_CONFIGURATION_MANAGER,22352127/cb1c722f/30c4e431/18064/2dca72/6f336575/573c0b09/abe4c6a3/8f4414e3…]; it is in use by another owner. The lock collision occurred with user .

There is an standard SAP note for the same issue – 2023974 – Write object failed – Cannot lock as it is in use by another owner.

We followed the SAP note as it’s saying to delete the JAVA locks from PO system & re import, after deleting locks from PO system we again restart the PO & NWDS system again & then import the same, now this time it’s successfully imported in the system.

Reference note for JAVA system performance
2579836 – AS Java system has performance problem – FCAException – Best practices and tuning recommendations

SAP:Cannot import instance profile via RZ10

Hello,

Recently after system we are facing below issue while importing the Instance Profile from SAP in RZ10 Tcode.

While multiple restart & also changed the profile authorization to 777 that also doesn’t work.

There is an SAP Note – 2157878 which describe the solution for this error, however in our case that we need to change profile parameter in START profile(if start profile is separate then Instance) & also change the entry in /usr/sap/sapservices file.

Correct Entry
DIR_PROFILE = $(DIR_INSTALL)$(DIR_SEP)profile

Adjust /usr/sap/sapservices, change /sapmnt//profile/ to /usr/sap//SYS/profile/

After updating above 2 entries & then restart the application server by killing the SAPSTARSRV process, that resolve the issue & we are able to import the profile in RZ10 successfully.

Hope this Helps.

SAP – RSLOGSYSDEST Table update

Recently after BDLS we have seen in our BW system that some of the logical system entries in table – RSLOGSYSDEST didn’t change to QAS/Pre Prod & still pointing to production.

BW team was facing some issue & request us to change the table entries manually to QAS/Pre Prod.

As we have already closed the client to – No changes allowed, it’s not allowing us to change directly in table from SE16.

we have found another way to change the same from SE37 – Execute Function.

RSAR_UPDATE_RSLOGSYSDEST

This Function will update the table as per our requirement by giving the new value.