Relinking TNSLSNR fails - undefined reference to `snprintf@GLIBC_2.4'

Lsnrctl File Zero Bytes After Upgrade To 10.2.0.4.0

Applying 10.2.0.4.0 PATCHSET gives the following error :-

During the relink phase for tnslsnr executable you are receiving the bellow error:

INFO: gcc -m64 -o tnslsnr -L/opt/oracle/product/10.2/db/network/lib/ -L/opt/oracle/product/10.2/db/lib/ -L/opt/oracle/product/10.2/db/lib//stubs /opt/oracle/product/10.2/db/network/lib/s0nsgl.o /opt/oracle/product/10.2/d..... /opt/oracle/product/10.2/db/lib/sysliblist`


INFO: /lib64/libcrypt.so.1: undefined reference to `snprintf@GLIBC_2.4'
collect2: ld returned 1 exit status


INFO: make: *** [tnslsnr] Error 1

Solution to this problem is that
1. Install the Oracle 10.2.0.4 patchset and ignore this relink error
2. Take a backup of the file $ORACLE_HOME/network/lib/env_network.mk

$ cp $ORACLE_HOME/network/lib/env_network.mk $ORACLE_HOME/network/lib/env_network.mk.orig3. open for edit the file $ORACLE_HOME/network/lib/env_network.mk and identify the following line:

TNSLSNR_LINKLINE+=$(SYSLIBS)
4. add above the line TNSLSNR_LINKLINE+=$(SYSLIBS) the following entry:

LLIBCRYPT = -lcrypt 5. modify the entry TNSLSNR_LINKLINE+=$(SYSLIBS) as bellow:

from
        TNSLSNR_LINKLINE+=$(SYSLIBS)
to
        TNSLSNR_LINKLINE+=$(SYSLIBS) $(LLIBCRYPT)

Regenerate the lsnrctl file using the following procedure

$cd $ORACLE_HOME/network/lib
$make -f ins_net_server.mk ilsnrctl

$cd $ORACLE_HOME/bin
$ls -la lsnrctl Make sure that lsnrclt file is regenerated.

Cheers!!!

Patching scenarios in Oracle Applications

Lots of DBAs having issues while applying patches to the Oracle Applications. Before going to apply any patches to the applications, we have to analyse the downtime in general. Best way to do patching to the system is that:
1. Clone to a TEST instance from the current working Production system. Try your patching on this TEST instance and see how it works.
So you could easily identify any issues while applying those patches to the system. Also will get a general idea of downtime. Always keep your TEST instance as much as identical to your production system.

2. Read carefully the patch README file, if any doubts check with Oracle by logging a SR. Also ask the experts from Oracle forum and they will explain to you everything based on that patch.

3. For the application patches, requires to enable maintenance mode before applying any patch.

Sometimes, you may use "options=hotpatch" to apply the patch online without enabling maintenance mode but all its depends on the following conditions
 a. What is the patch
 b. Size of the patch, and
 c. What are the modules it is going to affect.

Always requires to enable maintenance mode for the following
 a. All patchsets
 b. Family packs
 c. Minipacks, and
 d. AD/FND patches

But for small patches which only need to replace a package with a new one you can apply the patch online.

As I told you earlier, always best practice is that, try your patching on a TEST instance first and see how it works.

In Windows systems, you have to shutdown the application services before relinking executable files, otherwise the patch will fail and you will have to run it again after stopping the services.


Cheers!!!

Running AD Merge Patch - ADMRGPCH

ADMRGPCH is a utility to merge two or more patches into a single merged patch.

This executable is located in AD_TOP/bin

To run AD Merge Patch with the following arguments

$ Admrgpch [source directory] [destination directory]

Source directory - Where patches to merge have been unloaded.

Destination directory – Where the unified patch will be created.


It reads the all drivers and Readme files and merge into a single driver file

Actions in the merge patch are grouped by product then by patch number.

AD merge patch creates the merged patch drivcer files and copies the actual files needed by the merged patches into the destination directory.

Log file located in the current directory where we run the admgrpch. The default log file name is admrgpch.log.

To specify the name of the merged patch drivers, run AD merge patch with
$ admrgpch -s [source directory] -d [destination directory] -merge_name [name]
eg:-
$admrgpch -s /u01/patches/source -d /u01/patches/destination -merge_name NLS20.
The merged patch drivers will be named cNLS20.drv, dNLS20.drv, and gNLS20.drv

We can merge only same releases, same platforms or same parallel modes and if they are not incompatable patches.


Cheers!!!

EBS Release 12.1 RUP3 for AD/ATG/HRMS Released on My Oracle Suppport

Oracle announced the availability of 12.1 RUP3 for the following products along with corresponding iHelp patches on My Oracle Support:

R12.AD.B.Delta.3 (Patch 9239089)
R12.ATG_PF.B.Delta.3 (Patch 8919491)
R12.TXK.B.Delta.3 (Patch 8919489)
R12.HR_PF.B.Delta.3 (Patch 9114911)
Oracle iSetup (AZ) Metadata Patch for 12.1.3 Release Update Pack (Patch 8919537)

iHelp patch and Install document details
Following are the details on the iHelp patches and My Oracle Support document number for these patch sets:

AD iHelp Patch 9399683 - Document 1077769.1
ATG iHelp Patch 9400185 - Document 1066312.1
HRMS iHelp Patch 9401789 - Document 1081427.1

Cheers

Forms Runtime Diagnostic - Steps to take FRD trace in 11.5.10 & R12

In 11i

1.   Login to your Instance Unix session as Apps.
2.   Login to System Administrator Responsibility in Oracle Applications.
3.   Add the following profile to the USER LEVEL as follows
ICX: Forms Launcher = http://vis11i.apps.local:8090/dev60cgi/f60cgi?&record=collect&log=testing.log

4.    Logout and Log back in again, for this profile to take effect. When you login again, a message box pops up, which alerts you that Diagnostics is enabled.
5.    Reproduce the Forms-Issue.
6.    Exit the application.
7.    Locate the file in the directory set by variable $FORMS_TRACE_PATH.

In R12

1. Login to your Instance Unix session as Apps.
2.Make the USER value for profile option 'ICX: Forms Launcher' the same as the SITE value.
3. Append the USER value of ICX: Forms Launcher with the "?record=collect" and save this at the user level.
http://visr12.apps.local:8091/forms/frmservlet?record=collect

4. Logout and Log back in again, for this profile to take effect. When you login again, a message box pops up, which alerts you that Diagnostics is enabled.
5. Go to Help->About Oracle Applications, go to the section 'Forms Server Environment Variables'
There you should see a value in FORMS_TRACE_DIR. Note down the value.
6. Reproduce the Forms-Issue.  
7. Locate the file in the directory noted in FORMS_TRACE_DIR.
Note: By default, this directories named as forms_[pid].trc where pid is the process identifier.
Collect _[form_process_id]

Cheers!!!

HOME button on 11i Navigator is not working

Yesterday I  have been applied 11i.ATG_PF.H.delta.7 patch to one of my instance, and completed successfully without having any errors.
I can able to login and do access the application as usual. But when I click the Returning To Homepage Using Link Home, got error like
 "Error Page
You have encountered an unexpected error. Please contact the System Administrator for assistance."

When I have change my Self Service Personal Home Page mode from "Personal Home Page with Framework" to "Framework Only", everything work fine for me. Still I don't know the real cause of this issue.

Login as system administrator -> System Profile Values ->

Change profile option APPLICATIONS_HOME_PAGE (Self Service Personal Home Page mode) to 'Framework Only'

Logout and login again

Its working....
Cheers!!!

11i.ATG_PF.H.delta.7 (Patch:6241631) fails on icxwtab.odf

When attempting to apply 11i.ATG_PF.H.delta.7 (Patch:6241631) a worker fails on icxwtab.odf.

The adpatch worker log file shows

AD Worker error:
The index cannot be created as the table has duplicate keys.


Use the following SQL statement to identify the duplicate keys:


SELECT TRANSACTION_ID, count(*)
FROM ICX.ICX_TRANSACTIONS
GROUP BY TRANSACTION_ID
HAVING count(*)>1


AD Worker error:
Unable to compare or correct tables or indexes or keys
because of the error above

The solution is to run the "Purge Inactive Sessions" concurrent program. However it is not possible to start the concurrent managers as you are in the middle of a failed patch.

As an immediate solution, run $ICX_TOP/sql/ICXDLTMP.sql scipt as the APPS user.

However this scripts with the error

When followed the steps, getting the following ERROR

SQL> @ICXDLTMP.sql
FND_SIGNON.audit_user_end(session_rec.login_id, session_rec.last_connect + nvl(session_rec.limit_time,4)/24);
*
ERROR at line 18:
ORA-06550: line 18, column 16:
PLS-00302: component 'AUDIT_USER_END' must be declared
ORA-06550: line 18, column 5:
PL/SQL: Statement ignored

The reason for this error is that,  $ICX_TOP/sql/ICXDLTMP.sql script has been updated to 11i.ATG_PF.H.delta.7 levels during the copy driver of patch:6241631. However the D driver has not updated the underlying tables yet.

To implement the solution, please execute the following steps:

1. Run the backup copy of ICXDLTMP.sql as the APPS user from the 11i.ATG_PF.H.delta.7 patch backup directory
sqlplus apps/apps_pwd @/$PATCH_TOP/6241631/backup/[SID]/[HostName]/icx/sql/ICXDLTMP.sql
2. Restart the failed worker

Installing JDK Version 6 with Oracle E-Business Suite 11i

Version of EBS 11.5.10.2
OS : OEL 4.6

Download JDK 6u2 or higher from http://java.sun.com/javase/downloads/index.jsp

Check the file size before proceeding.
jdk-6u-linux-i586-rpm.bin

You can download to any directory that you can write to in Linux box
Login as root and perform these actions
# chmod a+x jdk-6u-linux-i586-rpm.bin
# ./jdk-6u-linux-i586-rpm.bin

This will install JDK 6 under /usr/java/jdk1.6.0_16/ folder 
So remember this path, require to apply it in application tier [JDK60_TOP]
 
You can verify the new Java 6 
/usr/java/jdk1.6.0_16/bin/java -version
java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) Server VM (build 14.2-b01, mixed mode)

Now need to login as Applmgr user and make sure you have access permission to this folder

[applmgr@visr12 ~]$ /usr/java/jdk1.6.0_16/bin/java -version

java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) Server VM (build 14.2-b01, mixed mode)
[applmgr@visr12 ~]$

These messages will confirm that you have access to that executables
OK, fine. Now we can start looking at the required patches for application tiers

Obtain the JDK 6 Consolidated Patch and other EBS Interoperability Patch(es)
Obtain the following patches and their prerequisites for Oracle Applications 11.5.10CU2:

1. JDK 6 consolidated patch for 11.5.10CU2, patch 5622511
2. Contracts Core patch 5977502

Apply 11i.ATG_PF.H.RUP5 or Higher (if required)
(Skip this step if you have previously applied ATG_PF.H.RUP5 or higher or are running an 11.5.10.CU2)

Stop all Application Tier Server Processes

cd $COMMON_TOP/admin/scripts/CONTEXT_NAME
adstpall.sh APPS username/APPS password

Apply 11i.ATG_PF.H.RUP5 patch 5473858 or higher using AutoPatch.

Apply JDK 6 Consolidated Patch 5622511 and other EBS Interoperability Patch(es)
Apply the following patches and their prerequisites, using AutoPatch:

JDK 6 consolidated patch 5622511
Contracts Core patch 5977502

Configure all Application Tier Nodes to Support JDK 6

Run the JDK 6 upgrade script to upgrade the configuration template files for each [APPL_TOP]. Enter the appropriate commands and follow the instructions given by the script:

txkrun.pl -script=SetJDKCfg -contextfile=$CONTEXT_FILE -runautoconfig=Yes -appspass= -jdktop=[JDK60_TOP]
Here [JDK60_TOP] = /usr/java/jdk1.6.0_16/

$COMMON_TOP/admin/scripts/ adautocfg.sh

Regenerate the appsborg2.zip File & Product JAR Files

$APPL_TOP/APPS[CONTEXT_NAME].env

Run AD Administration, -> " Generate Applications Files" ->"Generate Product JAR Files"

Restart all Application Tier Server Processes

cd $COMMON_TOP/admin/scripts/[CONTEXT_NAME]

adstrtal.sh [APPS username/APPS password]

Verify the Application Tier Node Configurations

Run the AOL/J test page from a web browser using the following URL:-

http://[web server]:[port]/OA_HTML/jsp/fnd/aoljtest.jsp

Verify Configurations on all Application Tier Nodes
$ADJVAPRG -version
echo $CLASSPATH

$AFJVAPRG -version
echo $AF_CLASSPATH

That's it.

Please read Metalink Note:401561.1 - Using J2SE Version 6 with Oracle E-Business Suite 11i and
JavaTM SE 6 Release Notes - JDK RPM Installation for Linux (32-bit)

New Apps 11i AutoConfig + Templates Rollup Patch U Now Available

Today two important announcements from Mr.Steven Chan about

New Apps 11i AutoConfig + Templates Rollup Patch U Now Available

This new Autoconfig+Template Rollup includes the following features and enhancements:
1. Applications password validation while using adcmctl and in other various
commands.
2. Mini AutoConfig: it is possible now to run AutoConfig using a customized driver.
3. AutoConfig performance improvements.
4. adchkcfg enhancements to remove false diffs/false positives
5. Database 11gR2 support.
6. Oracle HTTP Server (OHS) related changes for security and refining of aliases, log names changed as per standards
7. Addition of ANTISAMY run-time rehosted JAR files.
8. OHS performance improvements by removing loading of unwanted modules.
9.For DB tier on Solaris 64-bit platform, use of 64-bit JVM during CVM phase of AutoConfig.
10. Support for DB listener name customization.
11. Support of Forms and Reports in the Icelandic language.
12. Addition of LOCAL_LISTENER parameter to init.ora template.
13. Introduction of FORMS60_RECORD_GROUP_MAX to restrict the number of records fetched by a List of Values (LOV).
14. New context variables: s_httpd_pls_pid_file, s_icx_language, s_forms60_record_group_max, s_db_listener.

Warning for E-Business Suite Issues with Sun JRE 1.6.0_21.

According Mr.Steven Chang, at this stage, better to remain on JRE 1.6.0_17; and wait for JRE 1.6.0_22

How to remove an OA Framework Personalization

Using this method, you can remove personalization from an OAF page.

1) Login to Oracle Applications as SYSADMIN

2) Select Functional Administrator Responsibility -> Personalization

3) Fill in the criteria to locate the OA Framework page and press Go
    Tip: Check 'Personalized' checkbox to only query the pages having a personalization

4) Identify the correct page from the results returned

5) Depending on the situation you have following options:
  
Cause of failure is known (e.g. last change made) and can be removed

Click pencil "Personalize Page" and confirm personalization levels.
Revert the personalization step causing the problem
Apply the changes
Retest if the page is now accessible again.

Cause of the failure is unknown  or can not be simply removed

Click pencil "Manage Personalizations"
Now Deactivate or Delete the personalization document.

The difference between Deactivate and Delete is that in the case of Deactivating the personalization document in the MDS repository is not removed, while this is the case for Delete. Especially when there are multiple personalization documents, but it's not clear which causes the problem the 'Deactivate' allows these to be tested individually without the need to reload or recreate the personalization documents.

Get the source file of any 'Pl/Sql Stored Procedure' on current program

All of us knows how to find-out the source of a stored procedure in database using TOAD.

Using SQL*Plus, to retrieve the sources, please do the following :

1. From the Define concurrent Program forms, query your PL/SQL concurrent program then note Executable Name  .

2. Go to the Define Executable forms, query the executable name of step 1 then note the Execution File Name.

An executable file name is generally formatted as :

Package_name.Procedure

3. To retrieve the source, connect to apps then run the following sql statement from SQL*Plus :

set long 20000
set pages 9999
set lines 80
select text from user_source
where name =''
order by line
/

Cheers!!!

How to enable and collect debug for HTTP, OC4J and OPMN - R12

When we used to send debug details to oracle support for troubleshooting, we have to perform the following  for enable and collect debug information from HTTP, OC4J and OPMN services of R12.

Run the following script on all middle tiers as the application file system owner. A file named after the servername and date will be created in every server under the /tmp directory. Upload the resulting zip file from every server to your service request on Metalink.

zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_iAS_CONFIG.zip \ 
$ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf \ 
$ORA_CONFIG_HOME/10.1.3/config \ 
$INST_TOP/pids/10.1.3/Apache \ 
$ORA_CONFIG_HOME/10.1.3/j2ee/ \ 
$ORA_CONFIG_HOME/10.1.3/javacache/admin \ 
$ORA_CONFIG_HOME/10.1.3/network/admin \ 
$ORA_CONFIG_HOME/10.1.3/opmn

First of all, need to shutdown the http server, oc4j and opmn services, for that:
cd $ADMIN_SCRIPTS_HOME
$adapcctl.sh stop
$adoacorectl.sh stop
$adformsctl.sh stop
$adoafmctl.sh stop
$adopmnctl.sh stop

Then need to take the backup existing log files for safe custody.
$LOG_HOME/ora/10.1.3/Apache
$LOG_HOME/ora/10.1.3/j2ee
$LOG_HOME/ora/10.1.3/opmn

1. Enable http ODL logging 

Edit httpd.conf file, add the following to the end of file $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf

OraLogMode oracle
OraLogSeverity TRACE:32
OraLogDir $LOG_HOME/ora/10.1.3/Apache/oracle
Please use the full path to $LOG_HOME e.g.
OraLogDir = /u01/inst/apps/JCB_atg/logs/ora/10.1.3/Apache/oracle

Warning: the log.xml file created by the http ODL log can get very large.
Diligence must be taken to monitor this file and maintain its size under the 2GB limit which when exceeded can cause issues with login

2. Increase OC4J logging for oacore

edit j2ee-logging.xml adjust the following in file:
$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/config/j2ee-logging.xml

<logger name='oracle' level='TRACE:32' useParentHandlers='false'>

3. Increase OC4J logging for forms

edit j2ee-logging.xml adjust the following in file:
$ORA_CONFIG_HOME/10.1.3/j2ee/forms/config/j2ee-logging.xml

<logger name='oracle' level='TRACE:32' useParentHandlers='false'>

edit orion-web.xml adjust the following in file:
$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/application-deployments/oacore/html/orion-web.xml

<param-name>debug_mode</param-name>
<param-value>true</param-value>

4. Increase OPMN Logging

edit opmn.xml adjust the following in file:
$ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml

<debug path="/u01/inst/apps/JB_atg/logs/ora/10.1.3/opmn/opmn.dbg" comp=" internal;ons;pm" rotation-size="1500000" rotation-hour="0" />

Startup the http server, oc4j and opmn

Reproduce the issue

Access the following urls and inform support if the url was successful or unsuccessful:

http://<server>.<domain>:<PORT>/OA_HTML/ServletPing
http://<server>.<domain>:<PORT>/OA_HTML/OA.jsp?OAFunc=OAHOMEPAGE
http://<server>.<domain>:<PORT>
http://<server>.<domain>:<PORT>/OA_HTML/jsp/fnd/aoljtest.jsp
http://<server>.<domain>:<PORT>/OA_MEDIA/FNDLOGOL.gif
http://<server>.<domain>:<PORT>/forms/frmservlet
http://<hostname.domainname>:<port>/OA_HTML/fndvald.jsp?username=sysadmin&password=<sysadmin_password>

Run this inventory script and upload the html output

$ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl -script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP -outfile=$APPLTMP/Report_Inventory.html

Shutdown the http server, oc4j and opmn

Run these commands to zip up the required log files:

zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_iAS_HTTP.zip $LOG_HOME/ora/10.1.3/Apache/*log* $LOG_HOME/ora/10.1.3/Apache/*lock* $LOG_HOME/ora/10.1.3/Apache/oracle $INST_TOP/pids/10.1.3/Apache
zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_iAS_OC4J.zip $LOG_HOME/ora/10.1.3/j2ee/ $LOG_HOME/appl/rgf/*log*
zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_iAS_OPMN.zip $LOG_HOME/appl/admin/log/ $LOG_HOME/ora/10.1.3/opmn/

The commands will create zip files in the /tmp directory named after the server name and date. Please now run the following script to compile the 3 zip files into 1 zip. Upload the resulting zip file to the associated service request.

zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_iAS_Debug.zip /tmp/*zip*

Finally, revert the changes and re-start the services

R12 Error 'Function not available to this responsibility' When Trying to Access Custom Form

After Upurade from 11i to R12 users cannot access the custom form in R12 which was working fine in 11.5.10.2.
Following errors getting whenever access custom form
"Function not available to this responsibility. Change Responsibilities or contact your System Administrator".

The reason for this issue is that, missing entry for CUSTOM_TOP under default.env file. In R12, CUSTOM_TOP should be defined under default.env in $INST_TOP/ora/10.1.2/forms/server directory.

Solution

1. Login to server with APPLMGR manager user access.

2. Go to $INST_TOP/ora/10.1.2/forms/server directory.

3. Ensure that your CUSTOM_TOP's are registered in the default.env file.

This means that, you should create an entry for environment variable CUSTOM_TOP (which contains physical path to your custom directory) like below in default.env file present under location $INST_TOP/ora/10.1.2/forms/server directory.

For Example:
APPL_TOP=/home/applmgr/PROD/apps/apps_st/appl
CUSTOM_TOP=/home/applmgr/PROD/apps/apps_st/custom

4. Restart the middle tier services.

5. Retest the issue.

Cheers!

appsborg2.zip does not exist in target after cloning

When attempting to clone using rapidclone the appsborg2.zip does not exist anymore in the target CLASSPATH. Due to the missing appsborg2.zip no workflow process are started.

When using jdk 1.3, 1.4 or 1.5 the appsborg2.zip is included in the AF_CLASSPATH. As jdk 1.6 is not in the list it will assume jdk 1.1.8 is used and excludes appsborg2.zip resulting in a incorrect AF_CLASSPATH.

To implement the solution, please execute the following steps:

1. Use the workaround to change the CLASSPATH after cloning to include the appsborg2.zip, use OAM to
    change the CLASSPATH

2. Run autoconfig after changing the CLASSPATH

3. Retest the issue

Cheers!!!

How to change and which Apps database users passwords can be changed in a Multi-Node Apps Installation

In 11i oracle applications, the following list shows which database users should be changed and which should not be changed

APPLSYS -  Y
APPLSYSPUB -  N
APPS -  Y
CTXSYS -  Y
DBSNMP -  Y
PORTAL30 - Y
PORTAL30_SSO - Y
PRODUCT_SCHEMAS (AMB_XTR) - Y
SYS - Y
SYSTEM - Y

The following database users passwords may be changed

MDSYS
ORDPLUGINS
ORDSYS
PORTAL30_DEMO
PORTAL30_PUBLIC
PORTAL30_SSO_PS
PORTAL30_SSO_PUBLIC

The passwords of APPS and APPLSYS must be identical.
So you just need to change APPLSYS password not APPS , by changing APPLSYS
password , APPS password assumes the same value

Shutdown the Applications Tier services on each of the Nodes
Database Tier Services must be up and running

To change the APPLSYS password:

FNDCPASS apps/apps_schema_password 0 Y system/system_schema_password SYSTEM APPLSYS new_password
Ex: FNDCPASS apps/apps 0 Y system/manager SYSTEM APPLSYS WELCOME123

To change an Oracle user password:

FNDCPASS apps/apps_schema_password 0 Y system/system_schema_password

ORACLE apps_database_user new_password
Ex: FNDCPASS apps/apps 0 Y system/manager ORACLE GL GL_123

Run Autoconfig on each of the Nodes wherever Application Tier is installed

Bring up the Application Tier services on each of the Nodes

Fnd_cant_insert_user_role when assigning a responsibility in R12

Whenever assign user responsibilities, next time user responsibilities are queried they are no longer in the form (FNDSCAUS) they are not visible.

A second attempt to assign the same responsibility produces the error
Oracle Error -20001: ORA-20001: FND_CANT_INSERT_USER_ROLE has been detected in 
FND_USER_RESP_GROUPS_API.INSERT_ASSIGNMENT

The cause of the issue is that there are no subscribers defined for AQ WF_BPEL_Q

Solution
1. Drop / recreate the subscriber as follows:

declare
lagent sys.aq$_agent;
subscriber_exist exception;
pragma EXCEPTION_INIT(subscriber_exist, -24034);
begin
lagent := sys.aq$_agent('WF_BPEL_Q',null,0);
dbms_aqadm.add_subscriber(queue_name
=>APPS.WF_BPEL_Q',subscriber=>lagent,rule=>'1=1');
exception
when subscriber_exist then
-- just add the rule if subscriber exists
dbms_aqadm.alter_subscriber(queue_name =>'APPS.WF_BPEL_Q',
subscriber=>lagent,
rule=>'1=1');
end;
/

2. Run the concurrent request Workflow Directory Services User/Role Validation.

1.1 "Workflow Directory Services User/Role Validation" Parameters : 100000, Yes, No, No"
1.2 "Workflow Directory Services User/Role Validation" Parameters : 100000, No, Yes, No"

3. Bounce Apache

Also read Note:549040.1 - How Standard Business Events Work With WF_BPEL_Q?

Cheers!!!

There are no patches that need to be applied from the patchset Oracle Application Server PatchSet 10.1.3.5.0

When I have tried with upgrade iAS to patchset 10.1.3.5.0 in my cloned R12 instance, getting the following error.

"There are no patches that need to be applied from the patchset Oracle Application Server PatchSet 10.1.3.5.0"

The reason for this error is that the Global inventory and Local inventory data of this instance is corrupt. Hence the patchset 'runInstaller' command couldn't find the correct inventory information of 1013_ORACLE_HOME on this instance, then got the above error message.

Solution to this issue is that
1. Confirm there is the instance inventory folder :
/.../[sid]/inst/apps/[sid]_[hostname]/admin/oraInventory

2. Create oraInst.loc file under 1013_ORACLE_HOME. Edit oraInst.loc to add the following line and save it:
inventory_loc=/.../[sid]/inst/apps/[sid]_[hostname]/admin/oraInventory

3. Since there is the valid and correct Local Inventory under 1013_ORACLE_HOME, invoke the 10.1.3.5.0 patchset runInstaller program by running the command:
$ ./runInstaller -invPtrLoc [1013_ORACLE_HOME]/oraInst.loc

Cheers!!!

Autoconfig Error on Appstier Tier: Context Value Management Failed

After clone, while running adautocfg.sh getting the following error.

Context Value Management will now update the Context file


Updating Context file...COMPLETED


Attempting upload of Context file and templates to database...Exception 
in thread "main" java.lang.AbstractMethodError: oracle.sql.LnxLibThin.lnxnuc([BILjava/lang/String;)
at oracle.sql.NUMBER.toInt(NUMBER.java:413) 
at oracle.jdbc.dbaccess.DBConversion.NumberBytesToInt(DBConversion.java: 2959) 
at oracle.jdbc.driver.OracleStatement.getIntValue(OracleStatement.java:4583) 
at oracle.jdbc.driver.OracleCallableStatement.getInt(OracleCallableStatement.java:635) 
at oracle.apps.ad.autoconfig.oam.InDbCtxFile.uploadCtx(InDbCtxFile.java: 216) 
at oracle.apps.ad.autoconfig.oam.CtxSynchronizer.uploadToDb(CtxSynchronizer.java:328) 
at oracle.apps.ad.tools.configuration.FileSysDBCtxMerge.updateDBCtx(FileSysDBCtxMerge.java:678) 
at oracle.apps.ad.tools.configuration.FileSysDBCtxMerge.updateDBFiles(FileSysDBCtxMerge.java:222) 
at oracle.apps.ad.context.CtxValueMgt.processCtxFile(CtxValueMgt.java:1586) 
at oracle.apps.ad.context.CtxValueMgt.main(CtxValueMgt.java:665) 
Error returned from java. Context Value Management was aborted 
ERROR: Context Value Management Failed. 
Terminate. 


The logfile for this session is located at: 
%APPL_TOP%\admin\%CONTEXT_NAME%\log\\adconfig.log 

The reason for this error is that JAVA_TOP/oracle/sql contain the following two files
LnxLibThin.class, 
LdxLibThin.class 
Need to remove and re-run autoconfig again.

As part of the 11.5.10 Maintenance Pack, these files should have been removed by adpatch
After successfully removed these two files, need to re-run the autoconfig from Applmgr user

Cheers!!!

Proxy user functionality of R12

In R12, there are new features/functionalities introduced in User management (UMX), and one of the best functionality is Proxy User.

• This functionality provides the delegator the ability to grant/revoke the proxy privilege to individuals.
• Provide a mechanism throughout the application’s framework where the user can access the proxy switcher feature 
• Provide a mechanism throughout the application’s framework which indicates to the user that they are acting as a proxy 
• Provide the ability to track the delegate’s actions within the system, while the delegate is acting on behalf of the delegator (Audit).
• Provide a mechanism throughout the application’s framework where the user can access the proxy switcher feature.

How this functionality is working in R12

Assume that one user (OPERATIONS) going on vacation and need to delegate responsibilities to perform transactions on behalf of the user (OPERATIONS) to another user (AJIKUMAR) in Vision R12 Instance.

The following steps are involved to perform this task

• Connect to instance as SYSADMIN and switch  responsibility as “User Management” and open “Users” form.
• Search for “Operations” user

   

• Assign the role “Manage Proxies” to this user.

     

• Login as “OPERATIONS” user
• Add user as Manage Proxy using Preference -> Manage Proxies -> Add People
• This will allow adding the usernames of those users that wish to perform action on behalf of OPERATIONS user. Now add username (AJIKUMAR) that wish to assume operations user responsibilities.

• Now login as AJIKUMAR
• Note the new options “Switch user” and “Return to Self” available

• If you want to perform any transactions on behalf of “Operations”, simply click the “Switch user” menu.

• Now you are in the Operations User Responsibilities page and the message at the top is that “Logged as AJIKUMAR Proxy for OPERATION”

Cheers!!!

How to change default layout of XML publisher concurrent request is always PDF

Suppose one required Excel outpout from an XML Publisher concurrent request, need to change from output format option button from Request form.

A list of the available Format Types on the Submit Request form:

•EXCEL
•FO
•HTML
•PDF
•RTF

Using this Submit Request form, the Layout field of the Upon Completion region displays the currently selected template. To change the template, template language, or output format select the Options button.
If you are using the HTML-based Schedule Request interface, select the template and output format from the Layout page of the process.

The XML Publisher development team improved this functionality, which allows the layout template creator to specify a default output format at template level by applying the following patches to your instance.

Oracle E-Business Suite Release 11i:

FNDRSRUN.fmb 115.185 or higher - patch 7627832 or patch 6241631 11i.ATG_PF.H.delta.7
xdotables.ldt 115.12 or higher - patch 5612820 or patch 6241631 11i.ATG_PF.H.delta.7

Oracle E-Business Suite Release 12.0.x:

FNDRSRUN.fmb 120.29.12000000.20 or higher - patch 7017250:R12.FND.A
xdotables.ldt 120.4.12000000.2 - patch 5612820:R12.XDO.A

Oracle E-Business Suite Release 12.1:

FNDRSRUN.fmb 120.44.12010000.2 or higher - patch 9228359:R12.FND.B
xdotables.ldt 120.4.12010000.3 or higher - patch

The patches corrects the Template output type issue that allows a default output type to be set in the template definition which overrides the original format type of the template. Additionally, a profile option, FND: Default Template Output Type, can be set to override the original format type, only if the default type is not set in the template definition.

Cheers!!!

How to enable debugging for Concurrent Managers

Sometimes concurrent managers debug information is requires to solve issues related with CM, for achieve this purpose we have to do the following

Steps for enable DIAG for the ICM
you have to modify adcmctl.sh script using vi editor

vi $COMMON_TOP/admin/scripts/SID_HOSTNAME/adcmctl.sh
Replace line :
CONTEXT_PARAMS="diag=N wait=N"
With :
CONTEXT_PARAMS="diag=Y wait=N"
Save file.

Steps for enable the Debug Service

Login with System Administrator responsibility
Navigate to Concurrent / Manager /Define
Query manager with name "Debug Service"
Tick the "Enabled" checkbox if not yet ticked
save

Once completed these two steps, required to bounce the concurrent managers
cd $COMMON_TOP/admin/scripts/SID_HOSTNAME
adcmctl.sh stop apps/password

wait for all your concurrent process to complete, you can monitor using
ps -ef|grep -i fndlibr

All CM process completed, start your CM

adcmctl.sh start apps/password

Please don't forget to switch it off after the debug information has been generated.

Cheers!!!

What happening when executing adcfgclone.pl in DB Tier as well as Apps Tier?

adcfgclone.pl - This script will performs the clone configuration in the target machine.

During this configuration phase will reconfigure the copied instance after gathering the required information.

adchkutl.sh
The first step performed is the OS utility check (adchkutl.sh).
This ensures make, cc, ld and ar executables are available in the PATH.

CloneContext.class
This will create the Target Context file from the values entered on the screen.

All the questions, values entered, results of the port availability check etc will be logged in a log file located in same location as the adcfgclone.pl script:-
dbTier : $ORACLE_HOME/appsutil/clone/bin
appsTier(s) : $COMMON_TOP/clone/bin

Log files for this phase:-
dbTier: $TMP/CloneContext_xxxxxx.log

appsTier(s): $TMP/CloneContext_xxxxxx.log

Both Tiers: $TMP/adclonectx.err
Executing adcfgclone.pl will generate logs as follows:-

     dbTier: $ORACLE_HOME/appsutil/clone/bin/CloneContext_xxxxxx.log
     appsTier(s): $COMMON_TOP/clone/bin/CloneContext_xxxxxx.log
     Both Tiers: $TMP/adclonectx.err

If any of the variables in the Target system Context file appear to be incorrect, then check this log file to see how the incorrect value arose.

ApplyDBTier.class/ApplyAppsTier.class

Once the XML is created, or passed as a parameter on the command line, then the configuration phase begins.

This part will instantiate all the templates created during the preparation phase under:
$ORACLE_HOME/appsutil/template.

Templates to instantiate are located in driver file
$ORACLE_HOME/appsutil/driver/instconf.drv

Log files for this phase:-
dbTier: $ORACLE_HOME/appsutil/log/[Target Context Name]/ApplyDBTier_xxxxxx.log
appsTier(s): $APPL_TOP/admin/[Target Context Name]/log/ApplyAppsTier_xxxxxx.log

Relink the ORACLE_HOME(s)
The executables in the ORACLE_HOME(s) will be relinked during the configuration phase.
The commands used, and results of the relinking will displayed in log file :
$ORACLE_HOME/install/make.log

Create the Control files (DB Tier only)
Once the templates are instantiated, and the ORACLE_HOME is relinked, then the control files can be recreated.

The templates used here are located in
$ORACLE_HOME/appsutil/clone/data/stage/adcrdb.zip
The files in this zip file are used to create a SQL file called
adcrdbclone.sql

This above SQL script is called from a shell script
adcrdb.sh.

Both of the above scripts can be found under
$ORACLE_HOME/appsutil/install/$CONTEXT_NAME

If there are any problems creating the control files, then check the log files :
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/adcrdb_SID.txt
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_xxxxxx.log

AutoConfig
The next phase will run AutoConfig using use the usual templates and drivers associated with AutoConfig on db or appsTier.

The following log files will contain the information on the results of running AutoConfig
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_xxxxxx.log
$APPL_TOP/admin/$CONTEXT_NAME/log/ApplyAppsTier_xxxxxx.log

Update the oraInventory (OracleHomeCloner.class)
Once the Target instance is configured, it will update the Global Inventory will the locations of the new ORACLE_HOMES

Any problems with this step will be detailed in the log files:-
[path to oraInventory]/logs/OracleHomeCloner_MMDDHHMM.log
[path to oraInventory]/logs/apps[Year]-[Mon]-[Day]-[Hour]-[Min]-[Sec]-[AM/PM].log

What happening when executing adpreclone.pl in DB and Apps Tiers

What happening when executing adpreclone.pl in DB and Apps Tier?

adpreclone.pl - This is the preparation phase, will collects information about the source system, creates a cloning stage area, and generates templates and drivers. All of these are to reconfigure the instance on a Target machine.
Preclone will do the following:

Convert Symbolic links
All symbolic links pointing to a static path will be converted into relative paths

Create templates
Any files under the $ORACLE_HOME that contain system specific information, will be replicated and converted into a template. These templates are placed into the $ORACLE_HOME/appsutil/template directory.

Create driver(s)
A driver file, relating to these new templates is created called instconf.drv. This contains a list of all the templates and their locations, and the destination configuration files that these templates will create.
This driver file is called instconf.drv and is placed into directory
$ORACLE_HOME/appsutil/driver

Create Stage area
A clone stage is created containing the required java code and scripts to reconfigure the instace on the Target machine

Rapid Clone stage area:
dbTier : $ORACLE_HOME/appsutil/clone
appsTier(s) - $COMMON_TOP/clone

The stage area(s) consist of the following directories:-
 jre used to run the java code on the Target machine.
 bin contains the RapidClone scripts that can be run on the Target machine:-

•    adclone.pl is the main cloning script
•    adcfgclone.pl is used to configure the Target system, this calls adclone.pl
•    adclonectx.pl is used to clone a Source XML file manually
•    adaddnode.pl is used to add a new node to the Patch History tables
•    adchkutl.sh checks for existence of require O/S utils, cc, make, ar and ld

 jlib contains all the Rapid Clone java code, jdbc libraries etc
 context contains templates used for a Target XML file
 data (Database Tier only) contains the driver file, and templates used to generate the control file SQL script
 adcrdb.zip contains the template and list of datafiles on the Source
 addbhomsrc.xml contains information on the datafile mount points of the Source
 appl (Applications Tier only) this is used when merging appltops, i.e Multi-node to Single node cloning

Executing adpreclone.pl will create a log file:-
Rapid Clone:
dbTier : $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDBTier_xxxxxx.log
appsTier : $APPL_TOP/admin/$CONTEXT_NAME/log/StageAppsTier_xxxxxx.log

Once this adpreclone.pl step has been completed successfully, all the java .class files under the following directories should be identical to those under $JAVA_TOP/oracle :

RDBMS $ORACLE_HOME/appsutil/java/oracle
RDBMS $ORACLE_HOME/appsutil/clone/jlib/java/oracle
$COMMON_TOP/clone/jlib/java/oracle

Next post, What happening when executing adcfgclone.pl in DB Tier as well as Apps Tier?

How to restart a Terminated worker - AD Controller

Actually, when patches are applying to application, there are situation like restart a terminated worker.
To do so, required these steps to achieve
1. Start AD controller
2. Review worker status
3. Take the appropriate action for each status
  a. If the worker shows Failed, restart the failed job
  b. If the worker shows running or Restarted status, but the process is not really running, tell manager that a worker has failed its job. Then, tell manager to start the worker

If you terminated a hanging worker process, you must manually restart the workder. In this scenario, complete these steps
1. Start AD Controller
2. Review worker status
3. Take the appropriate action for each worker status:
 If the worker shows failed, restart the failed job. When prompted, enter the number of the worker that failed.
 If the worker shows Running or Restarted status, but the process is not really running, select the following options

• Option 4: Tell Manager that a worker has failed its job. When prompted, enter the number of the hanging worker
• Option 6: Restart a worker on the current machine. When prompted, enter the number of the worker that failed

Do not tell the manager to start a worker that has shutdown if the worker process is running. Doing so will create duplicate worker processes with the same worker ID and will cause incorrect results

Hope, this will clear to understand the process
Cheers!!!

After Clone Concurrent Managers not coming up

When cloned my instance last time, I have an issue with concurrent manager not coming up using
Finally decided to stop the concurrent manager using
$COMMON_TOP/admin/scripts/HOST_NAME
. adcmctl.sh stop apps/password

When I have checked the ICM logfile there is an error entry as

Could not initialize the Service Manager FNDSM_HOSTNAME_SID. Verify that HOSTNAME has been registered for concurrent processing. 

The cause of this issue was FND_NODES table having incorrect node entries (old node from the Source Instance) and that no Service Manager was found for the existing node.

To solve this problem, need to perform the below steps

1.  Stop all middle tier services including the concurrent managers.
2.  Stop the database.
3.  Start the database.
4.  Connect SQLPLUS as APPS user and run the following :

EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;

5.  Run AutoConfig on all tiers, firstly on the DB tier and then the APPS tiers and webtiers to repopulate the required system tables.
6.  Connect to SQLPLUS as APPS user and run the following statement :

select CONCURRENT_QUEUE_NAME from FND_CONCURRENT_QUEUES where CONCURRENT_QUEUE_NAME like 'FNDSM%';

If the above SQL does not returning any value then please do the following:

Go to $FND_TOP/patch/115/sql

Connect SQLPLUS as APPS user and run the following script :

SQL> @afdcm037.sql;

This script will create libraries for FNDSM and create managers for preexisting nodes.

Check again that FNDSM entries now exist:

select CONCURRENT_QUEUE_NAME from FND_CONCURRENT_QUEUES where CONCURRENT_QUEUE_NAME like 'FNDSM%';

7.  Go to cd $FND_TOP/bin
$ adrelink.sh force=y "fnd FNDLIBR"
$ adrelink.sh force=y "fnd FNDSM"
$ adrelink.sh force=y "fnd FNDFS"
$ adrelink.sh force=y "fnd FNDCRM"

8.  Run the CMCLEAN.SQL script from the referenced note below (don't forget to commit).

9.  Start the middle tier services including your concurrent manager.

Now cloned instance concurrent managers became active. This will applicable to both 11i and R12 instances.

Is it mandatory to run all services in Middle Tiers for a Multi-node R12 Installation?

The concept of Multi-Node installation of the E-Business-Suite Release 12 using Rapid Install is changed a lot compared with 11i. Oracle have introducted a new feature "Edit Services" Using this feature, we can change the required services available on each application node, however after the install is complete. The noticable change is that application tiers are configured to run all services rather than those specified during rapid install.

Lets see why are all application tier nodes configured to run all services ?

Suppose we have 2 Application Tiers
APPS1 (Admin- and Concurrent Manager- Services)
APPS2 (Forms- and Web Services)

During the Rapid Install, selected the above configuration via the "Edit Services" button as follows :

APPS1 : enable "Root Service Group", "Batch Processing Services"
APPS2 : enable "Root Service Group", "Web Entry Point Services", "Web Application Services", "Other Service Group"

The logic behind these service classification is as follows :

1.Root Service Group (Runs services on AS, 10.1.3 OH)
  Oracle Process Manager (OPM) : adopmnctl.sh

2.Web Entry Point Services (Runs services on AS, 10.1.3 OH)
  HTTP Server : adapcctl.sh

3.Web Application Services (Runs services on AS, 10.1.3 OH)
  OACORE OC4J : adoacorectl.sh
  FORMS OC4J : adformsctl.sh
  OAFM OC4J : adoafmctl.sh

4.Batch Processing Services (Concurrent Managers and Apps Listener)
  APPS TNS Listener : adalnctl.sh
  Concurrent Managers : adcmctl.sh, adsvcm.sh, ieoicsm.sh, ieosvicsm.sh
  Fulfillment Server : jtffmctl.sh, jtfsvfm.sh

5.Other Service Group: (Runs services for Forms on 10.1.2)
  Oracle Forms Server : adformsrvctl.sh
  Oracle MWA Service : mwactlwrpr.sh
After the installation you see that the Autoconfig XML file is generated with the following entries for APPS1 and APPS2 :

s_isDB=NO
s_isAdmin=YES
s_isForms=YES
s_isConc=YES
s_isWeb=YES

The reason for enabled all services on the two application tier nodes instead of the configuration that was selected via the "Edit Services" feature is that:

When installing R12 with multiple nodes. all the nodes are now set as 'Y' in FND_NODES.

This occurs because in R12, concept of unified APPL_TOP is introduced which means everything is laid down on all servers.

From the APPL_TOP perspective, all the Servers on a Multi-Node Environment will have the same files and can now potentially start any Service if needed.  In some cases, additional configuration will be required before this can be done since there can be profiles, etc associated with each Server.

For R12, the only difference between the Servers, are the Services that have been activated on each Node.
The Services are identified by the variables on the /service_group/ section in the APPS Context File:

 Root Service Group : s_root_status
 Web Entry Point Services : s_web_entry_status
 Web Application Services : s_web_applications_status
 Batch Processing Services : s_batch_status
 Other Service Group : s_other_service_group_status

Depending on the value of these variables (enabled or disabled), adstrtal.sh / adstpall.sh will only start/stop the Services associated with them, ignoring the rest.

For example, if a node has only /s_batch_status/ "enabled" and the rest of the services are disabled, when you run adstrtall.sh on that Server and it will only start the Concurrent Managers and the TNS Listener for Apps.

Cheers !!!

FRM-92130, FRM-92120 Registry.dat missing occurs while access the applicaitons

When the user attempting to run the form :

FRM-92120 REGISTRY FILE: \ORACLE\FORMS\REGISTRY\REGISTRY.DAT IS MISSING
or
FRM-92130 Registry is either missing, or contains invalid, entry for Dispatcher.
Full Details: FRM-92030: Internal Error: Failed to find name of class for handlerClassId=1. Missing in registry?

The cause of this issue is that, the file Registry.dat file has not been accessed by the client machine because of the Jinitiator does not find the proper proxy / firewall through which the client is authenticated to access the files from the application server. There is a  line feed / special characters that were causing the error.

To solve this issue do the following at the client PC

1) Open the Jintiator control panel through "Start-->Settings-->Control Panel--> Jintiator 1.3.1.x
2) Go to Proxies tab
3) Uncheck "Use Browser Settings" check box
4) Enter the proxy details
5) Apply changes

or
Change the following setting from:
In IE -> Tools->Connections-> Lan settings-> Exceptions
On NS -> Edit -> Preferences -> Advanced -> Proxies -> No proxy for
eg:

visr12.apps;
visr12.test;
visr12.apps.local

To:

visr12.apps; visr12.test; visr12.apps.local

This removes the line feed / special characters that were causing the error.

Audit - responsibilities and Data in Oracle Applications

There two levels of auditing in Oracle applications,

1. System usage
2. Data itself, its called auditTrail.

We can enable system usage auditing from
system administrator responsibility ->system profile -> Sign-On:Audit Level. Sign-On:Audit Level

Set this option can be site, responsibility, user level and can be done at the form, responsibility and the user level.

Depends on exiting scenario of the organization, set this value.

If decide to audit a responsibility, then need to decide at what level: form, responsibility or user.

If audit the responsibility at the form level, then will get the most details about who used the responsibility, what forms they accessed using this
responsibility, when they accessed the responsibility and when they left the responsibility.

If auditing the responsibility is done at the responsibility level, will not get the forms that the responsibility accessed but you will get everything else.

If audit the responsibility at the user level, will get the user that accessed the responsibility, when they started and when they stopped. But this is very limited data.

The second level of auditing is for the data itself. This auditing is called AuditTrail and is more involved in setup. Here need to identify the data want to audit and then translate that to the specific table and column for later tracking. Based on these data, can create Audit groups for auditing of these columns and tables.

Some of the useful reports for the Sign-On:Audit Level found fat System Administrator responsibility:

1. Signon Audit Concurrent Requests
2. Signon Audit Forms
3. Signon Audit Responsibilities
4. Signon Audit Unsuccessful Logins
5. Signon Audit Users

Cheers !!!

Patch Terminology in R12

Always I have confusion with oracle patch terminology in 11i, but in R12 have introduced new way of patch terminology and very easy to follow. Letme start with Releases first,

A release update pack increments the third section of the release number, for example to 12.1.1 or 12.1.2, and consists of a cumulative aggregation of product family release update packs created across Oracle E-Business Suite since the initial (base) release.

Oracle E-Business Suite Release 12 includes numerous products, all have abbreviated names or two or more usually three letters. These short names are used in special patch names. Common examples include Applications Technology (ATG), Business Intelligence (BIS), Financials (FIN), Human Resources (HR), and Supply Chain Management (SCM).

A product family release update pack is an aggregation of patches created for a specific product family after the initial release, and designated by the release number followed by the product short name and a letter. For example, in 'R12.ATG_PF.A.DELTA.6', '12' is the major release number, 'ATG_PF.A' means 'Applications Technology Product Family pack A', and 'delta.6' means this is the sixth rollup patch on top of Applications Technology Product Family pack

A codelevel is essentially a rollup patch with a name that looks like a version number. Higher codelevels include more changes, and include all fixes from lower codelevels. For example, R12.AD.A was the initial release of AD in Release 12. R12.AD.A.1 was a higher codelevel, which included all code in R12.AD.A plus additional fixes; R12.AD.A.2 was a higher codelevel again, which included all code in R12.AD.A.1, plus further fixes.

Hope this will help you too

Cheers!