OIM upgrade from 11G R2 PS1 to 11G R2 PS3 (11.1.2.1.0 - 11.1.2.3.0)

The posts to be followed will elaborate on the path to be followed for updating to OIM 11GR2PS3 version for your existing system (from 11.1.2.x.x). 
There are simply too many tasks involved, so I decided it'll be better to break it up to shorter steps, each focusing on a single task and each to be followed in the same sequence. 

Do not use this posts if the OIM deployment was done with Oracle LCM tools. 

ROADMAP : 

STEP 1 - Pre-Upgrade tasks :  

1.a : Take complete backups of the servers and all the database schema , this upgrade do not come with a roll-back process, so make sure to have backups of the MW_HOME, Domain_Home, and the DB schema. 
1.b :  Check the system and product compatibilities 
Read more about it here
note  : make sure all the relevant OS packages are present, upgrade jdk version if required, also pay attention separaetly to the OUI , RCU , and DB requirements. 

1.c : Run and verify the pre-upgrade report utility 
Read more about it here  : Generating the Pre-Upgrade Report

1.d : Download the softwares : 
OIM 11GR2PS3 : Patch 20996201    
Latest OIM Bundle patch  : Doc ID 2031368.1[at the moment, it is patch 29347961] (16-aor-2019)
SOA Suite 11g Patch Set 7 (11.1.1.9) : Patch 20995651
SOA Suite 11g Bundle Patches : Doc ID 1641787.1
Weblogic Server 10.3.6 : Patch 13529623
RCU 11.1.1.9 : Patch 20996068
JAVA 1.7.0_80 : JAVA Archive
Oracle Identity Manager 11gR2 PS3 (11.1.2.3) Upgrade Advisor ( Doc ID 2002373.2 )

STEP 2 - Update OPatch : 

Oracle says  'OPatch used must be an 11.1.0.x patch of version 11.1.0.10.3 or higher to complete successfully.  
Testing in lab with the current version available under Patch 6880880 as of March 2016 leads to successful upgrade'.

To correctly update OPatch utility and update the Central Inventory with this information, the following steps are needed(Doc ID 2046456.1).

1. Download the 11.1.0.0.0 version of Patch 6880880 and save it to the $ORACLE_HOME.
2. Rename the existing OPatch directory to OPatch.orig, or if preferred remove it.
3. Unzip the newer OPatch archive file directly in the ORACLE_HOME. This creates a new OPatch directory.
4. Change directory to this new OPatch directory. Example: cd OPatch
5. New Step: Run the command ./opatch util UpdateOPatchVersion

6. Run the command ./opatch version and ensure it is the version expected.

STEP 3 - Upgrade weblogic to 10.3.6.0 :

In this environment, the webLogic was already in the required version, hence I skipped the step, but if required, you can always refer to  :
Upgrading to WebLogic Server 10.3.6.0

STEP 4 - Upgrade SOA binaries to 11.1.1.9.0 (to both SOA_HOST1 and SOA_HOST2) : 

• check the SOA version  : oracle_SOA home> bin > soaversion.sh

Make sure all Edit sessions in SOA Composer are committed or reverted.

Log in to the SOA Composer (/soa/composer) as any user with active sessions.
Go to Open > My Edits to view all of your active sessions.
Open each document listed in "My Edits" and do one of the following:

1. Click Commit to commit the session changes.
2. Select Revert > Clear all session edits and saved changes to abort the changes.

Stop the soa managed servers. 
CD to Disk1 and ./runInstaller , if prompted , provide the java_home if it's not defined in env  or mention in the command itself ./runInstaller -jreloc java_home. 

Provide your middleware home and soa oracle home when prompted for, and complete the following steps. 
once done, set Oracle home env variable to soa oracle home and check with ./opatch lsinventory , the soa version should be 11.1.1.9.0. 

STEP 5 - Upgrade OIM binaries to  (11.1.2.3.0)(to both OIM_HOST1 and OIM_HOST2) : 

Stop the oim managed servers.
Follow the steps same as mentioned in step 3 , and copy the new binaries to oim oracle home. 

check the installer logs at the following location:
On UNIX: ORACLE_INVENTORY_LOCATION/log
To find the location of the Oracle Inventory directory on UNIX, check the file ORACLE_HOME/oraInst.loc.

NOTES : Oracle highly recommends to apply patch 24615124 at this point, prior to doing the Schema upgrade. This addresses large tables issues.

STEP 6 - Upgrade existing OIM and SOA schema or create new BI Publisher and OPSS schema : 

OIM PS3 comes with embedded BI Publisher , so it's important we create the BIPUBLISHER and MDS schema for the new BIP . Follow the step 2 here  :
http://bitoshok-das.blogspot.com/2019/09/bi-publisher-111190-standalone.html

Once the BIP schemas are created , update the below existing schemas with Patch-Set assistant(PSA)  :

Oracle Platform Security Services (OPSS) schema
Metadata Services (MDS) schema
Oracle Identity Manager (OIM) schema
ORASDPM schema
SOA Infrastructure (SOAINFRA) schema 

Check the schema versions pre and post update to verify  :
SELECT OWNER, VERSION, STATUS, UPGRADED FROM SCHEMA_VERSION_REGISTRY where owner=<SCHEMA_NAME>;

run the PSA : cd to <MW_HOME>/oracle_common/bin > ./psa

Screen	Description
Welcome	This page introduces you to the Patch Set Assistant.
Select Component	Select the component you wish to upgrade.
Prerequisite	Verify that you have satisfied the database prerequisites.
Schema	Specify your database credentials to connect to your database, then select the schema you want to update.
	Note that this screen appears once for each schema that must be updated as a result of the component you selected on the Select Component screen.
Examine	This page displays the status of the Patch Set Assistant as it examines each component schema. Verify that your schemas have a "successful" indicator in the Status column.
Upgrade Summary	Verify that the schemas are the ones you want to upgrade.
Upgrade Progress	This screen shows the progress of the schema upgrade.
Upgrade Success	Once the upgrade is successful, you get this screen.

Middle tier upgrade is performed using the OIMUpgrade.sh utility. Oracle Identity Manager middle tier upgrade is carried out in two stages and both are mandate :

1. Middle tier upgrade offline : This is the first stage where OIMUpgrade.sh is run in offline mode, that is, with the Administration Server and the Managed Server(s) in shutdown state.

2. Middle tier upgrade online : This is the second stage where OIMUpgrade.sh is run in online mode, that is with the Administration Server and the SOA Managed Server(s) in running state, OIM and BIP managed server in shutdown stage.

STEP 7 - Upgrade Oracle Identity Manager middle tier in offline mode on OIM_HOST1 : 

7.a  : Creating a Truststore for Upgrading SSL Enabled Middleware

• To create a truststore, complete the following steps:
• Export the public certificate from the identity store for each server, and place all of them in a single directory.
• Import all of the public certificates to a single truststore.
• Copy the truststore to a location accessible by upgrade script.
• Specify the truststore location and type for the properties wls.trustStore.loc and wls.trustStore.type respectively, when updating the properties file.

7.b : Updating the Properties File :

Upfate the oim_upgrade_input.properties file loacted under ORACLE_OIM_HOME/server/bin/ directory as explained here :
Parameters to be specified in the Properties File

7.c : run the OIMUpgrade utility in offline mode
Make sure that you have stopped the WebLogic Administration Server, the Oracle Identity Manager Managed Server(s), and the ALL the Managed Server(s).

cd to OIM_ORACLE_HOME/server/bin >  ./OIMUpgrade.sh offline

Provide the OIM, MDS, SOA, BIP, OPSS schema passwords. 

Check the HTML reports generated at ORACLE_HOME/server/upgrade/logs/MT/oimUpgradeReportDir_offline.

Check the logs files generated at ORACLE_HOME/server/upgrade/logs/MT/ to verify if the middle tier offline upgrade was successful.

-----------------------------------------------------------------
If you do NOT have high availability oim set up with multi node, you may jump to step 8, if not, execute the following steps  : 

Replicate the domain configuration on OIM_HOST2 by packing the upgraded domain on OIM_HOST1 and unpacking it on OIM_HOST2 :

cd to $MW_HOME/oracle_common/common/bin

on oim_server1
sh pack.sh -domain=<Location_of_OIM_domain> -template=<Location_where_domain_configuration_jar_to_be_created> -template_name="OIM Domain" -managed=true


Copy the jar created on oim_server2 : 
sh unpack.sh -domain=<Location_of_OIM_domain> -template=<Location_on_OIM_HOST2_where _you_copied_jar_file_created_by_pack_command> -overwrite_domain=true

After you unpack the domain, copy the content of the following directory on OIM_HOST1 to the same directory on OIM_HOST2:
DOMAIN_HOME/soa/autodeploy
----------------------------------------------------------------

STEP 8 - Upgrade Oracle Identity Manager middle tier in online mode ONLY on OIM_HOST1 : 

start the WebLogic Administration Server and the SOA Managed Server(s)

cd to OIM_ORACLE_HOME/server/bin > ./OIMUpgrade.sh online

Check the HTML reports generated at ORACLE_HOME/server/upgrade/logs/MT/oimUpgradeReportDir_online

After you upgrade the Oracle Identity Manager middle tier online, you must start the Oracle Identity Manager Managed Server (s) and the BIP Server.
-----------------------------------------------------------------
Before starting the servers, you must add the following property below the JAVA_PROPERTIES entry in the 

DOMAIN_HOME/bin/setDomainEnv.sh, to ignore hostname verification:
-Dweblogic.security.SSL.ignoreHostnameVerification=true

When we start the Managed Servers for the first time after middle tier upgrade, the servers must be connected to the non-SSL Administration Server port. To do this, complete the following steps:

Before we start the Managed Servers, enable the non-SSL port for the Administration Server :

Ensure that the Managed Servers connect to the non-SSL admin port while starting. For example, if managed server is started using startManagedWebLogic.sh script, update the ADMIN_URL in this script to use the non SSL url.
These changes can be reverted back once the servers are up.
------------------------------------------------------------------

8.a: Changing the Deployment Order of Oracle Identity Manager EAR

1. Log in to the WebLogic Administration console and Click Deployments on the left pane.

2. Click oim.ear and update the deployment order from 47 to 48.

NOTES : Oracle highly recommends to apply the latest bundle patch (patch 29347961 at the moment),  

• Stop the Admin Server, all Oracle Identity Manager managed servers, and all SOA managed servers
• Download the patch p29347961_111230_Generic.zip file, unzip it, set oracle-home env var to Oraclie_OIM_Home, cd to Oraclie_OIM_Home/OPatch and apply the patch 

        ./opatch apply  ../29347961/iamsuite/patch/oim/29551893

If OPatch fails with error code 104, cannot find a valid oraInst.loc file to locate Central Inventory, include the -invPtrLoc argument, as follows:
opatch apply -invPtrLoc ORACLE_HOME/oraInst.loc

• After patch is applied, Oracle Identity Manager server staging directories must be deleted. To do so,

remove cache/*, stage/*, and tmp/* directories 
cd to DOMAIN_HOME/servers/OIM_SERVER_NAME
rm -Rf cache/* stage/* tmp/*

• cd to OIM_ORACLE_HOME/server/bin/ directory, and enter the details provided in the patch_oim_wls.profile file. 

Make sure that the WebLogic Admin Server, SOA Managed Servers, and Oracle Identity Manager Managed Server are running.

Set the following environment variables:
setenv PATH $JAVA_HOME/bin:$PATH

• Execute patch_oim_wls.sh to apply the configuration changes to the Oracle Identity Manager server. We must run the script in a shell environment using the following command:

          sh patch_oim_wls.sh

Note: This script needs to be run on each OIM managed nodes while the servers are up.

Delete the following directory in domain home:
IDM_DOMAIN/servers/OIM_SERVER_NAME/tmp/_WL_user/oracle.iam.console.identity.self-service.ear_V2.0

To verify that the patch_oim_wls script has completed successfully, check the OIM_HOME/server/bin/patch_oim_wls.log log file.

Stop and start WebLogic Admin Server, SOA Servers, and Oracle Identity Manager Servers.
------------------------------------------------------------------
Note :
On running the patch_oim_wls script, the $DOMAIN_HOME/servers/MANAGED_SERVER/security/boot.properties file might be deleted. If you use a script to start the Managed Server and use the boot.properties file to eliminate the need of entering the password in the script, then create a new boot.properties file.

Ignore the following exception traces in the patch_oim_wls.log file:
[java] Aug 11, 2015 3:45:28 AM oracle.jdbc.driver.OracleDriver registerMBeans
     [java] WARNING: Error while registering Oracle JDBC Diagnosability MBean.
     [java] java.security.AccessControlException: access denied (javax.management.MBeanTrustPermission register)
     [java] at java.security.AccessControlContext.checkPermission(AccessControlContext.java:374)
------------------------------------------------------------------

STEP 9 - Upgrade OIM Design console :

cd to OIM_ORACLE_HOME/bin >  ./config.sh and on the components to configure screen, select only "OIM Design Console"
Provide OIM server hosts and port names and complete the next steps. 

• Once done,run the following command to build the wlfullclient.jar file:

java -jar <MW_HOME>/modules/com.bea.core.jarbuilder_1.7.0.0.jar

Copy the wlfullclient.jar file to the <IAM_HOME> where you installed the Design Console. For example:
cp wlfullclient.jar <Oracle_IDM>/designconsole/ext

If the Design Console is SSL enabled, do the following :

1. Copy the webserviceclient+ssl.jar file from the directory WL_HOME/server/lib/ to the directory ORACLE_HOME/designconsole/ext/.
2. Copy the cryptoj.jar file from the directory MW_HOME/modules/ to the directory ORACLE_HOME/designconsole/ext/.

If DESIGN_CONSOLE_HOME/config/xl.policy does not contain the default grant policy for all, then add the following permission for cryptoj.jar at the end of the xl.policy file:

grant codeBase "file:DIRECTORY_PATH_TO_cryptoj.jar"{permission java.security.AllPermission;};

Open the xlclient.sh file (located at XLDC_HOME/xlclient.sh, and add the following argument to the java command:
-DAPPSERVER_TYPE=wls

Restore the xlclient.sh , xlconfig.xml files. 

STEP 10 - Post-Upgrade tasks :

Please refer below for all the post-upgrade activities, not all of them may be relevant for your environment though :
OIM Post-Upgrade Tasks

STEP 11 - Verify if the upgrade is successful or not by:

checking through ./opatch lsinventory
checking if the sefl-service portal and sysadmin portals are up and running.
checking if the BI Publisher is up and running on :
http://bip_host:bip_port/xmlpserver 

STEP 12 : 

Grab one iced-cofee and enjoy ! you earned it . 

As always, thank you for reading.