标签归档:weblogic

Oracle EBS weblogic 禁用筛选器T3导致weblogic服务不能重启问题

背景:
2020年1月15号安全网站出了一个 CVE-2020-2546,于是按照里面的“指示”,做了T3禁用,导致weblogic服务启动失败,EBS应用不能正常登陆,启动失败日常如下:

AdminServer logs are located at /u01/DEV/app/fs1/FMW_Home/user_projects/domains/EBS_domain_DEV/servers/AdminServer/logs

01/17/20-09:13:58 :: adadminsrvctl.sh: exiting with status 1

================================================================================

01/17/20-09:24:07 :: adadminsrvctl.sh version 120.10.12020000.10
Validated the passed arguments for the option ebs-get-serverstatus
AdminServer is currently not running.

Validated the passed arguments for the option ebs-nmstart-adminsrv
Checking if the Admin Server is already up.
The Admin Server is not already up.
FMW Version is 11.1.1.9
Checking if the Node Manager is already up..
Connecting to Node Manager …
Successfully Connected to Node Manager.

The Node Manager is already up.

Starting server AdminServer …
Error Starting server AdminServer: weblogic.nodemanager.NMException: Exception while starting server ‘AdminServer’

ERROR: Unable to connect the AdminServer.

StackTrace:
java.io.IOException
at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:209)
at weblogic.management.remote.common.ClientProviderBase.newJMXConnector(ClientProviderBase.java:97)
at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnectorFactory.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
at oracle.apps.ad.util.WLUtil.initMBeanServerConnection(WLUtil.java:131)
at oracle.apps.ad.tools.configuration.EBSProvisioner.ebs_nmstart_adminsrv(EBSProvisioner.java:3682)
at oracle.apps.ad.tools.configuration.EBSProvisioner.ebs_nmstart_adminsrv(EBSProvisioner.java:3915)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at oracle.apps.ad.tools.configuration.EBSProvisioner.main(EBSProvisioner.java:8880)
Caused by: javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3://devfin.guobaojinrong.com:7001: Destination unreachable; nested exception is:
java.net.ConnectException: 拒绝连接; No available router to destination]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:40)
at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:366)
at weblogic.jndi.Environment.getContext(Environment.java:315)
at weblogic.jndi.Environment.getContext(Environment.java:285)
at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
at javax.naming.InitialContext.init(InitialContext.java:242)
at javax.naming.InitialContext.<init>(InitialContext.java:216)
at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:193)
… 11 more
Caused by: java.net.ConnectException: t3://devfin.guobaojinrong.com:7001: Destination unreachable; nested exception is:
java.net.ConnectException: 拒绝连接; No available router to destination
at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:216)
at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:170)
at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:165)
at weblogic.jndi.WLInitialContextFactoryDelegate$1.run(WLInitialContextFactoryDelegate.java:345)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:340)
… 19 more
Caused by: java.rmi.ConnectException: Destination unreachable; nested exception is:
java.net.ConnectException: 拒绝连接; No available router to destination
at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:470)
at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:321)
at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:262)
at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:199)
at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:238)
at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:200)
… 25 more

AdminServer logs are located at /u01/DEV/app/fs1/FMW_Home/user_projects/domains/EBS_domain_DEV/servers/AdminServer/logs

01/17/20-09:24:24 :: adadminsrvctl.sh: exiting with status 1

================================================================================

应用界面登陆也显示weblogic连接失败。

 

解决过程探索:

一般遇到这类问题,第一映像就是先恢复之前的设置,于是去找相关的表或者配置文件,从表中很难找到指向表,于是从另外一个正常的环境进入相应的weblogic主页面,并进入之前配置的页面查看帮助,于是找到了一个比较有指向性的文件(不知道具体文件在哪个目录下,通过find命令搜索),config.xml。

以下文件是来自帮助文件里面的内容:

Changes take effect after you redeploy the module or restart the server.

If this attribute configures a module that you deploy (such as an application or a JDBC data source that is part of an application) or a system resource whose configuration is saved in a descriptor file instead of in the domain’s config.xml file (such as a JDBC data source that is scoped at the system level), the module or resource cannot process the change until you redeploy it or restart its host server. If the module is a component in an application, Oracle recommends that you redeploy the entire application to avoid complications due to intra-application dependencies.

If this attribute configures some other part of the domain (such as a server, a cluster, or an EJB container), the system cannot process the change until you restart the server or cluster.

进入相应目录后,发现这个config.xml文件有很多序列号的文件名,如config39.xml,config38.xml,看这些文件的更新时间,推测应该是每次更新都会做一次备份,于是备份当前的config.xml文件,恢复最近的一个config.xml文件,然后重启weblogic服务,重启(正常重启,登陆页面也正常出现,并可登陆)如下文本:

[appldev@devfin scripts]$ sh adadminsrvctl.sh start

You are running adadminsrvctl.sh version 120.10.12020000.10

Enter the WebLogic Admin password:
Enter the APPS Schema password:
Starting WLS Admin Server…
Refer /u01/DEV/app/fs1/inst/apps/DEV_devfin/logs/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at /u01/DEV/app/fs1/FMW_Home/user_projects/domains/EBS_domain_DEV/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile /u01/DEV/app/fs1/inst/apps/DEV_devfin/logs/appl/admin/log/adadminsrvctl.txt for more information …

说明:ebs weblogic尽量少变动,在改动前也一定要做好备份。

相关参考文件:
https://www.oracle.com/security-alerts/cpujan2020.html

https://docs.oracle.com/cd/E23943_01/web.1111/e13707/ssl.htm#SECMG389

http://www.ijiandao.com/2b/baijia/345629.html

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=508979212187743&id=1280374.1&_afrWindowMode=0&_adf.ctrl-state=e2y1sye3v_1027

如何查询Oracle EBS Weblogic访问地址及相关扩展知识

1. Oracle EBS Weblogic访问地址固定格式

http://xxxx:yyyy/console

xxxx为Oracle EBS域名地址

yyyy为Weblogic访问端口

/console为固定字符

2. 如何获取访问端口号

通过操作系统应用用户(比如appluat)执行如下命令:

grep wls_adminport $CONTEXT_FILE

如下图所示例子:

3. 访问最终地址

http://xxxx:7005/console

扩展,如何找Oracle EBS相关端口:

1. 通过操作系统应用用户执行命令,找到CONTEXT_FILE所在地址:

echo $CONTEXT_FILE

2. 直接cat /u01/uat/app/fs2/inst/apps/UAT_erp01/appl/admin/UAT_erp01.xml文件,或者通过ftp工具下载目标文件UAT_erp01,找出需要的端口名称(也可直接查看到端口)

3. 如果在平时知道端口名字,直接通过以下命令查询即可(如最上图所示):

grep web_port $CONTEXT_FILE    –ebs应用访问端口

grep wls_adminport $CONTEXT_FILE    –ebs weblogic访问端口

grep dbport $CONTEXT_FILE  –ebs 数据库端口

3. R12.2weblogic版本查询

/u01/test/app/fs1/FMW_Home
grep -i "Weblogic Server" registry.xml

4. R12.1ias版本查询

cd $IAS_ORACLE_HOME/Apache/Apache/bin

./httpd -version

Patch Set Update (PSU) Release Listing for Oracle WebLogic Server (WLS) (文档 ID 1470197.1)

Critical Patch Update (CPU) Program July 2019 Patch Availability Document (PAD) (文档 ID 2534806.1)