背景:
2020年1月15号安全网站出了一个 CVE-2020-2546,于是按照里面的“指示”,做了T3禁用,导致weblogic服务启动失败,EBS应用不能正常登陆,启动失败日常如下:
AdminServer logs are located at /u01/DEV/app/fs1/FMW_Home/user_projects/domains/EBS_domain_DEV/servers/AdminServer/logs
01/17/20-09:13:58 :: adadminsrvctl.sh: exiting with status 1
================================================================================
01/17/20-09:24:07 :: adadminsrvctl.sh version 120.10.12020000.10
Validated the passed arguments for the option ebs-get-serverstatus
AdminServer is currently not running.
Validated the passed arguments for the option ebs-nmstart-adminsrv
Checking if the Admin Server is already up.
The Admin Server is not already up.
FMW Version is 11.1.1.9
Checking if the Node Manager is already up..
Connecting to Node Manager …
Successfully Connected to Node Manager.
The Node Manager is already up.
Starting server AdminServer …
Error Starting server AdminServer: weblogic.nodemanager.NMException: Exception while starting server ‘AdminServer’
ERROR: Unable to connect the AdminServer.
StackTrace:
java.io.IOException
at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:209)
at weblogic.management.remote.common.ClientProviderBase.newJMXConnector(ClientProviderBase.java:97)
at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnectorFactory.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
at oracle.apps.ad.util.WLUtil.initMBeanServerConnection(WLUtil.java:131)
at oracle.apps.ad.tools.configuration.EBSProvisioner.ebs_nmstart_adminsrv(EBSProvisioner.java:3682)
at oracle.apps.ad.tools.configuration.EBSProvisioner.ebs_nmstart_adminsrv(EBSProvisioner.java:3915)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at oracle.apps.ad.tools.configuration.EBSProvisioner.main(EBSProvisioner.java:8880)
Caused by: javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3://devfin.guobaojinrong.com:7001: Destination unreachable; nested exception is:
java.net.ConnectException: 拒绝连接; No available router to destination]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:40)
at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:366)
at weblogic.jndi.Environment.getContext(Environment.java:315)
at weblogic.jndi.Environment.getContext(Environment.java:285)
at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
at javax.naming.InitialContext.init(InitialContext.java:242)
at javax.naming.InitialContext.<init>(InitialContext.java:216)
at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:193)
… 11 more
Caused by: java.net.ConnectException: t3://devfin.guobaojinrong.com:7001: Destination unreachable; nested exception is:
java.net.ConnectException: 拒绝连接; No available router to destination
at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:216)
at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:170)
at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:165)
at weblogic.jndi.WLInitialContextFactoryDelegate$1.run(WLInitialContextFactoryDelegate.java:345)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:340)
… 19 more
Caused by: java.rmi.ConnectException: Destination unreachable; nested exception is:
java.net.ConnectException: 拒绝连接; No available router to destination
at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:470)
at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:321)
at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:262)
at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:199)
at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:238)
at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:200)
… 25 more
AdminServer logs are located at /u01/DEV/app/fs1/FMW_Home/user_projects/domains/EBS_domain_DEV/servers/AdminServer/logs
01/17/20-09:24:24 :: adadminsrvctl.sh: exiting with status 1
================================================================================
应用界面登陆也显示weblogic连接失败。
解决过程探索:
一般遇到这类问题,第一映像就是先恢复之前的设置,于是去找相关的表或者配置文件,从表中很难找到指向表,于是从另外一个正常的环境进入相应的weblogic主页面,并进入之前配置的页面查看帮助,于是找到了一个比较有指向性的文件(不知道具体文件在哪个目录下,通过find命令搜索),config.xml。
以下文件是来自帮助文件里面的内容:
Changes take effect after you redeploy the module or restart the server.
If this attribute configures a module that you deploy (such as an application or a JDBC data source that is part of an application) or a system resource whose configuration is saved in a descriptor file instead of in the domain’s config.xml
file (such as a JDBC data source that is scoped at the system level), the module or resource cannot process the change until you redeploy it or restart its host server. If the module is a component in an application, Oracle recommends that you redeploy the entire application to avoid complications due to intra-application dependencies.
If this attribute configures some other part of the domain (such as a server, a cluster, or an EJB container), the system cannot process the change until you restart the server or cluster.
进入相应目录后,发现这个config.xml文件有很多序列号的文件名,如config39.xml,config38.xml,看这些文件的更新时间,推测应该是每次更新都会做一次备份,于是备份当前的config.xml文件,恢复最近的一个config.xml文件,然后重启weblogic服务,重启(正常重启,登陆页面也正常出现,并可登陆)如下文本:
[appldev@devfin scripts]$ sh adadminsrvctl.sh start
You are running adadminsrvctl.sh version 120.10.12020000.10
Enter the WebLogic Admin password:
Enter the APPS Schema password:
Starting WLS Admin Server…
Refer /u01/DEV/app/fs1/inst/apps/DEV_devfin/logs/appl/admin/log/adadminsrvctl.txt for details
AdminServer logs are located at /u01/DEV/app/fs1/FMW_Home/user_projects/domains/EBS_domain_DEV/servers/AdminServer/logs
adadminsrvctl.sh: exiting with status 0
adadminsrvctl.sh: check the logfile /u01/DEV/app/fs1/inst/apps/DEV_devfin/logs/appl/admin/log/adadminsrvctl.txt for more information …
说明:ebs weblogic尽量少变动,在改动前也一定要做好备份。
相关参考文件:
https://www.oracle.com/security-alerts/cpujan2020.html
https://docs.oracle.com/cd/E23943_01/web.1111/e13707/ssl.htm#SECMG389
http://www.ijiandao.com/2b/baijia/345629.html
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=508979212187743&id=1280374.1&_afrWindowMode=0&_adf.ctrl-state=e2y1sye3v_1027