My Oracle Support Banner

Spacewalk Service Failed To Start After OS Upgrade To OL7.5 With Error: "java.io.IOException: Cannot Run Program "rpm": error=13, Permission Denied" (Doc ID 2435529.1)

Last updated on JULY 02, 2020

Applies to:

Linux OS - Version Oracle Linux 7.5 and later
Linux x86-64

Symptoms

Unable to start spacewalk service after OS has been patched to version 7.5.

Below errors in the Spacewalk web GUI:

Internal Server Error
The server experienced a problem which prevented your request from being filled out. It may not be possible to execute this action at this time.
Please help us correct this problem by contacting us with details of how you received this message.

Screenshot as follows:

Below errors in the logs.

/var/log/messages:
May 12 23:28:06 <hostname> systemd: Starting Taskomatic...
May 12 23:28:07 <hostname> taskomatic: Starting RHN Taskomatic...
May 12 23:28:07 <hostname> wrapper[17419]: --> Wrapper Started as Daemon
May 12 23:28:07 <hostname> wrapper[17419]: Launching a JVM...
May 12 23:28:07 <hostname> systemd: Started Taskomatic.
May 12 23:28:07 <hostname> kernel: audit_printk_skb: 57 callbacks suppressed
May 12 23:28:07 <hostname> kernel: audit: type=1130 audit(1526182087.510:257): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=taskomatic comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 12 23:28:14 <hostname> : 2018-05-12 23:28:14,761 [ajp-bio-0:0:0:0:0:0:0:1-8009-exec-1] INFO com.redhat.rhn.frontend.xmlrpc.LoggingInvocationProcessor - REQUESTED FROM: <IP> CALL: auth.checkAuthToken(taskomatic_user, 0d94e9099a92ae402d26c1436cd4954b7049d3898d32e2ad7c9494b6d35bf81e) CALLER: (none) TIME: 0.005 seconds
May 12 23:28:14 <hostname> : 2018-05-12 23:28:14,761 [ajp-bio-0:0:0:0:0:0:0:1-8009-exec-2] INFO com.redhat.rhn.frontend.xmlrpc.LoggingInvocationProcessor - REQUESTED FROM: <IP> CALL: auth.checkAuthToken(taskomatic_user, 0d94e9099a92ae402d26c1436cd4954b7049d3898d32e2ad7c9494b6d35bf81e) CALLER: (none) TIME: 0.005 seconds
May 12 23:28:25 <hostname> kernel: audit: type=1400 audit(1526182105.470:258): avc: denied { execute } for pid=17584 comm="java" name="rpm" dev="dm-1" ino=2152227 scontext=system_u:system_r:tomcat_t:s0 tcontext=system_u:object_r:rpm_exec_t:s0 tclass=file permissive=0
May 12 23:28:25 <hostname> kernel: audit: type=1300 audit(1526182105.470:258): arch=c000003e syscall=59 success=no exit=-13 a0=7f30ca0cc859 a1=7f30540234a0 a2=7f30d83b1480 a3=7f30ca0cc4d0 items=0 ppid=1732 pid=17584 auid=4294967295 uid=91 gid=91 euid=91 suid=91 fsuid=91 egid=91 sgid=91 fsgid=91 tty=(none) ses=4294967295 comm="java" exe="/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/jre/bin/java" subj=system_u:system_r:tomcat_t:s0 key=(null)
May 12 23:28:25 <hostname> kernel: audit: type=1327 audit(1526182105.470:258): proctitle=2F7573722F6C69622F6A766D2F6A72652F62696E2F6A617661002D6561002D586D733235366D002D586D783235366D002D446A6176612E6177742E686561646C6573733D74727565002D446F72672E786D6C2E7361782E6472697665723D6F72672E6170616368652E7865726365732E706172736572732E5341585061727365
May 12 23:28:25 <hostname> : 2018-05-12 23:28:25,473 [ajp-bio-0:0:0:0:0:0:0:1-8009-exec-3] ERROR com.redhat.rhn.manager.satellite.SystemCommandExecutor - execute(String[])
May 12 23:28:25 <hostname> : java.io.IOException: Cannot run program "rpm": error=13, Permission denied >>>
May 12 23:28:25 <hostname> : at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
May 12 23:28:25 <hostname> : at java.lang.Runtime.exec(Runtime.java:620)
May 12 23:28:25 <hostname> : at java.lang.Runtime.exec(Runtime.java:485)
May 12 23:28:25 <hostname> : at com.redhat.rhn.manager.satellite.SystemCommandExecutor.execute(SystemCommandExecutor.java:71)
May 12 23:28:25 <hostname> : at com.redhat.rhn.frontend.action.LoginHelper.getRpmSchemaVersion(LoginHelper.java:343)
May 12 23:28:25 <hostname> : at com.redhat.rhn.frontend.action.LoginHelper.isSchemaUpgradeRequired(LoginHelper.java:313)
May 12 23:28:25 <hostname> : at com.redhat.rhn.frontend.action.LoginSetupAction.execute(LoginSetupAction.java:47)
May 12 23:28:25 <hostname> : at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
May 12 23:28:25 <hostname> : at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
May 12 23:28:25 <hostname> : at com.redhat.rhn.frontend.struts.RhnRequestProcessor.process(RhnRequestProcessor.java:105)
May 12 23:28:25 <hostname> : at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
May 12 23:28:25 <hostname> : at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
May 12 23:28:25 <hostname> : at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
May 12 23:28:25 <hostname> : at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
May 12 23:28:25 <hostname> : at org.apache.tomcat.websocket..WsFilter.doFilter(WsFilter.java:52)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
May 12 23:28:25 <hostname> : at com.redhat.rhn.frontend.servlets.AuthFilter.doFilter(AuthFilter.java:107)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
May 12 23:28:25 <hostname> : at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
May 12 23:28:25 <hostname> : at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
May 12 23:28:25 <hostname> : at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
May 12 23:28:25 <hostname> : at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
May 12 23:28:25 <hostname> : at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:57)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
May 12 23:28:25 <hostname> : at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
May 12 23:28:25 <hostname> : at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
May 12 23:28:25 <hostname> : at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
May 12 23:28:25 <hostname> : at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
May 12 23:28:25 <hostname> : at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
May 12 23:28:25 <hostname> : at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
May 12 23:28:25 <hostname> : at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
May 12 23:28:25 <hostname> : at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
May 12 23:28:25 <hostname> : at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
May 12 23:28:25 <hostname> : at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
May 12 23:28:25 <hostname> : at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
May 12 23:28:25 <hostname> : at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
May 12 23:28:25 <hostname> : at java.lang.Thread.run(Thread.java:748)
May 12 23:28:25 <hostname> : Caused by: java.io.IOException: error=13, Permission denied >>>
May 12 23:28:25 <hostname> : at java.lang.UNIXProcess.forkAndExec(Native Method)
May 12 23:28:25 <hostname> : at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
May 12 23:28:25 <hostname> : at java.lang.ProcessImpl.start(ProcessImpl.java:134)
May 12 23:28:25 <hostname> : at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
May 12 23:28:25 <hostname> : ... 52 more

/var/log/rhn/rhn_taskomatic_daemon.log:
java.lang.InstantiationException: this.scheduler failed
at com.redhat.rhn.taskomatic.core.SchedulerKernel.<init>(SchedulerKernel.java:121)
at com.redhat.rhn.taskomatic.core.TaskomaticDaemon.onStartup(TaskomaticDaemon.java:82)
at com.redhat.rhn.taskomatic.core.BaseDaemon.startupWithOptions(BaseDaemon.java:188)
at com.redhat.rhn.taskomatic.core.BaseDaemon.start(BaseDaemon.java:55)
at org.tanukisoftware.wrapper.WrapperManager$11.run(WrapperManager.java:2788)

STATUS | wrapper | 2018/05/12 23:17:38 | <-- Wrapper Stopped
STATUS | wrapper | 2018/05/12 23:27:05 | --> Wrapper Started as Daemon
STATUS | wrapper | 2018/05/12 23:27:05 | Launching a JVM...
INFO | jvm 1 | 2018/05/12 23:27:05 | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
INFO | jvm 1 | 2018/05/12 23:27:05 | Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.
INFO | jvm 1 | 2018/05/12 23:27:05 |
INFO | jvm 1 | 2018/05/12 23:27:06 | org.quartz.SchedulerConfigException: Failure occured during job recovery. [See nested exception: org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'rhnDs': org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
INFO | jvm 1 | 2018/05/12 23:27:06 | ) [See nested exception: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
INFO | jvm 1 | 2018/05/12 23:27:06 | )]]
INFO | jvm 1 | 2018/05/12 23:27:06 | at org.quartz.impl.jdbcjobstore.JobStoreSupport.initialize(JobStoreSupport.java:607)
INFO | jvm 1 | 2018/05/12 23:27:06 | at org.quartz.impl.jdbcjobstore.JobStoreTX.initialize(JobStoreTX.java:56)
INFO | jvm 1 | 2018/05/12 23:27:06 | at org.quartz.impl.StdSchedulerFactory.instantiate(StdSchedulerFactory.java:1274)
INFO | jvm 1 | 2018/05/12 23:27:06 | at org.quartz.impl.StdSchedulerFactory.getScheduler(StdSchedulerFactory.java:1465)
INFO | jvm 1 | 2018/05/12 23:27:06 | at com.redhat.rhn.taskomatic.core.SchedulerKernel.<init>(SchedulerKernel.java:103)
INFO | jvm 1 | 2018/05/12 23:27:06 | at com.redhat.rhn.taskomatic.core.TaskomaticDaemon.onStartup(TaskomaticDaemon.java:82)
INFO | jvm 1 | 2018/05/12 23:27:06 | at com.redhat.rhn.taskomatic.core.BaseDaemon.startupWithOptions(BaseDaemon.java:188)
INFO | jvm 1 | 2018/05/12 23:27:06 | at com.redhat.rhn.taskomatic.core.BaseDaemon.start(BaseDaemon.java:55)
INFO | jvm 1 | 2018/05/12 23:27:06 | at org.tanukisoftware.wrapper.WrapperManager$11.run(WrapperManager.java:2788)
INFO | jvm 1 | 2018/05/12 23:27:06 | Caused by: org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'rhnDs': org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Changes

 OS upgrade has been performed, which includes SELinux rpms.

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.