Recently, an upgrade on a couple of Splunk 10.2.5 servers on Red Hat 8.10 failed with unexpected messages. During the first startup of 10.2.7, the migration to the new version stopped with
ERROR: In order to migrate, Splunkd must not be running.
Splunk had presumably been stopped well before this. When ckecking splunk status, the license agreement and migration prompts would be presented — even though --accept-license --answer-yes were provided — and the process stopped again with the above error message.
I used an explicit splunk stop before starting the upgrade, then ran ps -e | grep splunk after the process reported that Splunk and helper processes had been stopped, and I still found Splunk processes.
# /opt/splunk/bin/splunk stop
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
................... [ OK ]
Stopping splunk helpers...
[ OK ]
Done.
# ps -e | grep splunk
4185174 ? 00:00:08 splunkd
4185175 ? 00:00:00 splunkd
4185306 ? 00:00:00 splunkd
4185307 ? 00:00:00 splunkd
4185327 ? 00:00:07 splunkd
4185328 ? 00:00:00 splunkd
4185331 ? 00:00:07 splunkd
4185332 ? 00:00:00 splunkd
4185336 ? 00:00:08 splunkd
4185337 ? 00:00:00 splunkd
killall splunkd fixed the problem in the above case, but on another server not only were there instances of splunkd running but other splunk processes like splunk-supervis also. I had to kill these individually to continue with the upgrade.
I could not find any reason in the logs for these processes to have been abandoned or ignored, nor do I see any known issues. A similar problem was reported by a user upgrading a Universal Forwarder to version 8.2.4 back in 2022. The article points out that the installer script checks for the Splunk binary in the file system and runs splunk stop if it exists. There is no check for an error result (though in my case stop does not return an error), nor does it check if all processes were terminated.
I added steps to check for running Splunk processes in my upgrade scripts.