[Spce-user] [EXTERNAL] Re: nginx and prometheus-nginx-exporter fail to load after reboot

[Cesar Mora]

Syslog shows that nginx will be commanded for start, but later will fail with:

Jan 22 10:12:00 spce systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jan 22 10:12:00 spce systemd[1]: nginx.service: Failed with result 'exit-code’.

Prometheus fails as well, but I guess it is nginx dependent:

Jan 22 10:12:00 spce systemd[1]: prometheus-nginx-exporter.service: Main process exited, code=exited, status=1/FAILURE
Jan 22 10:12:00 spce systemd[1]: prometheus-nginx-exporter.service: Failed with result 'exit-code’.

I also see that prometheus will try to restart:

Jan 22 10:12:01 spce systemd[1]: prometheus-nginx-exporter.service: Scheduled restart job, restart counter is at 2.
Jan 22 10:12:01 spce systemd[1]: Stopped prometheus-nginx-exporter.service - NGINX Prometheus Exporter.
Jan 22 10:12:01 spce systemd[1]: Started prometheus-nginx-exporter.service - NGINX Prometheus Exporter.

But nginx does not try to restart...

I also noticed before that ngcp-status will not work after at least 10 minutes after server starts, but could be unrelated, and I see this on the log:

Jan 22 10:12:01 spce systemd[1]: Starting ngcp-panel.service - NGCP Panel Webapp...
Jan 22 10:12:01 spce rmmod[1098]: rmmod: ERROR: Module ngcp is not currently loaded

> On Jan 20, 2025, at 5:07 AM, Michael Prokop via Spce-user <spce-user at lists.sipwise.com> wrote:
> 
> Hi,
> 
> * Cesar Mora [Sun Jan 19, 2025 at 02:58:45PM +0000]:
> 
>> I’m sorry, I said 11.5 LTS but it is actually 12.5.1 LTS. I tried
>> upgrading to 13 at one point to see if it solved the issue, but
>> the problem persisted, so I did another 12.5.1 clean install. No
>> certificates, all default config. The problem will arise as soon
>> as you reboot after installation is fully done, regardless it is a
>> clean slate install, or data has been added to it.
> 
> [...]
> 
>> As for the logs, that is the puzzling part…
>> 
>> /var/log/nginx/error.log is empty
>> /var/log/ngcp/nginx/error.log is also empty
>> 
>> When the server boots, right before the login prompt shows up, you can see:
>> 
>> [FAILED] Failed to start nginx.service - A high performance web server and a reverse proxy server.
>> 
>> [FAILED] Failed to start prometheus-nginx-exporter.service - NGINX Prometheus Exporter.
> 
> What's inside your syslog (/var/log/syslog)?
> 
> If that shouldn't provide any further information for you,
> try enabling journald:
> 
>  ngcpcfg set /etc/ngcp-config/config.yml systemd.journald.storage=auto
>  ngcpcfg apply "enable journald"
> 
> Then check out "journalctl -u nginx" after rebooting.
> 
> regards
> -mika-
> 
> -- 
> Spce-user mailing list
> Spce-user at lists.sipwise.com
> http://lists.sipwise.com/mailman/listinfo/spce-user_lists.sipwise.com