[Spce-user] nginx and prometheus-nginx-exporter fail to load after reboot

Sun Jan 19 09:58:45 EST 2025

I’m sorry, I said 11.5 LTS but it is actually 12.5.1 LTS. I tried upgrading to 13 at one point to see if it solved the issue, but the problem persisted, so I did another 12.5.1 clean install. No certificates, all default config. The problem will arise as soon as you reboot after installation is fully done, regardless it is a clean slate install, or data has been added to it.

I also encountered the admin problem as soon as I changed the password for administrator. Role changes to (reseller). I had to go into the database and change the administrator account to is_system as a work around. I thought it was something I had done wrong, but I saw it again after a clean install, so I assumed it was a bug.

As for the logs, that is the puzzling part…

/var/log/nginx/error.log is empty
/var/log/ngcp/nginx/error.log is also empty

When the server boots, right before the login prompt shows up, you can see:

[FAILED] Failed to start nginx.service - A high performance web server and a reverse proxy server.

[FAILED] Failed to start prometheus-nginx-exporter.service - NGINX Prometheus Exporter.

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Walter Klomp <walter at myrepublic.net>
Sent: Sunday, January 19, 2025 10:31:24 AM
To: Cesar Mora <cesarluis89 at gmail.com>
Cc: spce-user at lists.sipwise.com <spce-user at lists.sipwise.com>
Subject: Re: [Spce-user] nginx and prometheus-nginx-exporter fail to load after reboot

Maybe do have a look at why nginx doesn’t boot. The logs are all in /var/log/nginx and in /var/log/ngcp

Rate-o-mat depends on valid billing entries. Will not affect operations but will affect credit control and CDR etc….

Are you using any let’s encrypt certificates?

I’m running 12.5.1 as CE and other than a known bug on admin access rights it should just work out of the box. And I installed it from installer CD.

Just try to not update it to development 13.x first.

Regards
Walter.

On Sun, 19 Jan 2025 at 22:20, Cesar Mora <cesarluis89 at gmail.com<mailto:cesarluis89 at gmail.com>> wrote:
Yup, done all that per the manual, otherwise the install is incomplete and won’t run.

  1.  boot with install cd
  2.  Setup network interface
  3.  Wait until full install is complete and asks for the first reboot.
  4.  After first reboot, go into terminal multiplexer and run ngcp-initial-configuration.
  5.  Move listening services from lo to neth0 (interface name in my case).
  6.  Run first ngcpcfg apply ‘net config change’
  7.  Reboot again.
  8.  Config.yml is now updated with new listening interface name, run ngcpcfg apply ‘added network interface’
  9.  Now everything works perfectly fine, I added peering server, registered some test customers, all good!

Everything will keep working perfectly until I reboot the server for any reason. The server will boot, sip services work, ssh works, but ngix service will fail to boot, so no web UI. ngcp-services shows ngix and prometeus-ngix-exporter are managed and on-boot but status is failed to start. Manually restarting them will get them up and running, but this is not the expected behaviour. Btw, ngcp-rate-o-mat service will start on boot, but also fails after the server has been running for a while, but doesn’t seem to affect anything vital.

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Walter Klomp <walter at myrepublic.net<mailto:walter at myrepublic.net>>
Sent: Sunday, January 19, 2025 9:35:28 AM
To: Cesar Mora <cesarluis89 at gmail.com<mailto:cesarluis89 at gmail.com>>
Cc: spce-user at lists.sipwise.com<mailto:spce-user at lists.sipwise.com> <spce-user at lists.sipwise.com<mailto:spce-user at lists.sipwise.com>>
Subject: Re: [Spce-user] nginx and prometheus-nginx-exporter fail to load after reboot

After rebooting you need to do an ngcp-initial-install if memory serves me right. Just continue with the manual after the reboot.

[Signature image]
Walter Klomp
Head of Voice Network
[Email]         walter at myrepublic.net<mailto:walter at myrepublic.net>
Beyond Work :
11 Lorong 3 Toa Payoh, #04<https://www.google.com/maps/search/11+Lorong+3+Toa+Payoh,+%2304?entry=gmail&source=g>-11/15, Jackson Square, Block B, Singapore 319579
www.myrepublic.net<http://www.myrepublic.net>
[Facebook-f] <https://scribe-mail.myrepublic.net/s2/21iu7kgk>   [Instagram]  <https://scribe-mail.myrepublic.net/s2/kzqypegm>   [Twitter]  <https://scribe-mail.myrepublic.net/s2/geefi681>     [Linkedin-in]  <https://scribe-mail.myrepublic.net/s2/3bzu1nu9>
The information in this message is privileged and confidential. If you are not the intended recipient of this email, please don't read, copy, use, distribute or tell anyone about it; kindly destroy it and notify the sender by return email. Please advise immediately if you or your employer does not consent to email for messages of this kind. Any content that is not concerned with MyRepublic business reflects the views of the sender only and not those of MyRepublic.

On Sun, 19 Jan 2025 at 21:16, Cesar Mora <cesarluis89 at gmail.com<mailto:cesarluis89 at gmail.com>> wrote:
Hi,

I’ve been trying to get a stable install of Sipwise CE for the last couple of days with no avail. Everything goes smooth per the handbook, until I reboot the server. Each time I reboot the server, nginx and prometheus-nginx-exporter fail to load, and have to be manually started by ngcp-service restart or by ngcpcfg apply which also restarts services. I have tried using the installcd, doing a clean debian with ngcp installer, and with various versions of the release, and always get the same outcome. I even tried installing 11.5 LTS and upgrading from there to 13.1.1, same thing. I checked if systemclt has something in the logs for not being able to start nginx, but nope, clear logs. After I restart the services, everything keeps working butter smooth until the server gets rebooted again, which is not reliable for production.

This is running on a SuperMicro blade, dedicated host, no VMs. I have come to the conclusion that there might be some kernel issue with this hardware, but I cannot replicate the issue with a clean Debian install.

Any help is appreciated!

Cesar
--
Spce-user mailing list
Spce-user at lists.sipwise.com<mailto:Spce-user at lists.sipwise.com>
http://lists.sipwise.com/mailman/listinfo/spce-user_lists.sipwise.com

The contents of this email and any attachments are confidential and may also be privileged. You must not disseminate the contents of this email and any attachments without permission of the sender. If you have received this email by mistake, please delete all copies and inform the sender immediately. You may refer to our company's Privacy Policy here<https://myrepublic.net/sg/legal/terms-of-use-policies/privacy-policy/>.

The contents of this email and any attachments are confidential and may also be privileged. You must not disseminate the contents of this email and any attachments without permission of the sender. If you have received this email by mistake, please delete all copies and inform the sender immediately. You may refer to our company's Privacy Policy here<https://myrepublic.net/sg/legal/terms-of-use-policies/privacy-policy/>.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20250119/9fdc0677/attachment-0001.htm>