[Spce-user] Error during upgrade mr13.1.1 to mr13.4.1

[Danny Cappellari]

Hi, thanks for the reply. Today I tried the update from 13.1.1 to 13.5.1 again and I'm having the exact same problems. At the end of the second stage, mariadb doesn't start. I've attached the log from the mysql-error.log file. Can you help me?


------------------------------------

2025-10-13 21:38:14 0 [Note] Starting MariaDB 11.8.3-MariaDB-0+deb13u1 from Debian-log source revision b565b3e7e041c480fd8013e3beec6b99544d6bf8 server_uid m2/bxpiL3VRdHN>
2025-10-13 21:38:14 0 [Note] mariadbd: Read from /etc/mysql/keyfile, read bytes: 67, max key file size: 1048576 bytes
2025-10-13 21:38:14 0 [Note] InnoDB: !!! innodb_force_recovery is set to 2 !!!
2025-10-13 21:38:14 0 [Note] InnoDB: Compressed tables use zlib 1.3.1
2025-10-13 21:38:14 0 [Note] InnoDB: Number of transaction pools: 1
2025-10-13 21:38:14 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2025-10-13 21:38:14 0 [Note] InnoDB: Using io_uring
2025-10-13 21:38:14 0 [Note] InnoDB: innodb_buffer_pool_size_max=64m, innodb_buffer_pool_size=64m
2025-10-13 21:38:14 0 [Note] InnoDB: Initialized memory pressure event listener
2025-10-13 21:38:14 0 [Note] InnoDB: Completed initialization of buffer pool
2025-10-13 21:38:14 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2025-10-13 21:38:14 0 [Note] InnoDB: End of log at LSN=384735814166
2025-10-13 21:38:14 0 [ERROR] InnoDB: The change buffer is corrupted
2025-10-13 21:38:14 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
2025-10-13 21:38:14 0 [Note] InnoDB: Starting shutdown...
2025-10-13 21:38:14 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2025-10-13 21:38:14 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-10-13 21:38:14 0 [Note] Plugin 'wsrep-provider' is disabled.
2025-10-13 21:38:14 0 [Warning] 'innodb-change-buffering' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2025-10-13 21:38:14 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2025-10-13 21:38:14 0 [ERROR] Aborting

-----------------



A: Danny Cappellari
Cc: spce-user at lists.sipwise.com
Oggetto: Re: [Spce-user] Error during upgrade mr13.1.1 to mr13.4.1

Dear Danny,

I have tested the same upgrade path as you did from mr13.1.1 to mr13.3.1 and finally to mr13.4.1 and I did not get any errors.

Regarding the 1st problem you mentioned, the prosody_unique_index up script should not create any problems as this is only introduced in mr13.4.1. Maybe some leftover from prior attempts to mr13.4.1 that was not cleaned up?

Regarding the 2nd problem maybe some corrupted data present in the DB prior to upgrade?

Best regards,
Jose Su

On 8/20/25 21:02, Danny Cappellari via Spce-user wrote:
Hi Marco, the upgrade process to 13.4.1 is still having some issues. I managed to upgrade from 13.1.1 to 13.3.1 without any problems.

When I try to upgrade from 13.3.1 to 13.4.1, I encounter several issues during the second stage of the upgrade.

First problem:

CREATE UNIQUE INDEX prosody_unique_index ON prosody(`host`(20), `user`(20), `store`(20), `key`(20)) USING BTREE
--------------

ERROR 1061 (42000) at line 5 in file: '/usr/share/ngcp-db-schema/db_scripts/diff/15841.up': Duplicate key name 'prosody_unique_index'

ERROR: the step 'mr12.5/ngcp_update_db_schema' failed, upgrade aborted!
Please fix the root of the issue and restart 'ngcp-upgrade'
(see error details in log file /ngcp-data/ngcp-upgrade/mr13.3.1-mr13.4.1/logs/ngcp-upgrade-mr13.3.1-mr13.4.1-1755716002.log ).

Bypass by comment DROP and CREATE in /usr/share/ngcp-db-schema/db_scripts/diff/15841.up

Second problem:

After the update, some services won't start (including MariaDB). In the MariaDB logs, I see:


2025-08-20 20:56:27 0 [ERROR] InnoDB: The change buffer is corrupted
2025-08-20 20:56:27 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
2025-08-20 20:56:27 0 [Note] InnoDB: Starting shutdown...
2025-08-20 20:56:28 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2025-08-20 20:56:28 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-08-20 20:56:28 0 [Note] Plugin 'wsrep-provider' is disabled.
2025-08-20 20:56:28 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2025-08-20 20:56:28 0 [ERROR] Aborting


I have reverted the VM to 13.3.1 for now



________________________________
Da: Marco Capetta <mcapetta at sipwise.com><mailto:mcapetta at sipwise.com>
Inviato: mercoledì 20 agosto 2025 17:16
A: Danny Cappellari
Cc: spce-user at lists.sipwise.com<mailto:spce-user at lists.sipwise.com>
Oggetto: Re: [Spce-user] Error during upgrade mr13.1.1 to mr13.4.1

Hi Danny,

it seems that both SEMS and MARIADB services are failing.
You can check the respective logs and identify what it going wrong.

Regards
Marco

On Wed, Aug 20, 2025 at 2:23 PM Danny Cappellari via Spce-user <spce-user at lists.sipwise.com<mailto:spce-user at lists.sipwise.com>> wrote:
Hi, we are unable to upgrade from mr13.1.1 to mr13.4.1. We detect an error during the Second Stage of Update (ngcp-upgrade --target mr13.4.1)

ngcp-service: info: Action restart skipped, service acpid is not enabled
ngcp-service: info: Action restart skipped, service kannel-bearerbox is not enabled
ngcp-service: info: Action restart skipped, service kannel-smsbox is not enabled
ngcp-service: info: Action restart skipped, service ngcp-logic-engine is not enabled
ngcp-service: info: Action restart skipped, service ntpd is not enabled
ngcp-service: info: Action restart skipped, service prometheus is not enabled
ngcp-service: info: Action restart skipped, service rtpengine-recording is not enabled
ngcp-service: info: Action restart skipped, service sssd is not enabled
ngcp-service: info: Action restart skipped, service valkey is not enabled
ngcp-service: info: Action restart skipped, service vboxadd-service is not enabled
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xeu mariadb.service" for details.
A dependency job for kamailio-proxy.service failed. See 'journalctl -xe' for details.
Job for ngcp-sems.service failed because the control process exited with error code.
See "systemctl status ngcp-sems.service" and "journalctl -xeu ngcp-sems.service" for details.
2025-08-20 14:19:03 spce: Synchronizing current with expected services state
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xeu mariadb.service" for details.
A dependency job for kamailio-proxy.service failed. See 'journalctl -xe' for details.
Job for ngcp-sems.service failed because the control process exited with error code.
See "systemctl status ngcp-sems.service" and "journalctl -xeu ngcp-sems.service" for details.
ERROR: the step 'mr9.5/apply_new_configs' failed, upgrade aborted!
Please fix the root of the issue and restart 'ngcp-upgrade'
(see error details in log file /ngcp-data/ngcp-upgrade/mr13.1.1-mr13.4.1/logs/ngcp-upgrade-mr13.1.1-mr13.4.1-1755692307.log ).

Any advice?
--
Spce-user mailing list
Spce-user at lists.sipwise.com<mailto:Spce-user at lists.sipwise.com>
http://lists.sipwise.com/mailman/listinfo/spce-user_lists.sipwise.com



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20251013/99bdf605/attachment-0001.htm>