[Spce-user] Errror while upgrading from mr4.5.3 to mr5.5.2

[Alex Lutay]

Hi all,

Tracing such issues is like walking in a dark.
Initially the problem was reported about missing record in DB,
later report says about some missing packages (missing ngcpcfg) ...

See my in-line comments.

On 01/28/2018 12:42 PM, Hohl Matthias wrote:
> i did the upgrade 1 week ago from 4.5.5 to 5.5.2 and had no issues with
> the upgrade progress.
> 
> Everything went smooth and it worked as expected.
> 
> Maybe try to upgrade to the latest 4.5.5/4.5.6 first and then go the
> step to 5.5.2.

>From what I know it should have no difference here.

> Firt of all, it's a demo system, not a production one, I was just
> testing the upgrade proc, 

This information was not shared initially.

> that as usual, doesn't work, either on a freaking clean system.
...
> Sorry if sounds rude, but that happens to me also, when upgrading from
> previous LTS to 4.5, and it's very frustating folling the instructions
> and get stuck.

At the moment Sipwise uses fully automated upgrade testing procedures.
>From the stories we have the most of the issues are coming from the
local modification in the deep past. We are improving the checks to
identify them and clean before the upgrade while sure community feedback
is absolutely necessary here.

I have just performed the manual upgrade of newly started mr4.5.3
Vagrant VM to mr5.5.2 and it has been executed without any issues:
https://pastebin.com/0e16LvP5 + https://pastebin.com/nTgcCm4Q

Also Jenkins reported no issues for the automatic test upgrade
2.8->mr3.8->mr4.5->mr5.5

Can you compare the difference between the clean mr4.5.3 Vagrant VM
provided by Sipwise and you mr4.5.3 installation?

> What's the point of doing that? ... I know that on a new mr5.5.2
> everything works, I have another VM with it, this is a test of upgrading
> a mr4.5.3 to mr5.5.2

Lets focus on your mr4.5.3 setup and difference to Sipwise mr4.5.3
Vagrant VM we use here for the testing.

>> Can you please share the content of /etc/sipwise_ngcp_version
>> it will show us the VM life path (when it was installed, where it was
>> upgraded to and when).
> 
> support at vm-pre-sipwise:~$ cat /etc/ngcp_version
> mr4.5.3

Wrong file. The file /etc/sipwise_ngcp_version was necessary to execute
the same upgrade path as you had. Since it is reproducible on clean
mr4.5.3 VM you have, we should search the difference in mr4.5.3 in your
side and mr4.5.3 on Sipwise side.

> Its a clean direct mr4.5.3 install, with no upgrades done so far.

Can you please repeat the test with mr4.5.3 Vagrant VM Sipwise provides?
I am going to install mr4.5.3 (using Sipwise Install CD) and repeat the
test as well. BTW, did you use Sipwise Install CD to install mr4.5.3 or
Netinstall ISO + ngcp-installer.deb package?

>> You can check the consistency of YML files executing:
>>> ngcpcfg --validate check
>> It should report no errors.
> 
> Imposible, system got in an unstable state:
> 
> support at vm-pre-sipwise:~$ ngcpcfg --validate check
> -bash: ngcpcfg: command not found

This is really wired. It means packages installation was not finished
properly (is ngcp-ngcpcfg package installed?).
In this case ngcp-upgrade must be aborted earlier and you must not reach
DB upgrade stage and final ngcpcfg apply.
Please share full upgrade logs here (you can send them directly to me).

Ensure ngcp-ngcp-ce meta-package is installed after the upgrade
(it proves all necessary dependencies are installed properly).

> If I rollback to a snapshot before the upgrade try:
> 
> support at vm-pre-sipwise:~$ ngcpcfg --validate check
> support at vm-pre-sipwise:~$
...
> On a 4.5.3, that file doesn't exits ..
> 
> less /usr/share/ngcp-cfg-schema/validate/constants.yml
> /usr/share/ngcp-cfg-schema/validate/constants.yml: No such file or directory

ngcpcfg YML configs validation was not available in mr4.5

>> Also 'ngcp-status' might share some auto-detected issues with you.
> 
> At this point it doesn't make any sense either ... I'll try a new clean
> 4.5.3, then just after install it ... will try to do an upgrade to
> mr5.5.2 and see what happens.
> I hope not having to use and API dump and API restore of all the
> information to be able to upgrade.
> (null)

So far I am not able to reproduce the issue here locally.

JFYI, on my side in mr4.5.3 the only record in DB is "rtCengine",
and the record for "rtPengine" is created during the upgrade to mr5.5.2.

Do you have locally modified my.cnf (customtt?) which might affect the
process here.

To continue, please repeat the upgrade on mr4.5.3 Vagrant VM we provide.
If it works for you, check the difference of mr4.5.3 Vagrant VM and your
mr4.5.3 machine. Otherwise share the upgrade log file.

The important points to check are:
 * list of installed debian packages
 * local modifications (including customtt files, especially my.cnf)
 * database schema (try mysqldbcompare from package mysql-utilities)

-- 
Alex Lutay