[Spce-user] Logfile mysqld.err 17GB big

Tue Jul 2 10:00:09 EDT 2019

mysqld.err file can just be emptied with >/var/log/mysql/mysqld.err - you
can copy it somewhere else first... but that error file shouldn't be
growing that fast to begin with. fix that first.
* old log files can be removed from /var/log/ngcp/old if you don't need
them.
* you can reduce the logging of kamailio-proxy and kamailio-lb if you don't
need it for troubleshooting. That should save quite a bit iops and space.
* /var/backup/cdr should have old cdr and /var/backup/ other directories
have backups from upgrades which you may want to copy out and then empty
those directories

On Tue, Jul 2, 2019 at 9:38 PM Thomas Georg <t.georg at msg-gruppe.de> wrote:

> Hi list,
>
> we had an issue when our spce has been upgraded to 6.5.4.
> At this time our harddisk went out of space.
>
> We are planning to move our production system to a new machine so I made
> some investigations where the disk space has gone.
>
> First I found the mysqld.err file in /var log/mysql/ with a size of 17GB!
>
> I wondered about the following:
> ########## mysqld.err ########
> 2019-05-30 23:08:27 140628650698752 [ERROR] mysqld: Can't lock aria
> control file '/ngcp-data/mysql/aria_log_control' for exclusive use, error:
> 11. Will retry for 30 secon
> ds
> 2019-05-30 23:08:27 140112560863232 [ERROR] mysqld: Can't lock aria
> control file '/ngcp-data/mysql/aria_log_control' for exclusive use, error:
> 11. Will retry for 30 secon
> ds
> 2019-05-30 23:08:27 140253069703168 [Note] InnoDB:
> innodb_empty_free_list_algorithm has been changed to legacy because of
> small buffer pool size. In order to use backoff,
>  increase buffer pool at least up to 20MB.
>
> 2019-05-30 23:08:27 140253069703168 [Note] InnoDB: Using mutexes to ref
> count buffer pool pages
> 2019-05-30 23:08:27 140253069703168 [Note] InnoDB: The InnoDB memory heap
> is disabled
> 2019-05-30 23:08:27 140253069703168 [Note] InnoDB: Mutexes and rw_locks
> use GCC atomic builtins
> 2019-05-30 23:08:27 140253069703168 [Note] InnoDB: GCC builtin
> __atomic_thread_fence() is used for memory barrier
> 2019-05-30 23:08:27 140253069703168 [Note] InnoDB: Compressed tables use
> zlib 1.2.8
> 2019-05-30 23:08:27 140253069703168 [Note] InnoDB: Using Linux native AIO
> 2019-05-30 23:08:27 140253069703168 [Note] InnoDB: Using SSE crc32
> instructions
> 2019-05-30 23:08:27 140253069703168 [Note] InnoDB: Initializing buffer
> pool, size = 64.0M
> 2019-05-30 23:08:27 140253069703168 [Note] InnoDB: Completed
> initialization of buffer pool
> 2019-05-30 23:08:29 140253069703168 [Note] InnoDB: Setting log file
> /ngcp-data/mysql/ib_logfile101 size to 128 MB
> 2019-05-30 23:08:31 140253069703168 [Note] InnoDB: Setting log file
> /ngcp-data/mysql/ib_logfile1 size to 128 MB
> 2019-05-30 23:08:35 140253069703168 [Note] InnoDB: Setting log file
> /ngcp-data/mysql/ib_logfile2 size to 128 MB
> 2019-05-30 23:08:39 140253069703168 [Note] InnoDB: Setting log file
> /ngcp-data/mysql/ib_logfile3 size to 128 MB
> 2019-05-30 23:08:40 140253069703168 [Note] InnoDB: Renaming log file
> /ngcp-data/mysql/ib_logfile101 to /ngcp-data/mysql/ib_logfile0
> 2019-05-30 23:08:40 140253069703168 [Warning] InnoDB: New log files
> created, LSN=74857028238
> 2019-05-30 23:08:40 140253069703168 [Note] InnoDB: Highest supported file
> format is Barracuda.
> 2019-05-30 23:08:40 7f8f3665f800 InnoDB: Error: page 3 log sequence number
> 75407322724
> InnoDB: is in the future! Current system log sequence number 74857028620.
> InnoDB: Your database may be corrupt or you may have copied the InnoDB
> InnoDB: tablespace but not the InnoDB log files. See
> InnoDB:
> http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
> InnoDB: for more information.
> 2019-05-30 23:08:40 7f8f3665f800 InnoDB: Error: page 2 log sequence number
> 98893166510
> InnoDB: is in the future! Current system log sequence number 74857028620.
> InnoDB: Your database may be corrupt or you may have copied the InnoDB
> InnoDB: tablespace but not the InnoDB log files. See
> InnoDB:
> http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
> InnoDB: for more information.
> .......
> #########################
>
> The log is full of these InnoDB errors.
>
> Has anyone an explanation for this and how to fix it / do more
> investigation?
>
> Our disk is currently at 91%, is it possible to move the file to another
> place (scp to our central backup storage) and restart the logging without
> service interruption?
>
> Regards
> Thomas
>
>
>
> _______________________________________________
> Spce-user mailing list
> Spce-user at lists.sipwise.com
> https://lists.sipwise.com/listinfo/spce-user
>

-- 

Warmest Regards,

<https://myrepublic.com.sg/>
*Walter Klomp*
Head of Voice & Systems
MyRepublic Limited
T: +65 6816 1120
F: +65 6717 2031

MyRepublic Limited
11 Lorong 3 Toa Payoh Block B Jackson Square
#04-11/15 Singapore 319579

myrepublic.com.sg
Follow us on: Twitter <https://twitter.com/myrepublic> | Facebook
<https://facebook.com/myrepublicsg> | LinkedIn
<https://www.linkedin.com/company/myrepublic>

-- 
The contents of this email and any attachments are confidential and may 
also be privileged. You must not disseminate the contents of this email and 
any attachments without permission of the sender. If you have received this 
email by mistake, please delete all copies and inform the sender 
immediately. You may refer to our company's Privacy Policy here 
<https://myrepublic.net/sg/legal/terms-of-use-policies/privacy-policy/>.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20190702/f1a5bca2/attachment-0001.html>