I'm opening this one here since we're using Forge with DigitalOcean and hosting Laravel. I somehow doubt that any of these are related, but who knows.
Anyway, we have an issue from time to time where we need to restart droplet or at least MySQL server. After examining the logs, this is what I have found:
2025-02-26 6:57:11 0 [Note] Starting MariaDB 10.11.8-MariaDB-ubu2404 source revision 3a069644682e336e445039e48baae9693f9a08ee as process 1404100
2025-02-26 6:57:11 0 [Note] InnoDB: Compressed tables use zlib 1.3
2025-02-26 6:57:12 0 [Note] InnoDB: Number of transaction pools: 1
2025-02-26 6:57:12 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2025-02-26 6:57:12 0 [Note] InnoDB: Using liburing
2025-02-26 6:57:12 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2025-02-26 6:57:12 0 [Note] InnoDB: Initialized memory pressure event listener
2025-02-26 6:57:12 0 [Note] InnoDB: Completed initialization of buffer pool
2025-02-26 6:57:12 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2025-02-26 6:57:12 0 [Warning] InnoDB: Retry attempts for reading partial data failed.
2025-02-26 6:57:12 0 [ERROR] InnoDB: Failed to read log at 30831616: I/O error
2025-02-26 6:57:12 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2025-02-26 6:57:12 0 [Note] InnoDB: Starting shutdown...
2025-02-26 6:57:12 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2025-02-26 6:57:12 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-02-26 6:57:12 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2025-02-26 6:57:12 0 [ERROR] Aborting
Do anyone have any clue what could be the issue and how to resolve it? I do know my way around setting up server, but this one is beyond my skills.
The error logs you provided indicate an issue with the InnoDB storage engine, which is crucial for MySQL/MariaDB operations. The specific error messages suggest that there is an I/O error when trying to read the log files, which could be due to several reasons such as disk space issues, file corruption, or hardware problems.
Here are some steps you can take to troubleshoot and potentially resolve the issue:
Check Disk Space:
Ensure that your server has enough disk space. You can check the available disk space using the following command:
df -h
If the disk is full, consider cleaning up unnecessary files or increasing the disk size.
Check File System:
There might be file system errors. You can check and repair the file system using fsck. Make sure to unmount the file system or boot into a rescue mode before running fsck.
Inspect MySQL/MariaDB Logs:
Look for additional clues in the MySQL/MariaDB error log, typically located at /var/log/mysql/error.log or /var/log/mariadb/mariadb.log.
Repair InnoDB Tables:
If the issue is with specific tables, you might need to repair them. You can try starting MySQL with the innodb_force_recovery option. Edit your MySQL configuration file (usually /etc/mysql/my.cnf or /etc/my.cnf) and add:
[mysqld]
innodb_force_recovery = 1
Start MySQL and attempt to dump your databases. Increase the innodb_force_recovery value incrementally (up to 6) if necessary, but be cautious as higher values can lead to data loss.
Check for Corrupted Files:
If the log files are corrupted, you might need to move or delete them. Before doing this, ensure you have a backup of your data. You can move the log files to a different location:
Reinstall MySQL/MariaDB:
As a last resort, if none of the above solutions work, you might need to reinstall MySQL/MariaDB. Ensure you have backups of your databases before proceeding.
Consult Hosting Provider:
If the issue persists, consider reaching out to DigitalOcean support for assistance, as there might be underlying hardware issues.
After making changes, remember to restart the MySQL/MariaDB service:
sudo systemctl restart mysql
Always ensure you have backups of your data before performing operations that might affect your database.
That log shows your MariaDB InnoDB engine failing to initialize because of an I/O error while reading the redo log, which is often a sign of disk corruption, bad blocks on storage, or damaged InnoDB log/data files. First thing, back up whatever .ibd and .frm files you can. Then try starting MariaDB with innodb_force_recovery (values 1–6) to dump the data. If built-in recovery fails, you can use third-party tools like Stellar Repair for MySQL, which can read directly from corrupted data files and rebuild the database without needing InnoDB to start, saving you a lot of time when logs or tablespaces are damaged.
