Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.
Please help. All of a sudden around 2019-01-24 06:37:10 UTC I started receiving errors on my server that:
SQLSTATE[HY000] [1045] Access denied for user 'forge'@'localhost' (using password: YES)
I'm unable to access mysql. I did not change anything on my end, so I don't know what happened. When I look through the system logs, I do see some mysql activity around that time:
Jan 24 06:36:07 whispering-hurricane systemd[1]: Starting Daily apt upgrade and clean activities...
Jan 24 06:36:24 whispering-hurricane kernel: [1767450.357697] [UFW BLOCK] IN=eth0 OUT= MAC=32:f4:a6:e9:03:29:f0:4b:3a:4e:50:30:08:00 SRC=85.10.193.56 DST=206.189.227.234 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=37186 DF PROTO=TCP SPT=57178 DPT=22123 WINDOW=29200 RES=0x00 SYN URGP=0
Jan 24 06:36:32 whispering-hurricane systemd[1]: Reloading.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Reloading.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Stopped Daily apt upgrade and clean activities.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Stopping Daily apt upgrade and clean activities.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Started Daily apt upgrade and clean activities.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Stopped Daily apt download activities.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Stopping Daily apt download activities.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Started Daily apt download activities.
Jan 24 06:36:38 whispering-hurricane systemd[1]: Reloading.
Jan 24 06:36:38 whispering-hurricane systemd[1]: Stopping MySQL Community Server...
Jan 24 06:36:42 whispering-hurricane systemd[1]: Stopped MySQL Community Server.
Jan 24 06:36:42 whispering-hurricane systemd[1]: Reloading.
Jan 24 06:36:46 whispering-hurricane kernel: [1767472.035497] audit: type=1400 audit(1548311806.292:39): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/sbin/mysqld" pid=27468 comm="apparmor_parser"
Jan 24 06:36:46 whispering-hurricane kernel: [1767472.069261] audit: type=1400 audit(1548311806.324:40): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=27486 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 24 06:36:46 whispering-hurricane kernel: [1767472.132572] audit: type=1400 audit(1548311806.388:41): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=27501 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 24 06:36:50 whispering-hurricane kernel: [1767476.078756] audit: type=1400 audit(1548311810.336:42): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=27552 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 24 06:36:53 whispering-hurricane kernel: [1767478.885369] audit: type=1400 audit(1548311813.140:43): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/sbin/mysqld" pid=27601 comm="apparmor_parser"
Jan 24 06:36:53 whispering-hurricane systemd[1]: message repeated 5 times: [ Reloading.]
Jan 24 06:36:53 whispering-hurricane systemd[1]: Starting MySQL Community Server...
Jan 24 06:36:53 whispering-hurricane kernel: [1767479.454427] audit: type=1400 audit(1548311813.708:44): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=27677 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 24 06:36:53 whispering-hurricane kernel: [1767479.499368] audit: type=1400 audit(1548311813.756:45): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=27682 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=113 ouid=0
Jan 24 06:36:54 whispering-hurricane systemd[1]: Started MySQL Community Server.
Can someone help me understand if something in that system log is what caused the problem? How can I fix it? This is our production server and I can not access the database. Neither can our application. Not good.
I've even tried adding a new database user from the forge control panel, but that new user gets the same access denied error.
Here's the mysql version info:
mysql --version
mysql Ver 14.14 Distrib 5.7.25, for Linux (x86_64) using EditLine wrapper
In forge:
- Go to your machine
- Go to mysql
- Select create user
- Assign user to your database
- use the same credidentials in your .env file in your project.
You are trying to use default forge user:
'forge'@'localhost'
That is where the problem is.
@AURAWINDSURFING - Sorry, I'm not sure what you mean. I've been using this user for the last 2 months and it's worked without issue. Also, as I said in my question, I did try creating a new user called 'forge2', but when I try to connection to the database with that user from the command line I also get the same error.
can you connect via?:
mysql -u forge -p
@AURAWINDSURFING - Somehow my 'forge' database user got deleted from mysql. This must have somehow happened during the automatic updates that occur on the server. I can see following entries in /var/log/apt/history.log:
Start-Date: 2019-01-24 06:36:21
Commandline: /usr/bin/unattended-upgrade
Upgrade: mysql-client-5.7:amd64 (5.7.24-0ubuntu0.18.04.1, 5.7.25-0ubuntu0.18.04.2)
End-Date: 2019-01-24 06:36:22
Start-Date: 2019-01-24 06:36:25
Commandline: /usr/bin/unattended-upgrade
Upgrade: mysql-client-core-5.7:amd64 (5.7.24-0ubuntu0.18.04.1, 5.7.25-0ubuntu0.18.04.2)
End-Date: 2019-01-24 06:36:26
Start-Date: 2019-01-24 06:36:29
Commandline: /usr/bin/unattended-upgrade
Upgrade: mysql-server:amd64 (5.7.24-0ubuntu0.18.04.1, 5.7.25-0ubuntu0.18.04.2)
End-Date: 2019-01-24 06:36:29
Start-Date: 2019-01-24 06:36:38
Commandline: /usr/bin/unattended-upgrade
Upgrade: mysql-server-5.7:amd64 (5.7.24-0ubuntu0.18.04.1, 5.7.25-0ubuntu0.18.04.2), mysql-server-core-5.7:amd64 (5.7.24-0ubuntu0.18.04.1, 5.7.25-0ubuntu0.18.04.2)
End-Date: 2019-01-24 06:36:54
These update ran one minute before I first saw the mysql access denied error happening all over my application. I can only assume that these updates somehow deleted my 'forge' user. I followed the steps here to reset my root password:
This allowed me to access the database, and when I did, I found that there was no 'forge' user at all. This makes no sense since I have been logging in with this user for months. My application also uses this user. So something changed during those updates. It seems like the user got deleted, but I have no idea why.
Is that possible?
Ok. in normal circumstances the forge or digital ocean will not update mysql. The only way you could have done that would be manually from cli the question is why you did it?
Anyhow the solution will be simply to remove whole website or provision the whole new machine. backup your database, migrate to the new one and forget about that issue ;-)
Please or to participate in this conversation.