This site can’t be reached on production server after client made changes via SSH

Is the webserver (apache/nginx/whatever) still running on the server? You should be able to run pgrep apache (or 'httpd' or 'nginx' or 'php-fpm'). If it is, maybe try doing a curl localhost and see if it's reachable from inside the server - then check if there's a firewall blocking remote things (iptables --list --numeric) - if that gives a lot of output then you could try opening up port 80/443 (I think you can do that via forge).

If it all looks ok though - maybe have a look at your webserver logs - they'll be in /var/log/apache2 (or nginx if you're using that) - they might give you some further clues.

@ohffs Thank for you taking the time. I'm trying to follow your instructions step by step but I'm still a bit lost.

When I run pgrep apache, nothing happens. When I run httpd, I get:

(13)Permission denied: AH00072: make_sock: could not bind to address 127.0.0.1:81 no listening sockets available, shutting down AH00015: Unable to open logs

curl localhost shows that it's not reachable from inside the server. It returns curl: (7) Failed to connect to localhost port 80: Connection refused.

When I try and run iptables I get:

iptables v1.6.1: can't initialize iptables table `filter': Permission denied (you must be root) Perhaps iptables or your kernel needs to be upgraded.

So it looks like I'm failing pretty early on in the process, but not sure where to go from here. Thanks again in advance.

Tyler

I believe forge logs in using certificate, not password, so that shouldn't be a problem.

You could check what command he actually used on the server by logging in via SSH and use the history command.... It might be that the person did more then those 2 things.

You can check if port 80 / 443 is open and listening :

netstat -lntu | grep 80
netstat -lntu | grep 443

And check if nginx / apache is running

sudo service --status-all

your webserver should have a + beside it, if it doesnt you can start it with

sudo service <name-of-service> start

@lostdreamer_nl nginx is not running (is has a minus sign next to it). When I enter sudo service nginx start, I get:

Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.

Following the recommendation in the message, I enter systemctl status nginx.service and get:

● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2018-07-11 15:21:54 UTC; 1min 27s ago Docs: man:nginx(8) Process: 2499 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=203/EXEC)

When I use the history command, here are all of the commands he entered:

55  curl -sSL https://agent.digitalocean.com/install.sh | sh
   56  sudo nano test.txt
   57  curl -sSL https://agent.digitalocean.com/install.sh | sh
   58  exit
   59  mysql -uforge -pXXXXXXEDITEDOUTXXXXX
   60  export DEBIAN_FRONTEND=noninteractive; echo 'Acquire::ForceIPv4 "true";' | tee /etc/apt/apt.conf.d/99force-ipv4; apt-get update; apt-get install curl -y; apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade -y; curl --silent --location https://manage.runcloud.io/script/installer/03IIOz0sfLGH9R0GdHkPTrtMAD1531248913ns6tBiCZGrDuY5WJgDuOQLZQhG6MaSeSjklfdsjaklSDJFIlsdfsaklScGIQO0zcRi0AN/7X77cGB9L7Do2w6XrPx3nj5661nUzLifGQLjdoflBKiXs3efsjk32kjsdkjflUlMcCx7C0HWiRsh38xy913omOPbKJBzH08tofml4QTqdO4eaMQ16WlSHWfddfja39j2lkmafdsa324239kOZYfXB4Dn6i0ziVBOfdsfKiF | bash -; export DEBIAN_FRONTEND=newt
   61  exit
   62  su root
   63  cu root
   64  su root

There should be no need to use the root account, sudo should be enough in most cases. Also, the Digital Ocean monitoring tools should cause no trouble. As for the RunCloud.io script, that's a complete server management control panel (like Forge), who knows what configurations it might have changed. If you are already using Forge then you probably shouldn't mix and match like that.

Might be easier to just backup the database and set up a new droplet through Forge.

The 'runcloud' part is possibly what's broken things - I think (from a brief look at their website) it seems to do some kind of server magic so possibly has broken what forge had done to the server :-/ Purely a guess though. Try running nginx -t (you might need to run sudo nginx -t) and see if it can find it's config file and test it's valid - but if the runcloud thing has reconfigured your server you might be better off trying to bring up a copy of the site on another droplet rather than un-do whatever it's done - just make sure you copy any files/db's though :-)

Edit: ah, just noticed @Nash said much the same :-) I am a slow typer today ;-)

That makes sense. When I try and run

sudo nginx -t

I get:

sudo: nginx: command not found

Sounds like something is either seriously messed up or I've been looking at the wrong stuff. The server should be running nginx right?

If you guys don't have any other ideas I'll break the news to the client and start over.

I don't use forge, but yeah - I think it defaults to nginx. Maybe some people more familiar with forge can chip in - but if I was you I think I'd start work on the new droplet anyway and go wrap your clients knuckles - I hope you get to charge them some extra money too! :-)

Thanks guys, I'm going the route of setting up a new instance now. Really appreciate your expertise and time.

No worries - we've all been in similar situations sometime :-) Hope you get it all sorted :-)

@TylerW exit status 203 basically means: "no such file or directory"

It seems your nginx has been uninstalled (or reinstalled in a different location., but seeing how it cannot find nginx -t I don't think so)

The su root bothers me a bit, log in to SSH again, use su root yourself to change to the root user, and use the history command there as well to see what has been done.

Line 60 by the way was very dangerous and should never be done by someone who does not know what he's doing:

// shortened a bit
apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade -y; 

curl --silent --location https://manage.runcloud.io/script/installer/03IIOz0sfLGH9R0GdHkPTrtMAD1531248913ns6tBiCZGrDuY5WJgDuOQLZQhG6MaSeSjklfdsjaklSDJFIlsdfsaklScGIQO0zcRi0AN/7X77cGB9L7Do2w6XrPx3nj5661nUzLifGQLjdoflBKiXs3efsjk32kjsdkjflUlMcCx7C0HWiRsh38xy913omOPbKJBzH08tofml4QTqdO4eaMQ16WlSHWfddfja39j2lkmafdsa324239kOZYfXB4Dn6i0ziVBOfdsfKiF | bash -;

The first tells the system to update all installed packages, forces it to use the default option for replacing your configuration files, if there is no default option, it uses your current configuration files for any package that gets updated and clicks Yes on any question asked during install.

This might have corrupted some of your installed software because the new updated version is expecting config options that are now not set, or even worse, it's default option was to overwrite your config with the new default one.

The second line has downloaded a script and ran it, I tried downloading it, but it now gives a 404, so no idea what the script did exactly.