Florencio's avatar

General approach to debugging the server vs local

Hi, I´m a noob vibe-coder. My local works, my server doesn´t. I'm using forge/digitalocean.

I am having trouble to find a general approach to debugging the server.

First of all, I am confused because the server used to be fine a couple of commits ago, and now, when I reverse to the commit that used to work (with the forge rollback button), it does not anymore. I´m not sure which commands I may have ran on the server that may have caused this. Maybe vendor:publish or something along those lines.

Second, I struggle with a general debugging approach, as I fear that if I do changes that are solely on the server, I will have 2 basically different versions on local and server, which may cause further problems down the road.

I wonder if I can force whatever is working locally on to the server, erasing all differences between them, and if there may be a downside to this.

I also wonder if there is a good way to compare all differences between server and local; am I right in assuming the differences are probably in the git-ignored files?

I also wonder about the general strategy debugging the server; how would you proceed, keeping the above questions in mind?

0 likes
59 replies
Snapey's avatar

Your deployment process should take what you have working locally and make production behave the same.

If you have any manual steps in the process, these would be immediate red flags.

You shouldn't need to compare, blow away your production files and re-install, running composer install in the process.

Have you looked at the production logs to see what is wrong?

1 like
Florencio's avatar

@Snapey this is what I get.

[previous exception] [object] (InvalidArgumentException(code: 0): View [googlead] not found. at /home/forge/default/vendor/laravel/framework/src/Illuminate/View/FileViewFinder.php:139) [stacktrace]

I had removed those ads and my local is fine with it.

martinbean's avatar

@florencio You can’t fix a problem if you don’t know what the problem is, so that’s your first step.

“Doesn’t work” doesn’t help any one. What doesn’t work? What were you doing at the time? What result did you expect? And what result did you actually get?

When you answer the above questions, it will inform you as to where the error may lie, and where to drill down and focus your efforts. This is true if you’re hand-writing code or generating code with an LLM.

2 likes
Florencio's avatar

@martinbean Thanks for your reply. My primary focus with this post is learning a general approach to how to handle this type of situation. But I´m also thankful if we solve the actual problem.

Florencio's avatar

@martinbean I answered your question above (I removed google ads, but the server is still wants the ad). If I now start to fix this on the server, I would have 2 different code versions. I´m confused why this is a problem on the server and not on local. I may have ran some command on the server that caused this, I don´t know.

Snapey's avatar

@Florencio You should have artisan view:clear in your deployment script. Do you? You can manually run artisan commands from the Forge UI

Florencio's avatar

@Snapey This is the script; not sure whether artisan view gets triggered automatically, but I have tried manually. What if at this point I have already differences between local and remote? What are my options to get them back in line?

cd /home/forge/default git pull origin $FORGE_SITE_BRANCH

$FORGE_COMPOSER install

npm i npm run build

if [ -f artisan ]; then $FORGE_PHP artisan migrate --force $FORGE_PHP artisan cache:clear $FORGE_PHP artisan view:clear $FORGE_PHP artisan horizon:publish --ansi $FORGE_PHP artisan horizon:terminate fi

( flock -w 10 9 || exit 1 echo 'Restarting FPM...'; sudo -S service $FORGE_PHP_FPM reload ) 9>/tmp/fpmlock

Florencio's avatar

@martinbean this, I had posted it above

previous exception] [object] (InvalidArgumentException(code: 0): View [googlead] not found. at /home/forge/default/vendor/laravel/framework/src/Illuminate/View/FileViewFinder.php:139) [stacktrace]

martinbean's avatar

@Florencio OK, so read the error? It’s telling you that you’re trying to use a view named googlead somewhere, but that view doesn’t actually exist.

So, find out where in your application you’re calling view('googlead'), and either change the view name to the correct name, or actually create that view file, and then re-deploy.

Florencio's avatar

@martinbean I know that, what I want to know is how to deal with divergent local and remote repos. The bug is only on remote. If I start debugging it on remote, I will have divergent repos. So my question is what are my options:

-force local onto remote somehow (because I believe remote is already divergent, based on the fact that local is running and remote not (I may have changed remote somehow))

-or not do that because of some reason unbeknownst to me

-check the repos with some tool to see where they are diverging and then take a decision

-or something else that I don´t know about.

The main theme of this thread is to see what kind of strategy a pro would adopt in this scenario: Local works, remote doesn´t and may already have diverted from local.

martinbean's avatar

I know that, what I want to know is how to deal with divergent local and remote repos. The bug is only on remote. If I start debugging it on remote, I will have divergent repos. So my question is what are my options:

@Florencio The answer is, you don’t. In this case, if a view doesn’t existing on your remote server and you’re deploying the code from your local environment, then the view isn’t going to exist locally either. It’s impossible for this bug to exist “remotely only”. The view either exists or it doesn’t. If it doesn’t, then the bug is going to be present across all environments.

You should only ever be moving code from your local to remote environments, and never editing on a remove environment directly. That way you don’t get “divergent” environments.

Florencio's avatar

I have tried to re-deploy from the git repo, but it is not working. Pressing deploy on forge.

martinbean's avatar

I have tried to re-deploy from the git repo, but it is not working.

@Florencio “Not working” is of no help to any one. What exactly isn’t working? What result did you get? And what result did you expect?

We can’t see you’re screen, and we’re not clairvoyant, so we don’t know what you mean by “not working”.

Snapey's avatar

The likely issue is that you ARE still using that missing view, but irs not found on the server because you have some letter casing issue. Either the path or the filename has the wrong lettercase, and locally you are using a case insensitive operating system

Florencio's avatar

Thank you. ChatGPT tells me that there are programs to compare local and remote. I will check that and see what I get.

martinbean's avatar

ChatGPT tells me that there are programs to compare local and remote. I will check that and see what I get.

@Florencio How about you stop relying on ChatGPT—which got you into this mess in the first place—and start actually reading the error messages in front of you and engaging your own brain?

You have an issue with a missing view. So, look there. Like @snapey says, you’ll get the issue if you’re using inconsistent casing in filenames (i.e. googlead.blade.php versus GoogleAd.blade.php) which will only appear when you move from a case-insensitive file system to a case-sensitive one (which most Linux distributions are).

So, maybe check that this view file actually exists; and that it has the correct casing? 🤷‍♂️

Florencio's avatar

@martinbean Thank you for trying to help me. As I have said already, I fear that I have divergent code between local and remote, and this is at the heart of my problem. I am looking for general advice rather than specific advice, and I have learned to always fix problems on my local, not on remote, which is helpful.

I don't think what Snapey says applies, because otherwise I would be able to revert to a previous commit, where everything was working on local and remote, but I am not. As you can see, I am engaging my own brain ;).

Florencio's avatar

@Snapey why do you think it is pointless? What if it diverges because of some operation I may have ran only on remote?

martinbean's avatar

As I have said already, I fear that I have divergent code between local and remote, and this is at the heart of my problem. I am looking for general advice rather than specific advice, and I have learned to always fix problems on my local, not on remote, which is helpful.

@Florencio Stop talking about bloody “divergent” code. You’ve been told multiple times now how to rectify that: re-deploy your local version to the remote environment so they’re the same. Advice that is both “general” and specific to your current predicament.

I don't think what Snapey says applies, because otherwise I would be able to revert to a previous commit, where everything was working on local and remote, but I am not. As you can see, I am engaging my own brain ;).

Except you’re not, as you’ve posted an error, we’ve told you what to do to rectify that error, and you’ve not mentioned anything about checking if the view file actually exists, how it’s named, etc.

So, yet again, if you’re getting an error saying a view file exists, then check the view file does exist. If it doesn’t, create it. If it does, check the casing of the filename. But stop wasting all of our time—including yours—by constantly going on about “divergent code” but not actually doing anything to fix the actual bug and error you have in front of you. Otherwise you’re going to get nowhere.

Florencio's avatar

@martinbean Please don´t get upset. I am more about understanding than to quickly solve a problem that I don´t understand. When Snapey said "Re-Deploy", I said I had already done that, and elaborated that I had pressed "Deploy" again on forge. Now you are saying that I should deploy my local to remote, which is new information. Since I am deploying from Git, I would have to push everything to git again, I guess. But I am hesitant to overwrite the old git repo. Maybe I should open a new branch in git and push the code there and see what happens. Do you agree?

I´m trying to find the right rsync command that will show me divergent files. You say I should stop talking about it but you are not explaining how it is possible that my git repo does not work anymore as it did, even when I roll back some commits which were working previously, while local works. But this is my main understanding problem and the reason for this thread. If you think you can help me in other ways, I am open for that, and you could tell me how exactly I should re-deploy, given the information above.

Snapey's avatar

@Florencio You don't need to check for divergence. If you re-deployed then there won't be any. The answer is in the log files. You need to understand why your project does not run on your VPS.

martinbean's avatar

Please don´t get upset. I am more about understanding than to quickly solve a problem that I don´t understand.

@Florencio I’m not upset, but I am getting frustrated then Snapey and I are giving you actionable advice, and then you just keep ignoring it and continue to bang on about “divergent” code instead of just following the steps given to you. At least three times now you’ve been told to check the view exists and if it does, whether the name has the correct casing, and you’ve yet to actually show that you have checked this specific view. So you’re never going to solve errors if you just disregard the errors given to you—and then advice when asking on a forum about said error.

So, yet again, if the error is “view named googleads doesn’t exist”, how about you actually check this view does exist.

When Snapey said "Re-Deploy", I said I had already done that, and elaborated that I had pressed "Deploy" again on forge. Now you are saying that I should deploy my local to remote, which is new information.

It’s not “new information”. It’s the same advice. You should be coding on your machine. You then push code to your Git repository. Deployments then happen by taking the code from your Git repository and uploading it to a remote environment. Code should not be getting created or modified out of this. It couldn’t be simpler:

Code: your machine -> Git -> Remote server

There would be no “divergent” code or branches if you were following this process. I have no idea what you’re talking about when you say “overwrite the old git repo”. You’re not “overwriting” anything; you should just be pushing a new commit to your Git repository, and then deploying that commit.

I´m trying to find the right rsync command that will show me divergent files.

And there you go again. Stop. Going. On. About. Bloody. Divergent. Files! Deploy from your Git repository. It should be your source of truth. Code should not be getting on your server or changing on your server that hasn’t come from your Git repository, which in turn should have come from your computer.

So, yet again, you should be doing the following:

  1. Deploying what’s in your Git repository, to your server.
  2. Then reading and following the advice of any error messages. If you get an error that says a view doesn’t exist, then check the view exists.

Don‘t bother replying again until you’ve done these two steps.

Florencio's avatar

@martinbean I will do what Snapey suggested. I was concerned that I might break the chain of git, I thought there might be consequences to it that I could not oversee, but I guess it is not necessary to worry about it. There is a live version of the program that already has users, and I thought I may have to have a clean git chain from this to the new version. But I guess that is not the case, as long as the database has the same structure. Could you confirm this please? Thank you.

martinbean's avatar

@Florencio The code on your server should only come from your Git repository. The code in your Git repository should only have come from your computer.

Push your code, as it is on your computer presently, to your Git repository. Deploy the code from your Git repository, to your server. That way the exact same code that is on your computer, is on your server. Then address any issues that may crop up after deploying, knowing that it’s the same code from your computer, meaning you can fix it on your computer, push the fix to Git, and then deploy the fix from Git to your server.

ryangurnick's avatar

@Florencio It sounds like you might need to learn a bit how git works

breaking the chain of git

Might I recommend: https://laracasts.com/topics/git

The helpful thing to keep in mind, is unless you are doing profoundly bad things with git (which are possible now due to the GPTs recommending commands that people run blindly without researching their meaning) you will rarely end up with a detached HEAD (which I am guessing is what you mean by breaking the chain of git).

Git repos are normally quite hard to entirely break without being able to run some commands and recover, but they have very little to do with your database.

I think you are mixing things up with your question.

But I guess that is not the case, as long as the database has the same structure. Could you confirm this please?

Laravel does have migrations, which allow you to impact the schema (structure) of the database, those are files that when committed and subsequently deployed will take your database structure from one point to another. However the "git chain" by itself has nothing to do with the database UNLESS you create a migration locally, push those changes to the git repo, and then deploy those changes. If you needed to rollback database migrations, there are ways to handle that, as can be seen from this documentation (https://laravel.com/docs/12.x/migrations#rolling-back-migrations).

You may want to watch some of the series' on laracasts to gain a better understanding of the various technologies you are using, some of which are third party (git) and others are different components of the laravel framework.

As a personal aside, until you know what you are doing with git, I would NOT recommend using GPTs to automatically run git commands as they will likely do things you do not understand and quite possible cannot recover from.

Florencio's avatar

@ryangurnick Thanks for your reply. Yes, ChatGPT can send you down a rabbit hole sometimes, and after 2 days or so you abandon the approach and get back to your starting point. I have realized it can be dangerous, and I'm trying to wrap my head around the basics, to have a feel for when I´m going wrong. Thank you for the video recommendations, I will definitely watch some of them.

But essentially, if I happen to have a broken commit chain, it is not the end of the world, as long as I am working alone, right? I may have to repeat some fixes etc, but it is not crucial to have a clean chain from the old master to the new version, as long as the new version works with the "old master" database, that's what I was trying to say.

Florencio's avatar

@Snapey I know, I just thought I might need an unbroken git commit chain to make the update work, but I realize now that all the update needs in order to work is to have the same database columns... some confusion on my part.

Snapey's avatar

@Florencio The database is the database. Nothing will happen to it unless you run some code to change its state.

Why do you keep referring to it in relation to "git commit chain" ?

Florencio's avatar

Because in my mind, the git commit chain was as crucial to the working program as the database, which of course it is not... I was confused. I did redeploy everything and I kept getting the same error. So it turns out that this line

{{-- {!! \Domain\Advertisements\AdManager::render("left", auth()->user()) !!}--}}

is seen as commented out by local, but not by the server.

I had my head spinning thinking I had done something terrible breaking the commit chain. Still it is a mystery to me why this line is suddenly a problem when it was working before. Anyways, I learned some things in this journey, and I guess I will figure this out now... probably just deleting the line will do the trick. Thank you all for your patience.

Snapey's avatar

@Florencio This line is probably broken because of some letter case issue, as was suggested about a week ago.

Just remember, never edit application files on the server.

2 likes
ryangurnick's avatar

@florencio, can you please say git commit history/log instead of git chain... there is no such thing as a git chain, this is not a crypto currency.

I would recommend you re-read the thread.

I had my head spinning thinking I had done something terrible breaking the commit chain. Still it is a mystery to me why this line is suddenly a problem when it was working before. Anyways, I learned some things in this journey, and I guess I will figure this out now... probably just deleting the line will do the trick. Thank you all for your patience.

It seems like you have already made the mistake of editing files on the server, is this an accurate statement? Are your deploys failing in forge due to git errors (something like a merge conflict?)

martinbean's avatar

@ryangurnick It’s still a “mystery” to OP, because they never confirmed whether the view actually existed, or if it was named with the correct case if it did exist, despite being asked directly by me at least four times, and also at least once by Snapey.

Like, how do you help someone who refuses to read and then still sits there going, “Doesn’t work! It’s a mystery!”

Florencio's avatar

@ryangurnick Hi Ryan, thank you for your suggestion, since I come across of lots of new terms in little time, I will sometimes use a wrong wording, but I appreciate your hint and will try to follow it.

I´m still not sure what caused the fallout on the server, because something must have changed at some point but I don´t know what it is and it is strange that I could not roll back to a previous commit where things where working, completely reinstalling everything.

I have the error itself solved but still don't know what actually changed to cause it.

Florencio's avatar

@martinbean Hey Martin,

I have tried often to make you see my point of view, but you keep insisting on yours. I have a certain curiosity about what caused this; it is fixed now, but fixing it was not my main goal with the thread, it was to see what caused it.

Even if it is correct what Snapey says, that it is some letter case issue, I still don´t know what caused it (because it was working before and I could not roll back to the previously working commits) and that was my focus with my post.

Fixing the bug was easy, but what may have caused is still unclear to me. The question may not be interesting to you, and I may come to the same conclusion (that this does not matter) when I have more experience. But right now it is very interesting to me.

Florencio's avatar

Although I have successfully dealt with the problem, I now had to solve another problem that was related to the remote being out of sync with local. This is what I wish someone would have told me:

-SSH into your remote

  • run git status Check for uncommitted changes: This will show the differences between local and remote (right?) and it also showed me that the remote head was detached (don´t know why, maybe because of the uncommitted changes?)

-git checkout main If it says Please commit your changes or stash them before you switch branches., run

-git restore .

-git clean -fd

-git checkout main

-and then: git pull origin main

I believe this should put your remote back in sync with local. Please check this info before you put it to work, because I am still a noob, but this seems to have solved my new (not discussed here) problem that I had now, probably because of the same issue, remote being out of sync with local. As a noob, I did not come across the idea to do a git status on remote, cause I was always just operating from local to remote via git.

ian_h's avatar

How does the remote server have any idea what you have on your local box? Doing any git command like you're doing cross-references with the git repo itself.

The answer (and preventative action) is clear, as has been described multiple times in this thread by people with a lot of experience:

  • You make code changes on your local box
  • You commit those changes to your git repo
  • You deploy that git repo to your remote server

It really can't be any simpler. If you're doing anything outside of this process, the short response is: you're doing it wrong.

Never (there are exceptions, but this ain't one of them) SSH into your server and start running random commands regarding your application or start monkey-patching files... this will only ever end up in a bad situation.. especially if you have no real knowledge of coding/sysadmin and just relying on some AI generator.

I would say do yourself a favour.... take a step back from the keyboard and start reading/watching/learning before you start taking the plunge and hacking files on a production/live server.

Florencio's avatar

I´m learning as I go, I have no other option (I'm committed to my own site). Maybe I should clarify that I deploy via forge, pulling the remote repo from GitHub into forge. So forge, I guess, is its own repo then again, pulling from GitHub? Because I did what I told you, and it worked.

ian_h's avatar

All you have done with those commands is what Martin and Snapey were already telling you:

Redeploy!

git clean -fd will forcefully, recursively remove any files not under git (which will likely remove your .env file, assuming that forge uses this rather than injecting vars into the environment itself (I don't use forge, so don't know this detail)).

git checkout main will switch to, if not already selected, your main git branch

git pull origin main will then pull the contents of your git repo onto the server.

Essentially, this is the same as what deploying does (less the cleaning).

Forge isn't it's "own repo". I'm assuming you need to give it a path to your repo.. it will only issue commands to interact with that.. it doesn't magically create things.

I'm all for learning as you go.. I'm self-taught as I suspect a lot of others here are too.. but we were born with 2 ears and 1 mouth for a reason... and when someone with considerable experience (which is obvious by the quality of their replies in this thread) is trying their best to help you, for free, I would personally take advantage of that.

There are a lot of helpful people here on this forum and they will spent their (sometimes considerable) time to help and offer advice and even solutions.. so please, continue to ask things as your journey continues, just accept the advice when given.. no-one's going to deliberately mislead you here.

I would also strongly suggest not relying on AI to generate stuff. Hands on is by far a better approach.. and asking a bot to write code for you isn't hands on.

Either way... happy coding and enjoy your time here!

Florencio's avatar

Well I was "redeploying", pushing the deploy button on forge. Didn´t know there was another way, and I think no-one suggested that - maybe assuming I would just do what I did now. Anyway, if another noob runs into this, that's what helped me.

Florencio's avatar

BTW this was the output of the ssh-git status git status HEAD detached from abcf205 Changes not staged for commit: (use "git add/rm ..." to update what will be committed) (use "git restore ..." to discard changes in working directory) modified: composer.lock ...

so I do think it was comparing what's on forge to the GitHub repo; at least there must have been a comparison made...

Snapey's avatar

If your production code has changes on it, then it is because you edited some files on the server - which is bad practice and should never be done.

When you run git status on the server it will tell you what files have been changed, and it may remind you what you did.

When you have changes on the remote, you cannot just press Deploy on forge. It will fail because Git does not want to overwrite the files you have changed.

Deploy on Forge just pulls the latest commit to Git. If you need to change anything, change it locally, commit the change, push to Git. Then deploy.

If you follow this process, it will work every time.

Florencio's avatar

Hi Snapey, thx for your reply. Most of the time, I was pressing deploy on forge and it did not fail - but the code was not identical to what I had on the remote (and local) repo. I just asked ChatGPT about this and it basically said: If the forge pull from the remote (when pressing deploy) does not conflict with changes on the forge server, the changes will remain in the working directory of the server. That's what I didn´t know, and that's what the process I described above helped me to clean up.

Snapey's avatar

A successful gut pull will always give you an identical copy. Don't rely on AI for answers.

Florencio's avatar

But how then would you explain that I ran git status on ssh and it showed me divergences? If that was remote, what was it comparing it with? Maybe I had uncommitted changes on remote that were dragged along? Then a "hard reset" of remote would probably have done the job, right?

But then again, why was I sometimes (on 2 occasions) asked to deal with uncommitted changes when I pressed deploy on forge, but for the rest of the time it just let me deploy, even when I had a "detached head", although I was pulling the newest versions from remote, and what was on the server clearly was not quite the same as I had on local?

Sorry to bother with so many questions, I struggle to understand the mechanics

ian_h's avatar

git status will show the differences between your currently selected branch (probably main on prod) and the current state of the files in the directory on your server. So:

You SSH and make a change to index.php for example.. this file is now different to what's on the main branch in your repo.. so will flag up as a difference in the status output. This has nothing to do with whatever you may, or may not have on your local box. Running a hard reset on the server would indeed reset that index.php as it will reset the code state on the server to the commit hash you provide from the repo.. so in reality, you could go back to the previous commit that would just reset this change to index.php but you could specify a commit hash from 2 years ago and reset the code to that state. Think of hard reset as reverting back to a snapshot sometime in the past. How far in the past you go is up to you. This however, ideally, should never be run on a production server manually.. and please read and understand the implications of this command as it's destructive and could result in a lot of lost work.

If a file is changed on the server, and you try to overwrite that file from git (a deploy will just use a git pull command), it will fail as you need to make a choice in the state of that file you want.. git is smart enough not to just overwrite something without permission.. so you have a choice to either commit or stash the change. You can safely try this locally. Update one of you files and then run a git pull, you'll be presented with the same notification.

detached head means that the code state isn't attached to a particular branch. I'm assuming you set a "deployment branch" or something like that in forge that it will use.. so it doesn't matter if the head is detached or not as it will deploy the main (for example) branch.

I think there's a misunderstanding here on how and what git works and is.. especially between your local box and your server. In short:

  • Your local box knows nothing of what's actually on the server
  • The server knows nothing of what's actually on your local box
  • Git is a "proxy" between the 2.

You make a change locally. Only your local box knows about the change. A git status here would compare your local state to whatever is in git for the same branch.

You commit and push the changes from your local box to your git repo. At this point, the server and your local box will differ only because the server doesn't automatically receive the changes.

When you hit deploy, the server pulls the new changes (the difference between the code state currently deployed and what's in the repo) from git (nothing related to your local box.. you could have already made a bunch of new changes locally).

Once you understand this process more, the reasons for not editing files live in production will likely become more apparent and you'll have a smoother process and less issues in the future.

What I recommend to test this concept locally is:

  • Create a new folder on your local box.. call it prod or something. This will represent your server.
  • Make a change in your normal project folder.
  • In the prod folder, run a git pull command (to simulate a deployment).
  • Note no updates happen in the prod folder, "nothing to update".
  • Commit and push the change in your project folder as normal.
  • Run git pull in the prod folder and see that there was 1 file changed.

If you want to test the "hacking" scenario:

  • Make a change to a file in the prod folder.
  • Run a git pull command in the prod folder.

This will show a notification that you saw on the server about committing changes because git won't just blindly overwrite the manual changes you made with a pull.

Hope this helps explain the process more.

Snapey's avatar

To understand this, you need to say WHAT files are showing as changed.

It could be that these files are actually updated by your code, and what is actually required is a .gitignore entry to let git know that these files are not part of your code.

Git tracks all the files in your project. When you run git status, what you see are changes that have been made to files within your project. These files will probably show as untracked meaning that they are yet to be committed to source.

Git won't update these files incase it overwrites changes you have made.

So you need to spend 5 minutes understanding what files are being changed....

Snapey's avatar

To understand this, you need to say WHAT files are showing as changed.

It could be that these files are actually updated by your code, and what is actually required is a .gitignore entry to let git know that these files are not part of your code.

Git tracks all the files in your project. When you run git status, what you see are changes that have been made to files within your project. These files will probably show as untracked meaning that they are yet to be committed to source.

Git won't update these files incase it overwrites changes you have made.

So you need to spend 5 minutes understanding what files are being changed....

Please or to participate in this conversation.